-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
526 lines (494 loc) · 18.7 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<title>Load Testing in The Azure Cloud</title>
<link rel="stylesheet" href="css/reveal.css">
<link rel="stylesheet" href="css/loadtest.css">
<link rel="stylesheet" href="css/theme/black.css">
<!-- Theme used for syntax highlighting of code -->
<link rel="stylesheet" href="lib/css/monokai-sublime.css">
<!-- ToDo: Work on Slides first then on Theme modifications -->
<!-- Printing and PDF exports -->
<script>
var link = document.createElement( 'link' );
link.rel = 'stylesheet';
link.type = 'text/css';
link.href = window.location.search.match( /print-pdf/gi ) ? 'css/print/pdf.css' : 'css/print/paper.css';
document.getElementsByTagName( 'head' )[0].appendChild( link );
</script>
</head>
<body>
<div class="reveal">
<div class="slides">
<section>
<h2>Load Testing in the Azure Cloud</h2>
<p>Alan Barr</p>
<p>QA Engineer</p>
<img style="background-color:white;" data-src="images/azure.png" alt="Microsoft Azure Cloud Logo">
</section>
<section>
<h3>What is load testing?</h3>
<aside class="notes">
Load testing is conducting a test of a system to determine how much load it can sustain for a specific duration.
</aside>
</section>
<section>
<h3>Why Load test?</h3>
<p class="fragment">
Load Testing gives us reporting to know what changes are making the most impact
</p>
<aside class="notes">
Integrating load testing as part of your project ramp up is beneficial at any time. Having knowledge and reporting on what your project can support is beneficial for planning. As features are completed related to performance load tests can be written once and run many times to verify performance improvements.
</aside>
</section>
<section>
<h3>Load Testing myVu</h3>
<p>Where do I start?</p>
<p class="fragment">What tools are out there I can use off the shelf?</p>
<p class="fragment">What constraints do I need to consider when choosing a tool?</p>
</section>
<section>
<h3>Many Tools Available</h3>
<p>Open Source and Proprietary, GUI/Code</p>
<ul>
<li>JMeter</li>
<li>Gatling</li>
<li>Locust.IO</li>
</ul>
<ul>
<li>Grinder</li>
<li>Visual Studio</li>
<li>HP Load Runner</li>
</ul>
</section>
<section>
<h3>Use the right tool for the job</h3>
<aside class="notes">
There are many choices of tools available from open source to proprietary. Ideally choose one that is free and popular with
resources. Consider the tradeoffs of your challenge. Is GUI good enough? Does it need to be coded? Is it in a language your staff knows?
Is it using a domain specific language and is it worth the time to train on it?
</aside>
</section>
<section>
<img alt="My Veterans United Logo" data-src="images/myvu-logo.png">
<h3>myVu Challenges</h3>
<p class="fragment">All test environments use HTTPS</p>
<p class="fragment">Tool would need to handle multipart form uploads</p>
<p class="fragment">
Ability to route requests from different locations not from local network
</p>
<p class="fragment">Allow for different upload scenarios</p>
<aside class="notes">
The challenges inherent to testing myVU drove adoption of the tool. We needed to handle communicating with the server using the correct credentials,
handle a variety of different file types and sizes, and upload these files from different geographic locations to simulate real life user network load.
</aside>
</section>
<section>
<section>
<h3>The Tools</h3>
<section>
<h4>CasperJS/PhantomJS</h4>
<span class="fragment">
<p>Headless Browser with JavaScript API</p>
<img alt="PhantomJS Logo" style="margin-bottom:150px;" data-src="images/phantomjs-logo.png"> <img alt="PhantomJS Logo" data-src="images/casperjs-logo.png">
</span>
<p class="fragment">
Login as a user and save authorization tokens for future requests
</p>
<span class="fragment current-visible">
<p>Setup configuration for writing the file, command line options, and specific user config</p>
<pre>
<code>
var fs = require('fs');
var casper = require('casper').create({waitTimeout:10000});
var user = casper.cli.get('user');
var config = require('./config.json');
var urlApp = config["host"];
var textFileName = user == 'user1' ? "useroauth.txt" : "useroauth1.txt";
var tokenFileName = user == 'user1' ? "usertoken.txt" : "usertoken1.txt";
</code>
</pre>
</span>
<span class="fragment current-visible">
<p>Go to the page and fill the form out and submit</p>
<pre>
<code>
casper.start(urlApp, function() {
this.waitForSelector('form', function() {
this.fillSelectors('form', {
'#LoginId': config[user]['username'],
'#Password': config[user]['password']
}, true);
});
});
</code>
</pre>
</span>
<span class="fragment current-visible">
<p>Wait long enough for cookies to load grab the refresh code</p>
<pre>
<code>
casper.then(function() {
this.wait(5000,function(){
this.echo("saved oauth token");
var cookies = phantom.cookies;
cookies.forEach(function(e){
if(e.name == 'refreshCode') {
var savedCode = e.value.replace(/\%22/g, '');
fs.write(tokenFileName, savedCode, 644);
this.echo("saved refresh token");
}
});
});
});
</code>
</pre>
</span>
<span class="fragment current-visible">
<p>Wait for the authorization header code</p>
<pre>
<code>
casper.on('resource.requested', function(resource) {
var authHeader = resource.headers.forEach(function(e){
if(e.name == "Authorization"){
var savedAuth = JSON.stringify(e.value).replace(/\"/g, '');
fs.write(textFileName, savedAuth, 644);
}
});
});
casper.run();
</code>
</pre>
</span>
</section>
</section>
</section>
<section>
<h4>Locust.IO</h4>
<img class="fragment" alt="Locust Load Testing Logo" style="background-color:white;" data-src="images/locust-logo.png">
<p class="fragment">Python 2.7, Requests and Requests Tool-belt modules</p>
<span class="fragment current-visible">
<p>Load config data</p>
<pre><code>
from locust import HttpLocust, Locust, TaskSet, task, events
from requests_toolbelt import MultipartEncoder
import requests
import json
from random import randint
import sys
import datetime
requests.packages.urllib3.disable_warnings()
token0 = open("useroauth.txt", 'r').read()
token1 = open("useroauth1.txt", 'r').read()
refreshToken0 = open("usertoken.txt", 'r').read()
refreshToken1 = open("usertoken1.txt", 'r').read()
config = json.load(open("config.json"))
token = [token0,token1]
atlasID = {token0: config["user1"]["loanAccount"],
token1: config["user2"]["loanAccount"]}
host = config["host"]
</code>
</pre>
</span>
<span class="fragment current-visible">
<p>Create a task to run</p>
<pre>
<code style="max-height:500px;">
class MyTaskSet1(TaskSet):
@task
def file_upload(self):
global token
attach = open('assets/Intro_to_Agile.pdf', 'rb')
randomNumber = randint(0, 1)
thisToken = token[randomNumber]
headers = { 'Authorization': thisToken }
m = MultipartEncoder(
fields = {
'atlasLeadId':atlasID[thisToken],
'blitzDocFolderId':'-1',
'documentType':'upload',
'file0': ('Intro_To_Agile.pdf', attach, 'application/pdf', {'Expires': '0'})
}
)
r = self.client.post("/api/fileStorage/Upload",
data=m,headers=headers, verify=False, catch_response=False)
print("Locust instance ({}) executing my_task".format(self.locust))
</code>
</pre>
</span>
<span class="fragment current-visible">
<p>Let Locust know what tasks to run</p>
<pre><code>
class MyLocust(HttpLocust):
min_wait = 5000
max_wait = 15000
host = config["host"]
task_set = MyTaskSet1
</code>
</pre>
</span>
</section>
<section>
<h4>Start Testing?</h4>
<img src="images/starttest.png" alt="Locust Web UI">
<aside class="notes">
We could very well just start our tests straight away. However, there is some ground work one must do. One key piece of this is running load test
in a distributed environment. While we could certainly do it from one computer or local computers we wanted to simulate our users experience.
Which leads to setting up the cloud environment.
</aside>
</section>
<section>
<h4>Azure Cloud</h4>
<br>
<p>Create a distributed environment to simulate distant users</p>
<br>
<ul>
<li>Set up a network gateway</li>
<li>Set up Linux Virtual Machines</li>
<li>Configure ports so machines can communicate</li>
<li>Open a port so these machines can access your network</li>
<li>Optionally: Setup an Azure Function Endpoint to send results to an Azure Storage Table</li>
<li>Optionally: Graph our Data in an Azure Notebook</li>
</ul>
<aside class="notes">The benefits of using linux vms are that they were lightweight no GUI and fast to script the setup of. Deploying changes to our
test scenarios was as simple as rsyncing our changes and broadcasting the commands to execute. The majority of my testing was inexpensive the Azure Function and noSQL database added more cost due to the requiring of a specific azure plan.
</aside>
</section>
<section>
<section>
<h4>Storing Result Data</h4>
<span class="fragment current-visible">
<p>Configure endpoint to accept data and save it to the NoSQL storage table</p>
<img alt="Azure Functions Example Sending data between places" data-src="images/streamscenario.png">
</span>
<span class="fragment current-visible">
<pre>
<code>
module.exports = function (context, data) {
context.log("received report");
if(data) {
context.res = {status: 200};
context.bindings.locustoutputTable =
{
"partitionKey":"l",
"rowKey": context.bindingData.InvocationId,
"client_id" : data.client_id,
"user_count" : data.user_count,
"name" : data.name,
"method" : data.method,
"max_response_time" : data.max_response_time,
"total_response_time" : data.total_response_time,
"min_response_time" : data.min_response_time,
"last_request_timestamp" : data.last_request_timestamp,
"num_requests" : data.num_requests,
"total_content_length" : data.total_content_length,
"num_failures" : data.num_failures,
"start_time" : data.start_time,
"test_name" : data.test_name,
"num_reqs_per_sec": data.num_reqs_per_sec,
"response_times": data.response_times
};
}
else {
context.res = {
status: 400,
body: { error: 'Must be submitted as Locust Report Data'}
};
}
context.done();
}
</code>
</pre>
</span>
<aside class="notes"><p>An Azure function allows me to create an API endpoint without having to setup a server.</p>
<p>The Azure storage table is an inexpensive noSQL database I can put my data into. There are other options like noSQL blob storage
or a more expensive sql database. I could also send my data to other Azure services as well.</p>
</aside>
<span class="fragment current-visible">
<img alt="Azure Database with Rows" data-src="images/storagetable.png">
</span>
</section>
<section>
<h4>Azure Notebooks</h4>
<img src="images/notebook.png" alt="Azure Python Notebook">
</section>
<section>
<h4>Graphing Our Results</h4>
<img style="background-color:white;" src="images/response_times.png" alt="Response Time Results with a large distribution">
</section>
</section>
<section>
<h3>Testing Iteration</h3>
<span class="fragment"></span>
<aside class="notes">
We ran initial tests and the performance was not as robust as we expected.
We ran the tests and we were limited in how many users we could run without causing many disconnects from the server.
Talking with our team about the poor results we learned from our network team that the internal testing network was bandwidth constrained.
They were able to increase it from 10mb/s to 200mb/s. We were now able to push more users through the funnel.
We also did not have a lot of information on the state of our webservers we needed some type of monitoring system.
We purchased a service called LeanSentry to monitor our test bed webserver and were able to see much finer grain data about our test results.
We also ran out of space on our file storage server causing some headache for other teams.
</aside>
</section>
<section>
<h3>Monitoring Results</h3>
<aside class="notes">
Once we had monitoring setup we were able to better make changes during our tests and see the impact of those changes.
Drilling into the web server monitoring software we learned that our main bottleneck is logging.
Waiting for a shared log to become available was causing the biggest pause.
We determined that all loggers and log readers work on the same file causing performance issues
We had turned off logging and saw a nice increase in our uploads but it wasn't a solution we could depend on.
</aside>
<img alt="Graph with Monitoring Statistics showing many requests at different times" data-src="images/monitoring.png">
</section>
<section>
<section>
<h3 class="fragment">Testing Our Hypothesis</h3>
<p class="fragment">Can we improve upload times by changing how we log our data?</p>
<aside class="notes">
The turn around time for finding what a possible solution could be and reaching out to our senior engineers to come up with an experimental solution took a matter of days. Testing these changes was only a matter of ours. By automating your load testing procedure you can test your hypotheses and verify your results in a very rapid way.
</aside>
<span class="fragment">
<p class="skyblue">Test run with current logging on</p>
<table>
<thead>
<tr>
<td># Requests</td>
<td># Failed</td>
<td>Median</td>
<td>Average</td>
<td>Min Resp</td>
<td>Max Resp</td>
<td># Reqs/second</td>
</tr>
</thead>
<tbody>
<tr>
<td>4107</td>
<td>2135</td>
<td>31</td>
<td>32.4</td>
<td>0</td>
<td>108</td>
<td>0.63</td>
</tr>
</tbody>
</table>
<p><small>Request times in seconds</small></p>
</section>
<section>
<p class="skyblue">Test run with experimental logging utility</p>
<table>
<thead>
<tr>
<td># Requests</td>
<td># Failed</td>
<td>Median</td>
<td>Average</td>
<td>Min Resp</td>
<td>Max Resp</td>
<td># Reqs/second</td>
</tr>
</thead>
<tbody>
<tr>
<td>495</td>
<td>588</td>
<td>11</td>
<td>11.38</td>
<td>0</td>
<td>22.8</td>
<td>0.5</td>
</tr>
</tbody>
</table>
<p><small>Request times in seconds</small></p>
</span>
</section>
</section>
<section>
<h3>Results</h3>
<p>65% decrease in the median and average upload times</p>
<p class="fragment">Small change with an impressive impact</p>
<aside class="notes">
By automating the load testing continually auditing our systems and starting communication around the different teams we were more rapidly able to test our ideas and find a fix.
</aside>
</section>
<section>
<section>
<h3>Questions?</h3>
</section>
<section>
<h3>Thank You</h3>
</section>
</section>
</div>
</div>
<script src="lib/js/head.min.js"></script>
<script src="js/reveal.js"></script>
<script>
// More info https://github.com/hakimel/reveal.js#configuration
Reveal.initialize({
history: true,
// More info https://github.com/hakimel/reveal.js#dependencies
dependencies:
[
{ src: 'plugin/markdown/marked.js' },
{ src: 'plugin/markdown/markdown.js' },
{ src: 'plugin/gamepad/gamepad.js' },
{ src: 'plugin/notes/notes.js', async: true },
{ src: 'plugin/highlight/highlight.js', async: true,
callback: function() { hljs.initHighlightingOnLoad(); setupPad(); }
}
]
});
function setupPad() {
var gp = new GamepadMicro();
gp.onUpdate(function(gamepads) {
if (gp.gamepadConnected) {
var presenterPad = gp.gamepads[0];
var buttons = presenterPad.buttons;
if(Object.keys(buttons).length > 0){
var keys = Object.keys(buttons);
keys.forEach(function(button){
var btn = buttons[button];
if(btn.released){
switch(button) {
case "dPadLeft":
Reveal.navigateLeft();
break;
case "dPadRight":
Reveal.navigateRight();
break;
case "dPadUp":
Reveal.navigateUp();
break;
case "dPadDown":
Reveal.navigateDown();
break;
case "leftTrigger":
Reveal.navigatePrev();
break;
case "rightTrigger":
Reveal.navigateNext();
break;
default:
break;
}
}
if (btn.pressed) {
//button is being pressed, run desired action on release
}
if (btn.held) {
//button is being held
}
});
}
}
});
}
</script>
</body>
</html>