Congratulations to Innovation Birmingham Campus-based Droplet, who has produced the most techy tech-blog we have seen in a long time! Read it here:

Droplet Infrastructure: Stress Testing Basics

An engineering blog post by Innovation Birmingham Campus-based Charles Marshall from Droplet:
 Before even thinking about using Beta software in a production environment we need to make sure the architecture can hold up to the task. With a brand new operating system like CoreOS there are a lot of questions about performance we need try out before we can sleep soundly at night.With AWS being a very mature and widely adopted platform we can safely tick off most of the hardware related concerns. With the current terms and conditions, we had to let the right people know before we did any testing. If you are planning on doing high loads, you might want to enquire about ELB pre-warming as well.

Load testing has been around for a long time, and there are many good tools out there for it, but the most common open-source tool is Apache’s JMeter.

The key feature we are looking to use in JMeter is its ability to run an SSL proxy to record what our application can do, then we can mimic and replay these actions later, but scaling them up in numbers.

We will be using an iOS device, so we can install custom SSL certificates easily and use the latest version of our application.

Setup

To simulate a full end-to-end test of our CoreOS environment and new Docker based services, we want to be as close to how our user-base actually operate in real life. To do this, we use JMeter to act as a proxy and record everything the application does.

JMeter can be intensive, as its memory and CPU usage will scale on how many concurrent threads (Users) and tests are run. To ensure we don’t eat in to our testing environment, we run our tests on our own laptops.

As a Java based piece of software we need to do some preparation work (we are mostly Ruby based) to get JMeter running.

The main component is, of course, installing the JDK:

sudo apt-get install openjdk-7-jre

The version needs to be 7+ to ensure all the features we need are enabled. Once the install has finished, next we install JMeter itself (in this case, from the source binaries):

cd $DIR ;
wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org/jmeter/binaries/apache-jmeter-2.11.tgz ;
tar xvzf apache-jmeter-2.11.tgz ;
rm -f apache-jmeter-2.11.tgz ;

This will download and extract JMeter 2.11 into $DIR. Once you have done that, you just need to add to your $PATH. Firstly, lets keep things versioned and neat by creating a symlink to a more standard directory:

sudo ln -s $DIR/apache-jmeter-2.11/ /usr/local/bin/jmeter

Now we need to add this to the your $PATH:

PATH="${PATH}:/usr/local/bin/jmeter/bin" ;
echo "PATH=${PATH}" >> ~/.bashrc

You can change ~/.bashrc to ~/.profile or similar depending on what your setup is.

Running the SSL Proxy

For security, our application communicates to our API via SSL, so in order to record what is going on we have to insert a middle-man with an SSL certificate that both JMeter and the device have installed. This will log all traffic, so you probably want to disable Facebook, Twitter et al so you don’t get erroneous data.

Before we start JMeter, we want to change the SSL certificate validity length, as by default it is rather short lived. Open up the jmeter.properties file:

vim /usr/local/bin/jmeter/bin/jmeter.properties

Now find proxy.cert.validity in the file; this is the specified in number of days, so increase that to something like 365 and save the file. You can now start JMeter with a simple command:

sudo env PATH=$PATH jmeter
*This was run as sudo as during our testing we had issues with SSL cert creation otherwise.

You should now be presented with a GUI interface. As JMeter has hundreds of features and is a very in-depth toolset, I would recommend having a read of the various guides on how to it works (Blazemeter have an extensive set).

To get started, we need a workbench with a recorder:

With this added we can now run a proxy on our local machine on port 8888 that will record all calls except for static assets (images, css files, etc) that are excluded with a pattern match.

To get the SSL certificate, run this test and JMeter will auto-generate a USR and CRT file and show a information message about the certificate creation.

Installing the SSL Certificate on an iOS Device

The USR and CRT files are stored in JMeters bin directory:

/usr/local/bin/jmeter/bin/ApacheJMeterTemporaryRootCA.usr
/usr/local/bin/jmeter/bin/ApacheJMeterTemporaryRootCA.crt

If you can see these files, then your setup has worked and the certificates have been generated correctly. At this point, stop the JMeter test to avoid accidental data collection.

Make sure your iOS device is on the same (private) network as the machine you are testing on and is web accessible. You will need the JMeter host to be running a website, so either a default nginx or apachevhost etc that will respond based to the IP address.

Copy the CSR and CRT files from the JMeter bin directory to the public root of your default website (remove these after) and give them a sensible name, like jmeter.crt etc.

On the iOS device, open up Safari and vist the JMeter hosts IP:

http://$HOST_IP/jmeter.crt

This should open up a profile interface that will allow you to install the certificate on the device. Once that is installed, we can now set the device to use our JMeter host as a proxy.

iOS Proxy Setup

Go to you ‘Settings’ -> ‘Wi-Fi’, find the network you are currently attached to and click on the arrow next to it.

At the bottom of that screen will be a section header called ‘HTTP Proxy’, turn this to ‘Manual’. In the fields that are presented, fill in the ‘Server’ as the IP address of the JMeter host; for ‘Port’ enter 8888 (or whatever you set, some versions default to 8080) and leave authentication off. Here is a guide in pictures, but ignore the last step about logging in.

Recording

Go back to your JMeter host and run your test again. Open up the application on the iOS device and start doing things, your actions are now being recorded!

To view what is being recorded, expand your recorder (shortcut, press + while highlighted). You should have a list of top level calls that have been made.

Recorded Data

In the above we made a call to https://google.co.uk with a search query. If you select and expand the controller called “/”, you can see the HTTP Requests that it generated and the contents of those requests.

On typical web applications, or even sites that remember you in some way (like logins, shopping baskets and so on) you will normally be able to view the cookie information and any other parameterised data sent along.

On “/” we don’t see much, but if we take a look at the search call, there are many more parameters:

This is great if you are checking for data leakage (another time, maybe), but we want to load test out setup, so we need to be able to repeat these calls many times with concurrency; to do that we will add in a ‘Thread Group’ and a ‘Recording Controller’.

Applying the load

The thread controller enables us to specify frequency information about the request, looping, error conditions, etc and control the HTTP calls that are made. Our new test setup should look like this:

Above we have modified the HTTP(S) Recorder to store results under our new ‘Recording Controller’, which sits under the ‘Thread Group’.

By setting values on the thread group (see image below) we can decide just how much load we want to test out.

Not very hard in this case; we just want to see a working test, not cause any problems.

Although this test runs smoothly, with no errors, it is always useful to see the results for each HTTP request so we can diagnose any that do fail, or just confirm the resulting data is as expected.

Viewing results

With a ‘Result Tree’ (shown below) you can view the original request, the response headers and response data.

Sometimes this is just too much information, you may have thousands of requests and all you need to know are the basics such as failure rate and throughput.

This ‘Summary Report’ gives a good overview of what happened in your test without the fine detail.

Advanced Testing

These tests are very basic, they are using only a single user (via the cookie values) and although they have some concurrency, are not a real world example of load.

Most modern applications (including ours) will have cache based on the user and the data they are asking by implementing tools such as Redis and Memcached.

We need a more advanced test, that can modify the values stored in the cookies, allow us to swap auth tokens and split our end points in to different groups. JMeter can do this, and we’ll cover that next time.

Read more about Droplet or to download the app visit the Droplet website

Posted in News

Leave a Reply

Your email address will not be published. Required fields are marked *

*