Keeping It Real: Avoiding Pitfalls When Load Testing Websites Using A CDN
There was a time when doing a Website Load Test for a client using a Content Distribution Network (CDN) was an infrequent activity, but these days it’s a common occurrence. There are more CDN suppliers of course, the newer ones such as MaxCDN, the traditional big names like Akamai and LimeLight, and some Datacentres such as UKFast offer CDN services directly to clients themselves.
Over the years of testing websites using Clouds and Content Distribution Networks there are a number of pitfalls that we’ve seen organisations struggle with, so it’s nice now to be able to provide some tips to help people get the best from this sort of load test.
Pitfall 1: Testing as if there is no CDN in use
At first glance this seems sensible: a good load test will be a good load test no matter how the target web site is set up. But when a site is using a CDN there are actually a number of extra complexities that need to be planned for.
Firstly, if you test out of hours, to avoid inconveniencing the users of your own website then chances are that this means it is out of hours for your CDN provider. If you’re unlucky, you’ll hit a time when they are doing nightly backups or other background tasks and thus you’ll see an unusually high error rate!
Conversely, as it is out of hours for all their other clients too, the CDN will have huge swathes of free capacity, all available to you on your one load test. So you’re likely to find the CDN delivered parts of your website perform pretty well!
It’s easy to end up with unsatisfactory test. For example, a major UK retailer ran with a load test supplier who repported back that the target traffic levels had been hit but alot of errors seen. This contradicted the evidence of the client’s own servers, which had run at very low % utilisation during the overnight load testing, and were error free. It was also contradicted by the CDN supplier who disputed the errors had occurred and had logs to support their claim.
Needless to say that retailer didn’t continue with that load test supplier.
The important thing to get from a load test is realism. Without that the test results are worthless for decision making, even if they look good on the presentation!
Pitfall 2: Testing your website without the CDN elements
The pitfall above, may suggest to you the idea that testing of a CDN supplier is a pointless exercise, unless you can generate enough traffic to reproduce the load of all the clients’ peaks together.
That’s a lot of traffic!
If you’re buying your load test from a supplier who charges based on the amount of traffic you generate then that consideration alone means you may think about testing the website without the CDN elements.
This approach may not be 100% wrong, but it can easily lead to load test specs that are no longer ‘Doing what the Customer does’ and so are becoming unrealistic. Worse, depending on how your pages are constructed, it may be easy or very difficult to seperate out the CDN.
Pitfall 3: The Cloud /CDN is so big and reliable, we don’t need to worry about it !
In the real world, of course, Cloud systems have bugs and problems, CDNs have less than perfect load distribution models and human errors mean your website performance may not always be as well supported as you think you’re paying for.
We had that exact case just a few months ago, and as a result helped a major site, built on Microsoft’s Azure cloud, to track down a nasty performance bottleneck that caused them to have terrible user experience on the big first night after launch.
The root cause was the fact that major performance differences resulted from minor changes in subtle API usage, between default and tweaked settings in the Azure platform.
In that case, the whole application was in the Cloud so it had been tempting to ‘assume the cloud is infinite and don’t bother load tetsing’ .
It was the major fail on the first night, that forced a load test: the system clearly was in trouble, and the bottlenecks needed to be found quickly.
Pitfall 4: Ensure you Keep it Real
As part of the speed and effectiveness of pages helped by CDN the levels of caching are necessarily much higher.
If the User Journeys you specify are simplistic, if they only include a small percentage of all the products on your site, then the results you obtain will be very wide of the mark. This is inaccuracy from over simplistic scripts is amplified to much greater degree than on a non-CDN site where the caching efficiencies may not be so pronounced. After all, at the core of the CDN ROI is that it’s a caching technology that helps keep traffic off your own systems.
To measure meaningfully you need to ensure you’re using Dynamic Users Journeys that don’t just choose and buy products from a pre-determined short-list of, say, 50 products. Instead, dynamic Journeys look into each live page during the load test and actively pull from the page any of the choices possible, choosing one at random.
- start right at the home page,
- find the top level Categories and choose one at random (maybe on your site it’s Womens wear, Menswear and KidsWear)
- at the next level down, choose a random sub-category
- within that choose a random colour, or style or et
- finally choose a random product from those offered in the final page.
That way, your load test will visit potentially every product in your store.
The Out of Stock Load Test Problem: Solved
If during the load test, certain products go out of stock as a result of the ‘purchases’ made during the tes,t no worries, your site will no longer offer those products within the pages, and so the dynamic Journeys will not be able to find them and select them.
Another “keep it real” factor to bear in mind is the need to be able to cope with the subtle complexities of your site. For example, on some retail sites, as the visitor searches and narrows down their choices by factors such as style, colour, etc the logic is such that whenever the search only matches one product the visitor is taken directly to that product. They don’t ever see a search result page saying ‘ 1 product matches your search’. Whereas if the search finds 2 products, the visitor is shown a ‘2 products match’ result page.
You need to ensure that your dynamic script can handle that subtle complexity and detect the two equally valid search results. Not to mention handling all the other subtle differences that you’ve designed into your site that make it a better shopping experience, but a harder platform to test with simplistic tools.
Of course all of that is before we even start to think about how we should weight the different journeys that take place within a test to reflect an even great realism.
Websites underpinned by Content Distribution Networks, or built with their systems half inside the Cloud, are still web based applications.
To test their capacity meaningfully, to get performance measurements that are valid in the real world, can be achieved, but needs a little more thought and care in the planning, and a little more subtle power in your script running technology.
Your old load test tools and suppliers may no longer fit the bill.
In today’s environment where the rush to launch is getting faster, with less time to test prior to going public, it’s also a good time to bring in specialist help for those focused, furious periods just before go-live and to free up time for your own staff to use their unique knowledge, that no supplier can ever have, as to how they built and architected and coded your website. This way they are free and ready to fix the bottlenecks that load testing reveals, rather than tied up in the load testing themselves.