Website Load Testing with LoadRunner – running out of road
I just came out of an interesting load test planning meeting. It was with an organisation we’d worked with on web performance testing for some time, but this is the first time we’ll be doing major web load testing with them, as part of a bigger ITIL Capacity Planning delivery.
Like a number of companies, they have used LoadRunner in house for a few years, some of the guys in their test team are trained up on how to script in LoadRunner, and in LoadRunner best practise, etc.
What’s interesting is that they are moving away from LoadRunner, and calling on us to do the web load testing that they need.
It’s a trend we’ve seen more and more in the last year or so – I wonder if the trend towards more complex web sites – RIA, HTML5, Flash and so on – means that the wide range of load testing software tools, of which LoadRunner has often been the big business testers tool of choice, are no longer quite able to achieve the ROI they did when websites were simple, and javascript was rare.
With this particular client, there were several reasons why they have taken the step of no longer running LoadRunner themselves for their web load testing (apparently, they will still use the tool for some internal app testing and non-web based stuff.)
Firstly the scripting language can be a burden. In principle you can do anything you need with the (6 or 8 ) LoadRunner scripting languages, but in practise for Web testing, you need to drop down to C to do anything complex enough to mimic real users. After a while, maintaining a library of pre-built scripts is a considerable effort, and trying to do a quick system-wide load test of complex multiple page dynamic users journeys with lots of random choices (e.g. choose a pair of shoes from the 10 listed that matched your search) becomes difficult – you need more time to prepare. So quick load test assignments become two weeks projects.
One of our USPs at thinkTRIBE is that we have a highly flexible scripting platform, developed in-house. So it makes it much easier for us to script complex Journeys and then get them up and working.
Secondly, a keynote reason why the client wanted to give us the load testing work was so that the bulk of their own web testers would no longer need to know about LoadRunning scripts. They wanted their team to focus their time on other aspects of the in-house testing, where knowledge of the business logic is really vital.
Thirdly, the Record and Playback Myth (as my colleague R. Gomez here calls it) – driven by AJAX and Web 2.0. The whole idea that record and playback can be a quick and easy way to script load tests is gone. Engineering wise, trying to modify a recorded script to do what you want it to do, is more time consuming than scripting manually from scratch. Yes, bosses like the idea of record and playback as a time saver, but it’s not really true for anything except the most simple web site testing.
Fourthly, it’s not the Cloud Way. In this cloud world we’re all entering, the metaphor of the day is – don’t have resources sitting around all year, just for the occasional use: instead call up external resources to handle your peaks.
This applies to web load testing within capacity planning – the time when load testing is most critical is just before a major code release, and that is just the time when internal staff resources are stretched. The scenario of it’s inevitable that the code is a little late in delivery; that some bugs are picked up late in the day by UAT and regression testing; that the fixed code doesn’t get to the staging environment until there’s just too little time left to do a full web load test; and anyway, the testers who know LoadRunner just got pulled off to help with some testing around the newly fixed bugs and can’t do the main system-wide testing.
That was the story that came out of the client meeting today. Their last major release had failed when it went live. Yes, some load testing had been done before the code had been passed over. Yes even whilst in UAT in the test environment some performance testing had been done. But still parts of the user interface were way too slow on launch.
Despite the best will in the world, the actual load testing done was cursory. After digging into the details, it turned out that whilst the Dynamic User Journeys tested are supposed to mimic real users, by searching from a wide range of random terms, and picking products at random from the ones listed on the page – that was not how the latest code had been tested. Something in the new code meant that that approach didn’t work on the old LoadRunner scripts: it was going to take some time to update the scripts: and after losing 4 hours trying to achieve that and failing, the testers fell back to an old script that did work on the new code base – but that one did not do the random elements.
So the load test had simulated 1000 users all searching and buying the same product ID! With predictably rosy performance numbers. As the web site had various levels of clever caching built in, at the page, page element and database level – it of course performed like a rocket in testing!
When you have that kind of misleading metrics for your web site, confidence across the business in the ability of the eCommerce team to deliver working code is undermined.
Lastly, what swung it for the client, was that with our load testing engine and Portal, they can mix and match how the work for a load test project is divided down to different teams.
They want us to be in charge of keeping the Dynamic User Journey scripts up-to-date at all times. They want our engineers to run the web load tests at their main release cycle events – four times per year. But they want access to our Load Test portal, so that their own Capacity Planning teams can plan ad-hoc smaller scale testing between releases, so they can focus on just the Journeys that most changed in the release, and do web testing early and often, under their own control.
There was one other interesting angle this client had taken. As part of their ITIL Capacity Management activity. It didn’t come out of the meeting today, but I know that this client had looked at Cloud-based web load testing solutions too – but they are a big organisation, and it was clear that any one cloud was not going to be able to reliably and on-demand create the level of traffic they needed. They wanted to push their 622Mb bandwidth to the limit, and doing that from just one cloud was unlikely to be feasible. Our approach of having our load test servers spread out around over half a dozen or so data-centres, meant that our peak at each data-centre was much further down in the noise. It meant there was far less chance of our own infrastructure being the bottleneck.
PS – there are of course many fans of LoadRunner out there, and whether it’s because they haven’t experienced anything better, or because they have become such gurus of LoadRunners specific quirks that they are rarely phased by the difficult cases – those guys will be happy with what loadRunner can provider. Other folks have posted some of the LoadRunner annoyances and limitations they have found.
PS – this is what the guys at SQATS say about the record and playback myth: “Record and Playback myth is a test automation nightmare. This methodology alone is probably most responsible for software companies all over the world to almost give up on test automation out of sheer frustration and return to manual testing. Ironically, this is the tag line that most vendors use to sell their products.”
