If you already know that your websites don’t just have one kind of user, always taking the same path through the same site, taking the same amount of time, then you will realise the limitations of a website Load Test approach that is not based on Realism.
eCommerce managers need to know that web performance management is in place, ensuring they don’t have yo worry day by day about the website and ‘will our site be OK for the next peak season’. So data available from simpler, traditional, load testing methods that concentrate on basic metrics such as concurrent user sessions, and page views, is no longer enough.
The flexibility and multi functionality of sites has lead to increased sophistication in user behaviour as there are many more ways for customers to interact with sites. This can also make load testing more challenging where an organisation is determined to ensure that all these different types of users and usage patterns have been understood and optimised.
Exploring The Foothills
Let’s take the expected functions of an online grocery site as an example to explore. On such a site you might expect to visit for any of the following reasons:
- To browse products
- To buy goods
- To book or change a delivery slot
- To amend your order
- To change your account details or set preferences
- To take advantage of a particular offer
The amount of server resources taken will vary greatly between those different Journeys – your site may be able to cope with 100,000 users following one type of user journey per hour, but only 10,000 users following a different route! Because of the variation in several factors:
- Session length per journey, the weight of pages, and amount of computer power needed per request type
On top of defining your Journeys realistically: for an effective load test it is also important to ensure realism in deciding what % of your load traffic should be spread across each journey type: to reflect the percentage breakdown as per your real users on the site
Base Camp – start with your analytics not your server data!
Of course to calculate these ratios you will need to analyse and correlate data from your web analytics systems at the very least, and to get the kind of comprehensive understanding that you want you may very well need more. See Predicting next years website traffic – and Load testing realism
With sites becoming increasingly complex and relying on the integration of many different systems, effective load testing requires good use of your web analytics data.
Historic data may, of course, “not be a guide to future performance” (as the pension funds say!) , but the aim is to be defining your user journeys around “reality based” as opposed to merely “rate based” metrics.
(It can be very easy to fall into a false sense of security with overly simplistic load tests due to having inadequate data at the specification stage. What can be especially frustrating is testing that returns great results under test conditions but finding out that this is not backed up by user experience with a site that does not deliver the same performance in reality. If you are looking to your data for capacity planning then it is vital you that really do have realistic results.)
Go beyond ‘Concurrent User’ vagueness
Another thing to consider when looking to develop “reality based metrics” is the old measure of “concurrent users” in load testing, and this may be a measure you still find yourself getting asked for. However, concurrent users as a standalone number doesn’t really mean all that much!
That is worth repeating: Concurrent Users as a standalone capacity metric doesn’t really mean all that much!
‘Concurrent User’ metrics are poor because:
- An iPad user gets bigger pages, whereas an Android gets lighter pages, that are quicker to build on the servers
- A Concurrent User “on the site” means nothing without saying WHAT they are doing
- These real users: are both 1 Concurrent User: and yet are widely different in their demands on your system
- hits one page, and then bounces away
- hits 28 pages in the course of buying 2 shirts!
The question should rather be:
“How many users can complete their various different kinds of user journey : per peak hour?”
Approach the slopes – plan Realism for your dynamic journeys
Having decided to put concurrent users as a metric behind us:
Now we ask ourselves: How can I be more Realistic?
Many companies have a load test approach that dates back years: when the use of tools to ‘Record and Playback’ your session on your PC: could be used to then call up 10,000 such ‘virtual users’ for a load simulation; which we now realise is a terrible way to load test!
In the real world there will never be 10,000 users all following the exact same steps in order! All navigating to the same pair of trousers online and adding the same size and colour to the basket!
Real users: well they do not all choose the same item, or take the same route, or use the same features. You need to know how your site really handles the load of customers behaving the way that customers do, not customers all taking one idealised journey.
The thinkTribe test engine differs from other approaches: in that it was designed from the bottom up for realism: it does not simply ‘replay’ a fixed sequence of URLs and claim that is a ‘Journey’ , but instead dynamically looks into the live page at each step to determine the link for the next page to follow, and allows choosing randomly from the options on the page: options available to the real users taking such a journey.
We call that Dynamic User Journeys
This means that the virtual users of the load test are able to behave, and interact with content, in the same way as real users; taking a much wider range of paths, finding different products to add to the basket: and so emulate stress conditions accurately when testing for stability. This method also enables technical teams to identify problems with systems that may only present themselves when certain combinations of options are selected.
Ascending The Peak
Multichannel / OmniChannel call it what your will: joined up retailing involves a whole new set of dimensions. Whether you are measuring the load of journeys with similar goals across different platforms, or whether you are taking into account journeys that include multiple devices that may not take place over one session and so influence the breakdown of traffic in the load thinkTribe has the understanding to help create effective tests for such complex scenarios and to discuss the implications with both business and technology teams.
A valuable help we have provided to a number of major national names in the UK: is helping an organisation’s load testing process to go beyond reliance on tests of an offline, small-version of the true website: only testing the true live site (albeit out of hours): tells you truly how large your online store really is.
Tribe’s managed service approach means that we work extremely closely with clients to understand all their needs from testing, and what it is they are looking to discover or resolve.
Our understanding of end-to-end web performance from the server to real user experience means both that our load tests are unique in the complexity “under the bonnet” and that our analysis and reports, all of which are written by our lead test engineer with no generic automated outputs, are intuitive, bespoke, and business focused. We do not just leave our client with the report, but provided follow up workshops and discussions to ensure they are getting the best intelligence they can from the data.