What is a Load Test?

In the simplest terms a load test measures how much activity your site can support before performance is affected.

How Does A Load Test Work?

A load test engine uses scripts to simulate different numbers and configurations of users accessing your site and applications.

Is There A Difference Between Load Testing And Stress Testing?

Both terms refer to the process of putting demand on a system or device and measuring its response.

Load testing is performed to determine a system’s behaviour under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation.

When the load placed on the system is raised beyond normal usage patterns, in order to test the system’s response at unusually high or peak loads, it is sometimes known as stress testing. The load is usually so great that error conditions are the expected result, although no clear boundary exists when an activity ceases to be a load test and becomes a stress test.

Why Is Load Testing Needed?

Load testing identifies at what levels of traffic, or certain types of activity, user experience starts to degrade.

Knowing this in advance you can take steps to prevent this occurring at a crucial time. and removing the risk of impact on the bottom line through loss of sales, user dissatisfaction and brand damage.

Load Testing data is also helpful in making prioritisation and budgeting decisions.

Does Load Testing Measure Hardware Or Software Performance?

Load Testing Identifies problems, the conditions (traffic levels, activity mix, actions taken) under which they occurred, the errors thrown and their root causes. The root causes identified can be hardware or software related.

A complex test sequence helps uncover route causes that are otherwise difficult to find. For example high simultaneity runs simulate many virtual users following a journey and then stopping before hitting a defined page concurrently. These tests identify a range of race conditions – software coding problems that cause a percent of users to see error pages, but don’t typically show in UAT testing.

How Do I Know If I Need To Load Test?

In general it’s good practice to Load Test your site regularly to check for any unknown problems that may have arisen due to installation of new hardware or software.

Load Testing should not be thought of as only something you do once your site increases by a given amount, or in preparation for any particular activity (although both are good reasons for wanting to test). You need to Load Test to understand the limits of your site, and the location of any potential “weak spots” in order to make decisions on whether developments are needed or not.

If you do notice that your site has started to struggle in a way not seen previously then that is a sign that an exploratory load test is needed.

What Makes An Effective Load Test?

For an effective load test it is important to cover not only a mixture of different journeys, but to weight the number of each type of journey within the test by the proportional breakdown of those journeys in different time periods, reflecting the ways the site must be able to handle different kinds of traffic peaks.

What Can I Learn From A Load Test?

This really depends on the type of testing you undertake and can range from the basic self service tool results that provide “one big number” of “the site slows down when X amount of traffic hits the homepage concurrently”, to in-depth bespoke analysis.

With sites becoming increasingly complex and relying on the integration of many different systems the identification of any bottlenecks require in depth analysis and the correlation of many different sources of data. The flexibility of sites has also led to increased sophistication in user behaviour as there are many more ways for customers to interact with sites. This can also make testing more challenging where an organisation is determined to ensure that all these different types of users and usage patterns have been understood and optimised.

When you are looking for breaking points the important numbers are the ones about failures, not the ones about success. You need to know if a different proportional mix of journey types has an impact on results, and when and where that might be a problem.

Is Load Testing Just A Measure Of Traffic?

Simply, no. An actionable load test is a measure of entire site performance and that is about more than simple traffic count. You may find, for example, that site performance suffers more with a relatively low number of users if they all use one particular feature, or that there are significant differences based on user technology or point of origin.

If you are using a cloud based system (see below) there may be specific considerations to test as capacity is not entirely under your control.

Do I Need To Measure Concurrent Users?

Concurrent User numbers alone are not really very useful. In fact they are awful as a measure of your web-site’s capacity! See Predicting Next Years Website Traffic

In order to identify why performance suffers at a given level of users you need to know what those users were doing – not just that they were ‘on the website’. It is also important to measure how many users were able to complete the task or user journey they began, not just how many might be sitting stuck in a loop or got thrown out.

What Should I Be Load Testing?

Rather than simply hitting isolated pages. You should be looking at a number of test variations to replicate real world conditions including:

• Modelling ‘drop off ratios’ (the ratio of users that drop out during a journey) rather than testing loads where 100% of users starting a Journey follow it through to the end.
• Testing a mixture of Dynamic User Journeys in line with expected load ratios such as 20% checkouts, 30% add to baskets, 40% product searches and so on.
• Checking that page content is as expected and that the link required to move to the next step in the Journey is functioning – rather than simply checking page headers.

What Needs To Be Done Before Load Testing?

You will need to analyse past and current performance data to work out the parameters of the load that needs to be generated for the tests, whether testing based on current traffic or projections for the future.

It is also a good idea to benchmark performance for a number of weeks beforehand,  for example by 24/7 monitoring of User Journeys, taking measurements of journey completion time every 5 minutes which will provide context for the test results..

When Should Load Testing Be Carried Out?

If testing a live site many organisations choose to run tests outside of peak hours (peaks may be measured in terms of traffic numbers, volume or value or sales etc) so as to reduce the risk of inconveniencing real users on the site.

When testing in a dev environment, it can be planned into the schedule by the project manager.

If you are testing ahead of a significant event such as a launch, a marketing campaign, a sale or an expected seasonal peak always be sure to leave adequate development time after the test to fix any issues discovered and re-test.

Should I Run Load Tests With Different System Configurations And At Different Times For Comparison Purposes?

A mixture of tests is always useful. This will give you a better understanding of the limits of your site under different conditions and enable you to plan the redundancy you need in your system.

The flexibility and multi functionality of sites can make load testing more challenging for organisations determined to ensure that all different types of users and usage patterns have been understood and optimised.

Session lengths, the weight of pages and the speed of processes involved and load for these various kinds of journeys and the systems they run on will most likely differ greatly. In addition to that the proportion of the journey types will likely change at different times of the day or periods of the year.

For an effective load test it is important to cover not only a mixture of different journeys, but to weight the number of each type of journey within the test by the proportional breakdown of those journeys in different time periods, reflecting the ways the site must be able to handle different kinds of traffic peaks.

How Complex Can A Load Test Be?

The complexity of the test parameters possible depends on the sophistication of the test engine that is being used.

How Often Should I Load Test?

Many organisation have a plan that includes both regular load testing and benchmarking to ensure that changes made to the platform do not result in any unforeseen incremental performance issues, coupled with a number of in depth ad hoc tests that relate to activities such a major launches or sales and marketing activities.

What Is Included In A Load Test?

The tests themselves can involve complex user journeys of many separate dynamic steps and top level measures including:

• The number of users that can complete a given journey per minute within the expected mix of total journeys.
• Realistic number of user journeys per hour based on projections from web analytics about how users might “bunch” during different periods.
• Max pages per hour with further detailed breakdown to reflect the fact that not all pages are equal in the load they cause a system, or how they are called on as a proportion of page visits for a given time period and type of traffic peak

How Can I Be Sure My Load Test Results Will Hold True For Real Usage?

In order be sure that your test result reflect what happens in the real world rather than an ideal dev environment the tests must be as close to real life scenarios as possible.

You need to build a representative set of user journeys for your tests by collating information from benchmarked 24×7 monitoring, web analytics usage stats, and other relevant systems to generate the parameters for your loads that will deliver more precise insights than can be achieved with simpler traditional or self service methods

Is There A Way To Target Load Tests To Already Known Problem Areas?

Working with multiple data sources and correlating the information across all of them will enables the load test engineer to identify likely problem areas. In turn this means that tests can be devised to specifically explore those potential problem areas as a complement to the known mix of user journeys under investigation.

How Many Tests Should I Run?

Typically at least two rounds of testing take place within a load test project. The first identifies any problem areas and the second is a re-run using the same parameters to measure improvement. Depending on the work to be done more iterations may be needed.

Working in this way means that developers can focus on designing and implementing changes resulting from initial findings before re-running the test to see the results.

What Is The Difference Between Self Service Tools And Fully Managed Load Testing?

Self service tools tend to concentrate on basic metrics such as concurrent user sessions, page views, or technology driven performance measures. These numbers can work for benchmarking but are not able to provide the level of insight needed to support the complexity of many sites now.

Fully Managed Load Testing ensures realistic testing through the use of Dynamic User Journeys that really “Do What The Customer Does” and involves specialist load test engineers working closely with stakeholders from across an entire organisation to understand and explore problems in line with strategic objectives, producing bespoke reporting and often delivering follow up workshops with the local teams.

Are There Special Considerations For Load Testing Websites Using a CDN Or Based In The Cloud?

There are a number of pitfalls it is possible to encounter when testing with a site that uses a CDN (Content Distribution Network) and it pays to be aware of them.

1. Testing as there is no CDN in use. At first glance this seems sensible: a good load test will be a good load test no matter how the target web site is set up. But when a site is using a CDN there are actually a number of extra complexities that need to be planned for. If you test out of hours, to avoid inconveniencing the users of your own website then chances are that this means it is out of hours for your CDN provider. If you’re unlucky, you’ll hit a time when they are doing nightly backups or other background tasks and thus you’ll see an unusually high error rate! Conversely, as it is out of hours for all their other clients too, the CDN will have huge swathes of free capacity, all available to you on your one load test. So you’re likely to find the CDN delivered parts of your website perform pretty well!
2. Testing your website without the CDN elements. If you’re buying your load test from a supplier who charges based on the amount of traffic you generate then that consideration alone means you may think about testing the website without the CDN elements. This approach may not be 100% wrong, but it can easily lead to load test specs that are no longer realistic in terms of user experience. Worse, depending on how your pages are constructed, it may be easy or very difficult to separate out the CDN.
3. Believing the cloud/CDN is so big and reliable no testing is needed. In the real world, of course, Cloud systems have bugs and problems, CDNs have less than perfect load distribution models and human errors mean your website performance may not always be as well supported as you think you’re paying for.

Websites underpinned by Content Distribution Networks, or built with their systems half inside the Cloud, are still web based applications.

To test their capacity meaningfully, to get performance measurements that are valid in the real world, can be achieved, but needs a little more thought and care in the planning, and a little more subtle power in your script running technology.

Is Multi Channel Load Testing Possible?

Multichannel involves a whole new set of dimensions including measuring the load of journeys with similar goals across different platforms,  taking into account journeys that include multiple devices that may not take place over one session and so influence the breakdown of traffic in the load but it is possible to create effective tests for such complex scenarios and uncover the implications for both business and technology teams.

Is Load Testing Used For Marketing Campaign Planning And Measurement?

Companies are usually dismayed and surprised when a marketing campaign does not produce the online results expected – one contributing factor could be that your website is unable to handle the volume of traffic expected. Don’t lose sales, ensure that your site is ready for your online marketing campaigns.

User Journey load testing provides hard facts about your website’s ability to support traffic and can be easily interpreted by all departments – protecting technical departments from blame when results don’t meet expectations.

Is Load Testing Needed For Capacity Planning?

Capacity planners rely on load test data but do not tend to run the load testing themselves. The information from load testing is needed for complex modelling and forecasting of capacity needs.

Is Load Testing Data Used In Strategic Decision Making?

As eCommerce continues to rise in strategic importance the data available from load test results can provide a vital input into strategic planning, target setting of all kinds, and forecasting.

Business teams, or even the board, need to be able to make plans with genuine confidence that the sites tested will not be restricted by ‘technical issues’ at peak, only by the level of traffic marketing campaigns are able to generate and present to the platform.

Is Load Testing Used In User Experience Measurement?

Load testing is used to ensure that your users experience a consistent level of performance whenever they visit the your site. A user does not know or care if they are the only person on your site at the time of their visit, or one of ten million. They expect a perfect customer experience every time.

A negative user satisfaction test, or low scoring sentiment survey data, can often be the sign that further investigation is needed through load testing.