Monitoring Holiday websites – not easy given the caching bashing
Looking at the web performance of eCommerce sites, there are some things that make the Holiday sector unique. We work with a number of holiday companies and they have a unique challenge when it comes to accurate website monitoring.
When a holiday season nears its end, holiday companies are often presented with an increasing number of failed ‘user journeys’ that occur when a holiday is found by the visitor but is actually already sold out. And these failed journeys need to be identified and quantified, if they are to enable an accurate reflection of lost sales.
But first, let’s look at how the typical non-holiday site handles things.
To sell tangible products online, the website needs to link back to the warehouse engine, so that it knows what is in stock and what is not. If you’re selling fashion products – you’ll log down to which colours and which sizes are available for each SKU.
Many eCommerce platforms will hide products that are not in stock – to avoid the user being disappointed after searching and finding something they want, and then being told they can’t buy it after all. It’s important to build site confidence among the user base by ensuring they can buy what they like especially when the shopper is comparing suppliers.
Some websites don’t hide out of stock products. They flag them with a helpful offer – asking if you want to be emailed when they are next in stock. So far, so good!
But back to the holiday sector – a holiday website will have a relatively small number of items for sale, per package per week. Maybe as low as say 50 for a specialist company. And crucially once they are sold, there is usually no way to make any more available. So products are never too far away from being sold out.
Plus, the holiday package website is mixing and matching its holidays – based on flights from maybe external databases, or hotel rooms from a different internal database. It’s a much more complex back-end dataset than selling products.
End result – it takes a lot of computer time behind the scenes whenever a visitor to the website does a search.
And the final ROI problem – some of those external database searches cost real money – per search! If the flight databases you use make their revenue by charging per search, then to keep your costs down, you don’t want to do a fresh search for every single visitor on your site who searches – you want to re-use searches where you can.
Caching rescues web performance
So as a holiday company, you can’t afford to overspend on hardware, but you do want to offer a fast enough response to visitors even during the daily and weekly peak busy periods.
This is where CACHING comes into play.
For example, speaking to a company today, they do nightly cache-building batch jobs which calculate a matrix of the bulk of available and/or most searched packages: which is then the Cache, to be used all day when people search for holidays. It’s a slow, database heavy process – so doing it overnight in low usage time slots is good.
During the day, as a result of the caching, users get nice fast searches and the server power needed is lower – exactly as desired.
The Syndrome: The product you found by searching – is not available to Buy.
But the downside is the dilemma of Stale Caches. This is a nasty user web experience syndrome where the exact thing the visitor found on your site and wanted, when they click the Buy button, is not available after all. Because it sold out since the cache was last refreshed.
This is a particularly nasty experience in the holiday space because the visitor might have spent quite some time looking at the details of three resorts before trying to buy one – they’ve spent time looking at photos, reading the text descriptions, checking out the extra package options, etc – so it’s very frustrating to feel that the site ’should have told me before’ that that package was not available.
At the start of the holiday season – no one package sells out during any one day, so the chance is very low of hitting the stale cache problem.
But later on, as the popular packages are sold, there comes a point when there are only a few places left, and those places are sold during one final day. So anyone who finds those packages later on before the overnight re-cache, if they like them and try to buy, they’ll be turned down.
Web Monitoring when the Journey doesn’t always work!
But the interesting discussion today, was how can the client can get his money-making User Journeys monitored 24/7 – if they don’t always work! If occasionally a holiday found by searching online can’t be bought – the Journey fails, even though the site is performing exactly as desired. And they don’t want to receive Error alerts, for something they can’t fix! So a more dynamic way to monitor the website was the keynote at the top of this client’s reason for moving his website monitoring and load testing to us.
Test planning wise, they had done good work, and specified a number of sensible multiple page routes through the site – using random choices of resort / dates / holiday types / airports and so on – so that our web monitoring service would be doing a good representation of Do what the Customer Does.
Our Dynamic User Journey monitoring service is able to handle such journeys quite happily, in comparison to some other simple, static URL list approaches.
So it was just the issue of the stale caches that we needed to agree with the client.
The normal approach we take to website monitoring of holiday websites was explained – that on hitting a stale cache, our Journey would dynamically loop back, and run again for a different set of randomly selected dates/destinations, etc.
All we needed to help the client with, was for them to work out how many times our Journey should loop and fail – before it should be called an Error.
Obviously, they didn’t want to mask out and ignore genuine website problems – it is possible to imagine software bugs where all or many packages found can then not be found – they definitely want error warnings and Alerts by SMS 24/7 if that should happen!
There’s no right answer, it depends in our experience on each holiday websites testing needs and architecture.
High Season Low Season Web monitoring season
In this case, the client wanted 2 loop-backs, i.e. if the 2nd fails then call it an Error.
But, they decided that when it approaches the season end, they’d want to ratchet that up to 4 times, because statistically it will happen more often. No problem, we can adjust that setting during the year for them.
In fact it ties in with the use of the Lost Sales Calculation in our monitor engine – as a holiday website with strong seasonal patterns, the typical sales per week vary widely during the year. But our portal allows them to dial in themselves their typical sales figures (either sales revenue per hour – or else passengers per hour,etc), and change them at will.
That way the Lost sales figures in their monthly reports from us will be much more accurately in line with actual lost sales figures – which of course as the web monitoring company we are not privy to: but we can calculate and plot estimated Lost sales by weighting the hourly typical sales with the actual brief outages, blips and slow downs that we measure from our web performance user journeys.
So all on all, holiday ecommerce are some of the most interesting to test – I’ve not even mentioned this time Load Testing holiday websites: as well as having some of the most interesting technology, with clever caching engines behind the scenes.
And as the caching engines on each site evolve and are improved, dynamic website monitoring Journeys come into their own – as they graphically show the effect of caching changes, and of forced cache flushes, and of hardware changes to cache servers – it’s s game of chasing web performance that will keep us all interested for some time, I hope!