Introduction to Load Testing - Lessons from Journalism

Load test planning and journalism have some important similarities. Both activities need to answer the same core questions: Who? What? When? Where? How?

Who?

How many users do we need to simulate?  This is a fundamental question when planning any load test, and one that is surprising difficult to answer.  This parameter is also referred to as the Design Load of the system. I'll leave an in-depth discussion on this topic for another post, but the short answer is that the simulated user load should consider:

• Current traffic
• Addition traffic created by the ongoing project - Are we adding a mobile site?  Better SEO?
• Organic growth - a commonly used growth rate is 20% per year for 3 years (or whatever the planned time horizon is before a replatform, redesign, or hardware upgrade)
• Traffic spikes – these short term bursts in traffic can be caused by a variety of events: ad campaign gone viral, media mention, Digg, etc.

What?

What aspects of the system are we trying to test?  Are we trying to simulate transaction processing, test the system integration points, or simply pressure-test content delivery?  Ultimately, the "what" question can be answered with a list of URL's, or a "script", that the simulated users will follow.  Consider including the following:

• High profile pages – prominent pages on the site that get a lot of traffic (think Home Page)
• High impact pages – strategically important pages in user the experience, capture or conversion process
• System integration points – are there pages that invoke back-end integration with other systems?  These are often performance weak spots.
• Transactions – are there transactional elements to the system?  Maybe capturing a ContactUs form or processing a shopping cart?

When?

If the system to be tested resides in a shared environment with other live production sites, consider scheduling the load testing for an off-peak time to minimize the impact to live traffic.

Where?

From where should we run the load tests?  This right answer here depends on what we said we wanted to test.

• Internal testing - load testing from inside of our own network is the best option when we want to test specific elements of the system, like just the database, or just your new Web Content Management (WCM) system.  Testing from an internal environment is also preferable if we are trying to establish precise benchmarks or do troubleshooting - it eliminates as many uncontrolled variables as possible.
• External testing - load testing from outside of the production environment provides more of a "real-world" simulation and tests the entire system.  Because external testing includes system elements like the network, load balancing hardware, and even our ISP, it gives us a more realistic picture of the response times our users will see.  Unfortunately, these additional uncontrolled variables also make it more difficult to establish precise benchmarks or to pinpoint bottlenecks.  With external testing, it is important to specify exactly where the load test software will be running, especially if we're defining technical performance requirements.  For a site hosted in the U.S., network latency will be smaller from the Bay Area compared to Manila.

How?

If we've answered the first four questions, we're probably in a good position to start selecting the right load testing tool.  Here are some additional considerations to think about before we begin the hunt:

• What is our budget?  (think in terms of time, effort and money)
• Are we looking for a tool for one-time testing or to test on an ongoing basis? (load testing monthly releases)
• Who will be using the tool?  Some tools are intuitive enough to be used by any semi-technical individual; others require extensive training and/or developer skills
• What types of analytics are we looking for?

Just the Facts

One final point of similarity between load testing and journalism is that both are most effective when focused on the most important facts.  When answering the five questions above, it's crucial to focus intensely on just the most important functionality and pages.  Without such a focus, load testing can become a black hole.  It's very easy to go overboard and spend an endless amount of time and energy testing countless combinations and permutations of scenarios, pages, and benchmarks.  Stick to the most critical points and time-box your load testing effort to avoid the black hole.