Software Performance Tests: How to Know When to Stop

A common mistake during the performance testing process for a web application is not establishing clear, achievable, or realistic goals. Often, you’ll hear: “I want a fast response time application.”  “How fast?” “Extremely fast!”

These types of questions usually receive non-descript, vague answers that often lead to endless performance test cycles. These endless performance test cycles then add up to higher costs and delays with the project.

As part of the performance test process, we have to guide and support to the customer through the appropriate and accurate questions that can predict and represent the application behavior on the production environment. This guidance and support is important because most customers do not have the knowledge required to define the performance criteria of the web application.

To be successful in this process, it is best to write a simple questionnaire (you may want to look at the application capacity planning for some question ideas). The main objective of this questionnaire is to determine the most common scenarios the application will face.

For example:

(1) What are the most common action/set of actions that the users will do in the application? (It is best to try not to exceed five.)

(2) How many actions/set of actions will be performed in a day?

(3) What is the time range for a peak transaction load? (E.g. From 9:00 am to 11:00 am (two hours).)

(4) How many actions/set of actions will be performed during the peak transaction rage time? (It can be defined as percentage of an entire day as well.)

(5) What is the acceptable response time (define a range) for each action/set of actions? (Consider the system/application response time (as click, page load, etc.). E.g. from 1 - 3 seconds.)

(6) Does the performance test environment have same features as the production environment (CPU, Memory, Disk, and Network)? (Note: This is one of the most important items to consider.)

It is very useful and desirable to match this information and ratify it against the old application statistics (if you have one), or collect extra information from other parallel systems that can provide valuable data to support and complement the gathered information.

This information, plus an adequate and careful analysis and its interpretation, will generate measurable and achievable performance criteria that will help you define the most important feature: realistic goals.