How to Create a Reliable Baseline for Measuring Your App’s Performance
A baseline test measures an application’s performance and tells you whether it meets the service level agreements before it goes live to customers. One of the most challenging exercises in creating a baseline is making the test reliable.
Organizations that are aware of the importance and significance of performance testing understand that performance baselining is a basic requirement for a product’s or platform’s continuous performance analysis and monitoring. However, the results of the baseline test often can vary on the same piece of code. This means the baseline isn’t reliable and defeats the entire purpose of the test.
Finding and resolving the cause for the variations in the results take a lot of time, so creating a usable baseline can be expensive. What can you do to make sure you develop a reliable performance baseline?
Jotting down the variable factors that may change test results is the key. The state and configuration parameters for your test environment, application environment, load test scripts, network, and other factors that interact with your load testing environment can help you define a control state for running load tests. A baseline test can then be reliably run under the control state.
Some common factors that contribute to test result variation are CPU, memory, disk space availability, and network contention. Simple checks—like ensuring that the load test environment and the application environment machines have the same configuration during all tests, no applications or browser windows are open on the environment machines, and sufficient disk space is available—take care of issues resulting from CPU, memory, and space availability.
Network contention issues can be avoided by using a dedicated network for running all tests. Any domain name system or firewall changes should be closely examined.
In big organizations, the test start and stop times are crucial and may be a factor in the test result variation. The test results may vary when load tests overlap with certain planned activities on the application environment such as cron jobs, database backups, content loading, or indexing.
Although it’s necessary to perform load tests during such planned activities, it is not mandatory to run baseline tests then. You can choose to create multiple baseline tests and ensure that every planned activity runs during one of them.
If you plan to create a single baseline, carefully pick and choose the planned activities that should overlap with your baseline. For instance, if the database backup occurs at midnight and the product usage is low at midnight, you may choose to ignore it. Complete knowledge of the product enables you to consider and include such planned activities in your baseline tests and to define a control state for running them.
Paying attention to conditions and carefully planning when and how you test can make all the difference in creating a reliable performance baseline.
