Why You Should Build a Custom Performance Testing Framework

With the evolution from desktop to web and now cloud, performance testing involves an unparalleled combination of different workloads and technologies—which translates to a large set of variables for a performance test. There is no silver bullet, either commercial or open source, that meets all scale testing needs.

Some tools act as load generators; others only monitor system resources; and many only operate for specific applications or environments. Creating a custom simulation framework provides the capability to execute varied performance tests and have the flexibility of doing tests at any stage of the product development cycle.

Components of a holistic framework include a controller, data providers, load generators, and monitors.

• Controllers are effectively the brain behind the framework, where the workload models, percentage distribution, and number of virtual users are configured.
• Data providers feed the test data to the controller.
• Load generators are agents that execute simulated test threads to hit the application under test. The number of load generators can easily be scaled depending on the test planned. It could be a straight correlation of a single agent capable of simulating a hundred transactions per second, leveraging the number of agents depending on the scale of the test.
• Monitors record system utilzation—CPU, memory, network load, etc.—and report the results of the tests.

The level of control and flexibility through this approach is far greater than the offerings of any open source tool, and at the same time provides enormous savings when compared to commercial off-the-shelf options. Architecting and coding your own framework helps to mock services when required and could be plugged to test the product performance at any stage of the development cycle.

Designing and developing a framework gives flexibility to developers, DevOps, and architects. They can simulate a workload at any granular code level that can further be used to integrate with the continuous integration pipeline. The monitoring dashboard can compare the performance deviation over a period, helping project and program managers have insights to make data-driven decisions.

Customer experience is a top priority for any product under development, and the application's responsiveness is one of the foremost influencers of a good customer experience. In such a scenario responsibility for scaling not only lies on the performance engineer but also equally on all stakeholders of a project.

Prashant Suri is presenting the session Build Your Custom Performance Testing Framework at STARWEST, from October 12–17, 2014.