Detect Performance Issues in Production with Continuous Delivery

You can read about the benefits of continuous integration and delivery hundreds of times, but you never quite learn as well as when you experience something for yourself.

Continuous integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect different kinds of problems early.

Continuous delivery (CD) is the ability to get changes of any type (features, configuration changes, bug fixes, and experiments) into production, or into the hands of users, safely and quickly in a sustainable way. You can work toward a culture of continuous feedback loops, getting better and better from the feedback you get from different sources.

One important source of feedback—maybe the most important one—is what happens in production. In her book A Practical Guide to Testing in DevOps, Katrina Clokie talks about how testing can add value during development and in production. It’s amazing the amount of information you can get if you properly monitor and analyze the data.

After having read and experimented with performance testing CI and CD, I was able to work with a client that runs performance tests in production (A/B testing, monitoring, etc.) and see how they used it to solve a peculiar problem they faced.

The system performance appeared to have worsened according to some particular metrics they were monitoring. After taking a closer look, it turned out that a heavily used JavaScript file went from being about 100 KB to about 450 KB, yet no one had realized it before sending the changes to production. The issue could be seen thanks to testing in production—that is, constantly looking at and collecting metrics on all the user activity in the production environment, then using that information to evaluate the software’s quality.

Continuing with the analysis, they were able to identify the specific code merge that caused the problem. Observing the GitHub log, they could see all the files that were changed in that merge request; three specific changes appeared as potential culprits, and from there it was easy to discern which had generated the problem.

It would have been impossible to do this if it were not for the fact that the team follows a continuous delivery strategy where it releases small changes every day. If the team had released once a week, they would have had to analyze all the changes made that week, a much more complicated and time-consuming task.

Continuous delivery allows us not only to deliver value to the end-user more frequently and get feedback from them sooner, but also to obtain more precise feedback from different sources, thanks to observing the impact of every realized change while improving traceability.

Federico Toledo is presenting the session Challenges to Effective Performance Testing in CI at the Agile Testing Days USA 2018 conference, June 25–29 in Boston, MA.

• Login or Join to add your comment