Embedding Performance Engineering into Continuous Integration and Delivery

In the world of continuous integration and continuous delivery (CI/CD), where technology companies are rapidly deploying code and infrastructure changes so their application can gain a competitive advantage, the importance of ensuring good performance has increased tremendously. While functional and unit testing are relatively easier to integrate into these processes, performance engineering has typically raised more challenges, especially in analyzing the results and pass/fail decision-making.

My company is currently going through this transformation, and embedding performance evaluation of the services as part of CI/CD is a must. Here are some of the challenges we faced and changes we made across the board to make this happen.

Cultural challenges: If you want to evaluate the performance of every line of code that is checked in, culture is the most important yet difficult change required. There were several areas where we struggled. Performance was an afterthought; teams were focused on functionality more than performance. There was also a lack of understanding of performance and scalability needs of the product, and performance engineers were not part of the agile teams and weren’t included in the agile ceremonies.

To help alleviate these challenges, we needed to shift the performance lifecycle to the left. We empowered our developers to test, allowing good performance to be baked into the code.

Process changes: We also had to make significant changes in our Scrum process to create awareness of performance. We made nonfunctional and performance requirements part of the functional requirements, which required us to have scalability needs, API contracts, and service-level agreements clearly defined at the story level. This enabled us to learn what “performance-ready” services mean to our customers.

We mandated performance as part of the definition of “done” for sprints and included performance results during sprint demos with all stakeholders.

Tools and accelerators: At this point we had to make sure we had the right tools and accelerators to create, execute, and analyze performance tests at the sprint level, ready to be integrated into the CI/CD pipeline.

We built a performance engineering platform that manages the testing cycle of our services, leveraged functional test scripts to collect browser-side response time, created self-contained tests that create test data before each test and destroy it after each test, developed a central repository for all performance metrics, created an algorithm for dynamic thresholds for pass/fail of each build and individual REST APIs for test status and decision-making, and automated defect creations and notifications through our performance engineering platform.

All our services are in the cloud, making it easier for the performance environment to expand and contract as needed and adding flexibility to build and kill new environments. Such benefits can go unnoticed, but they are a huge driver to implement our test framework that can scale to address business needs.

Anjeneya Dubey is presenting the session Embedding Performance Engineering into the CI/CD Pipeline at the STAREAST 2018 conference, April 29–May 4 in Orlando, FL.