The Testing Practices and Metrics That Really Matter in Agile and DevOps
Scaled agile and DevOps change the game for software testing. It’s not just a matter of accelerating testing; it’s also about fundamentally altering the way we measure quality.
For agile, we need to test faster and earlier. But DevOps demands a more deep-seated shift. The test outcomes required to drive a fully automated release pipeline are dramatically different from the ones most teams measure today.
To help software testing professionals understand what’s really needed to navigate this shift, Forrester recently conducted research on what quality practices and metrics separate agile and DevOps leaders from laggards. The study found that companies with the most successful agile and DevOps initiatives do a number of things differently from their peers.
For one, they are transforming their software testing into continuous testing by automating end-to-end functional testing, integrating testers into cross-functional teams, and more modern practices. Agile and DevOps leaders are also almost twice as likely to consider automating the software quality process to be a critical business differentiator, and they are significantly more likely to have high levels of automation for key testing and QA processes such as test case design, functional test automation, and test data management.
Risk was a prevalent thread throughout the various parts of the report. Although risk-related metrics did not rank high in overall popularity, agile and DevOps experts measured them significantly more frequently than the nonexperts did. In fact, agile and DevOps experts consistently ranked risk-related metrics among their top three most valuable metrics in different phases and categories.
Interestingly, four out of five firms believed they deliver within acceptable business risk, but fewer than a quarter stated that their QA and testing processes completely cover business risk. Only 15 percent of respondents said their test suites reliably provide a good indication of business risk.
From what we've seen of Global 2000 companies’ test case portfolios, we’ve found that most test suites cover about 40 percent of the organization’s business risks. However, these test suites have a 67 percent level of redundancy on average—meaning that two-thirds of the tests don’t actually increase business risk coverage. This might be one cause of the disconnect: People assume that more tests means more business risk coverage, but that’s often not the case.
Another likely cause is that most test results focus on pass-fail status, which doesn’t provide the required insight into whether the release has an acceptable level of risk. As companies move towards continuous delivery and automated release processes, real-time insight into business risk becomes critical for promoting acceptable releases as rapidly as possible, while stopping potentially damaging ones from proceeding down the release pipeline.
