What Is the ROI of Test Automation?
Almost every test automation ROI calculator I have seen uses a simple formula: hours saved multiplied by an hourly rate. There is logic to this: If automation is displacing manual effort, then that produces savings. Right?
Wrong. Unfortunately, it’s not that simple.
The most obvious omission is the cost. Automation isn’t free. It requires tools—and specialized skills to use them. It takes time for manual testers to provide sufficiently detailed documentation to guide the automation engineers; for the engineers to develop, test, and maintain tests; and for manual testers to verify that the tests are executing as expected and to analyze any unexpected results.
Less obvious is the unspoken implication that any savings translate into reduced manual test resources. While this may appeal to managers struggling with budget constraints, the stark fact is that in thirty years of automation experience, I have never seen test resources reduced by automation. Ever.
Automation doesn’t reduce manual test resources because they aren’t testing everything to begin with. Keeping up with all the changes is a nightmare, but regression testing critical functionality for every release is just a dream. All testing organizations have to make trade-offs and cut corners in the effort to make sure they didn’t miss anything in order to meet the schedule.
To those managers who stubbornly insist that automation is about reducing testing budgets, I offer a foolproof, guaranteed way to cut costs: Stop testing! But that is ridiculous, right? Horrible things might happen. A single defect could cost the company more than the entire testing budget for the next decade if it results in loss of customers, damage to the brand, help desk meltdowns, engineering fire drills, and costly manual workarounds and delays. There are so many horror stories, there is no need to belabor this point; just read the paper or watch the news. Technical “glitches” have become a daily fact of life.
The truth is that to save money, you should test more.
However, automation can make manual testers more efficient. For example, automation can run environment checkout tests in the early morning hours to be sure everything is ready to go for manual testers before they arrive, so they don’t waste the first half of their day discovering issues and the second half waiting for them to be fixed. Automation can also create data for manual testers to use and reduce the effort involved in locating, conditioning, or creating the data needed for a test. And last but certainly not least, automation can make regression a reality. By filling in regression gaps and improving manual test efficiency, automation is absolutely a critical component of delivering a stable solution.
That is where the real savings come in. But of course, no one measures the cost of delivering unstable software, and it is impossible to measure the savings from something that didn’t happen.
So, where does this leave us? The original formula isn’t too far wrong, as long as we understand that it is telling us how much additional testing was performed through automation, and that the real return on investment is greater coverage. That could be worth more than we will ever know.
