What Does It Really Cost to Fix a Software Defect?
Everybody knows that the later you find defects, the more expensive they are to fix. Everybody knows that, right?
The curve was modified and enhanced over the years, and a second exponential relationship was added to it with a smaller slope designated for “Smaller Software Projects.” Members of the software engineering profession further modified the diagram in ensuing years into other versions like pyramids and histograms.
Turns out though, according to Laurent Bossavit, author of The Leprechauns of Software Engineering, the “underlying evidence justifying Boehm’s curve…just isn’t up to any reasonable standard of ‘research.’” The few studies Boehm did reference were misrepresented, and one study even found a 2:1 ratio in the other direction from that claimed in Boehm’s curve.
This suggests that the mantra of rising-cost-of-defects may be more prescriptive than empirical or it may suffer under the “tyranny of always.” Bossavit says the
empirical basis appeared to fade into anecdote as soon as we looked at it a little closely, morphed over the latter decades of software engineering’s history into the fully generalized (but still no better supported) "cost-of-change curve.”
What’s happening here? How can an empirically unsupported idea take root in a community of intelligent people and no one call it out?
Information processing experts call it “confirmation bias.” Confirmation bias is a tendency of people to favor information that confirms their beliefs or hypotheses. So when we hear that defects are more expensive to fix the later they’re found, we don’t question whether that is actually true because we already believe it. We thus don’t care too much if authors distort studies to make them claim things that the studies themselves did not claim or if the studies were even credible at all.
This confirmation bias leads us to throw out the critical thinking needed to determine if the “average cost to fix one defect” metric, which is what we really have to figure out to get the data points for the Boehm curve, is really even a valid metric in the first place.
We can’t lose sight of the job we are to do as software testers. We are hired to analyze, to understand all the pieces, and to question everything. This includes our own biases. As a community we should stop repeating folklore we don’t know is true and understand that the burden of proof is on us.
