The Problem with Software Measurement and Metrics
Many well-known laws are named for their discoverers: Avogadro’s law, Marconi’s law, Newton’s laws, Ohm’s law—and, of course, Murphy’s law. However, most important to those of us interested in software metrics is the lesser known Goodhart’s law. It was proposed in 1975 by Charles Goodhart, an adviser to the Bank of England and later a professor at the London School of Economics.
Goodhart’s law states that “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
What that means is that when a measure becomes a target, it ceases to be a good measure. It will be the inclination, and, in fact, may be in people’s best interest for them to game the system by changing their behavior in such a way as to favorably adjust the measure in order to achieve the target.
Goodhart developed the law to describe federal economic regulations, but it’s easy to see how it applies to the world of software. A classic example is the metric of lines of code (LOC) written per day. When used as a measure, this metric can be valuable for estimating, planning, and tracking. However, when LOC per day becomes a goal, developers may be incented to write more lines of (less efficient) code simply to meet the target.
If LOC is too old-school of an idea, consider velocity, a premier agile metric, measured in the number of story points that can be developed per unit of time. At first, it was used to predict delivery dates. Now, not only has it has become a goal for development teams to achieve, but managers also ratchet it up each sprint, trying to drive their teams to additional output. In many organizations, velocity targets are set and teams are expected to meet them—which they do, of course, by gaming the metric.
The same can happen by measuring stories completed. Setting goals encourages story “deflation.” As the size of individual stories shrink, more stories are created to describe the same system, and thus, the “stories completed” metric goes up in order to meet the “target.”
This reminds me of the old Smith & Dale vaudeville routine: A person goes to the doctor, raises his arm above his head, and says, “Doctor, it hurts when I do this,” whereupon the doctor replies, “Don’t do that.” Turning measures into targets will hurt also. So, as Goodhart advises, don’t do that.
