The Mismeasurement of Software
I’ve been involved with metrics in one way or another throughout my entire professional career. I’ve seen—and unfortunately been a part of—a number of unsuccessful metrics programs. Through this experience, I’ve identified and distilled four key principles that help prevent the mismeasurement of software. Upon consideration, each of these seems obvious and reasonable; yet I am amazed at the number of violations of these principles I have encountered.
Principle #1: Don’t measure something unless you know what it means. We measure many things, but most are actually surrogates for things we are really interested in. We are interested in program size, so we measure lines of code. We are interested in the thoroughness of our testing, so we measure the number of test cases. We are interested in product quality, so we measure the number of defects discovered.
Note that in each of these cases, the thing measured is a substitute for what we want to measure―not the thing itself. Why? Usually because the thing we want to measure has a complex definition, and the surrogate is easy to measure. Before you charge off and measure things, be sure you really know what they mean.
Principle #2: Don’t measure something unless you are going to use that measurement to control the object or process being measured. To do otherwise wastes your organization’s time, frustrates those who invested their time in the measurement, and generally shows your ignorance.
Principle #3: Don’t focus on measuring effort. Instead, focus on measuring results. This is related to the first principle. Effort is often much easier to measure than results, but what do we really care about? What are we trying to achieve? Results, of course. Testers love to count test cases planned, created, and executed. They love to report test cases passed and failed. What we’re really trying to measure is the thoroughness, effectiveness, and value of our testing. So, why the counts? They are easier.
Principle #4: Don’t turn your measurement into a goal. This is known as Goodhart’s Law, named after Charles Goodhart, former advisor to the Bank of England and emeritus professor at the London School of Economics. Goodhart states, “Once an indicator is made a target for the purpose of guiding policy, then it will lose the information content that originally made it useful.” In other words, “When a measure becomes a target, it ceases to be a good measure.”
How many times have we seen that? In the old days, it was lines of code―first measured as a way of sizing systems and predicting delivery dates. Yet it quickly became a target for developers to reach. Today, in the agile world, it’s velocity―the number of story points that can be developed per unit of time. At first, it was used to predict delivery dates. Now not only has it has become a goal for development teams to achieve, but managers also ratchet it up each sprint, trying to drive their teams to additional output.
Evaluate how your metrics work against these four principles. Do you need to make any changes?
Lee Copeland is presenting the tutorials Fundamental Test Design Techniques and Pairwise Testing Explained and the bonus session Speaking 101: Tips and Tricks at STARWEST, in Anaheim, CA, October 12–17, 2014.
