Improving Your Measurements to Influence Behavior
Measuring things is easy, but measuring the right things and acting appropriately based on what you have measured is hard. What you do with measurements can influence behavior, so it’s worth putting some thought into what you measure.
When you gather and apply metrics correctly, the information can help you improve. Using metrics poorly can frustrate people on the team and potentially reduce quality by encouraging people to optimize the wrong things.
Part of the reason metrics are hard to work with is that it’s difficult to know what you are actually measuring. Cem Kaner explains that a seemingly simple question such as whether you are measuring a direct or derived metric is more complex than it appears. Proposing to measure “lines of code,” for example, raises questions such as “what is a line?” and “what is code?”
In "Getting What You Measure," Eric Bouwers and others discuss some common pitfalls of measurement and emphasize the importance of evaluating metrics in context. For example, the number of lines of code in a project tells you little without considering other information. When lines of code grow over time, it could mean anything from lots of good features being added to lots of code duplication.
The core problem is that the things we really care about by are difficult to measure directly, so we measure proxies and attempt to derive information about the thing we care about. It’s hard to imagine a situation, for example, where you care about “lines of code.” You might care about productivity, quality, or something that reflects business value. Sometimes the information that’s easy to collect can be useful. Sometimes we’re just being lazy.
An analogous situation in the education world illustrates this. When faced with the question of how to identify students who needed help with their writing, teachers were at a loss to know what to do:
The harder they looked, the teachers began to realize, the harder it was to determine whether the students were smart or not—the tools they had to express their thoughts were so limited that such a judgment was nearly impossible.
The tests that were available were measuring things like basic reading ability and vocabulary, which did not help teachers understand the real problem. In this case a teacher devised a test that measured the students’ grammar skills, which led to a path to improvement.
Using metrics effectively is challenging. It’s easy to be impressed by the quantity of data and easy to react to numbers. Unless you evaluate the numbers in the context of a goal, you won’t learn what you want to know. You may even influence teams to work in a way that may take you away from where you want to be.
In the face of this, it might be tempting for you to avoid measurement out of fear that it will lead you astray. This can be counterproductive, as without data you can’t improve. If you keep some things in mind when you start using metrics, you can avoid many of the pitfalls of badly applied metrics badly applied:
- Understand limits of your metric.
- Seek to understand the underlying causes for the results you are seeing.
- Evaluate metrics in context. Consider changes over time, the organizational context, and the history of the team.
- Start off by thinking about what you want to measure. Analyzing statistics without a goal can be interesting, but taking actions without a sense of what you are trying to measure can lead you astray.
- Remember that metrics can clue you in to aspects of a project that you need to look at more closely, but few, if any, metrics, stand on their own outside of context.
Metrics can be a useful signal that you may need to act, but metrics are only one part of the improvement process. You need analysis and understanding.
