Providing Visibility into Testing Processes That Matter
Awhile ago I was running a test competition. Teams were evaluated on bugs filed and the reports they turned in. Most of the reports were dashboards: They’d show the number of test cases that ran, the number of bugs filed, the number planned, and so on.
I have no idea what 65 test cases out of 85 means. Nor do I know what 1 critical, 5 severe, 20 medium, and 15 low-severity bugs are. What were those critical and severe bugs, anyway? Would they be hard fixes or easy fixes?
We didn’t know.
The one-page dashboard report, designed for executives, didn’t really tell us. If our goal as customers of the report was to figure out what needs fixing, how close we are to shipping, or how much time we need to do additional testing, we don’t get any of those answers.
But there is another, more insidious risk to this sort of report: Testing becomes that thing that we have to do in order to get to production.
The Checkbox People
Management often sees testing as just a step in the process, a black box, that doesn’t have any visibility. All black boxes being equal, you might as well pay as little as possible for it.
We know that all black boxes are not equal. Lower-skilled testers are done more quickly because they don’t find the bugs that matter. Senior testers look like they are slowing down the process because they see more risk areas, and they do find bugs that matter.
When I write project reports, they don’t read like a dashboard. Instead, they start with a sentence or two that answers key customer questions. Below that, I list the worst of the open bugs and what I would do if we had more time. If the team thinks those bugs aren’t important and the “more time” risk investigation is low value, we’re done in the middle of page one. Below that, I describe our test approach and why we chose that approach.
My preference is to engage the stakeholders in the work. Having the key decision-makers in the room flips the conversation from accountability and governance to what we are working on and what we will be doing next. It can be a very powerful thing; it zooms in on the work, shrinking the expectation gap on schedule from months (“We want this done in Q1! What do you mean you can’t get it done in March?”) to a day or two.
Putting It All Together
Focus on the information you provide to senior management. Figure out what they need to know to make decisions, and provide that. That is very different from the metrics many teams provide, which tend to be focused on the process—the number of test cases run, and so forth. In object-oriented terms, those are the algorithm. Focus on the algorithm, and the report won’t answer customer questions. If you don’t answer customer questions, then you’re just a checkbox on the way to production.
