Jelly Beans and Defect Classification: Different Strategies for Success
I recently attended a meeting discussing whether the next phase of a large IT system was ready for go-live. Given the quantity and severity of open defects published a week earlier, many participants in the meeting had reservations about releasing the system. The development vendor tried to reassure the gathered skeptics, explaining that nearly all the significant defects had been addressed.
Pressed to elaborate, the vendor clarified that several critical defects had recently been resolved, and a third of the functions with known defects had been postponed to a subsequent project phase. The rest of the known issues had been reviewed and downgraded from “show-stopper” to “acceptable” error levels. While these dispositions might be reasonable in some cases, I couldn’t help but think of jelly beans.
When there’s a bowl of jelly beans, some people grab a few at random, but most of us have favorites, and we aren’t too proud to go after them. While I’ll eat any jelly bean in a pinch (even the black ones), if I have the option, I’ll favor red and orange, like most people. I do this because I have a strategy.
If you watch the bowl over time, you will see the red and orange beans disappear pretty quickly. When they’re gone, my strategy kicks in. Most people have a hierarchy: They will cherry-pick beans in their order of preference, eventually stopping when their favorite colors are gone.
While I have preferences, I also know that I will eat any bean in the bowl. This means that if I want to maximize my bean consumption, I must have pretty low standards—or, more accurately, flexible standards.
I would never start a fresh bowl of jelly beans by seeking a black one. Among my friends and family, I’m one of the few who enjoy black jelly beans, so why waste an early-round draft pick on a jelly bean that will likely be one of the last left in the bowl? So, like most people, I start by going after the prized red beans. When red is exhausted, I change my criteria to orange.
When everyone has lowered their standards to consume most of the beans and black is all that is left, the one or two of us who enjoy licorice can now finish off the bowl. My fluid standards have maximized my jelly bean consumption, and this strategy means I maximized my opportunities to get my favorite beans.
Classifying defects should not be like choosing jelly beans; the standards shouldn’t be too flexible. At project inception there should be an earnest effort to define objective standards for defect classification, and an agreed-upon process for assigning defects to the appropriate category. Everyone should be rigorous and thoughtful about the severity assigned.
Sometimes, circumstances will change; mistakes will be made and defects will require reassignment. But these aren’t jelly beans we’re talking about. Any time a project team or vendor discovers right before go-live that a bunch of defects have been inappropriately categorized, it is probably a very bad sign, and your project team and user community are likely to end up with more than indigestion.
