A Holistic View of Test Coverage
Test coverage is hot lava: No one wants to touch it. Or at least no one wants to until it is time to push to production, or we discover we missed a bug—then everyone wants to know how much testing is left, and how that bug could have possibly made it out the door.
The problem is that no single person on the team ever knows much about test coverage at a high level. Developers might understand coverage for the parts of the code base they worked on. Testers might understand coverage for the last handful of features they tested. But generally neither is able to talk about coverage in a meaningful way.
You need a holistic view to find an answer to these questions. No amount of exploratory testing will save bad code, so my version of a holistic view begins with the code.
I started by creating inventories. Each of our code repositories has at least one directory where tests are stored, so I took a look in these repos to analyze coverage and how often changes were happening. Areas of code that change frequently and don’t have associated test changes were good places to start asking questions. So were test directories with a minimal number of files.
We started using code coverage tools to help us understand coverage in code repositories where coverage felt just okay. These tools don’t tell you about the quality of your testing, but they do show trends and where coverage is lacking.
After we have an understanding of code quality, I like to talk to the testers. My first question is usually whether they can even talk about test coverage. This is an important skill that most testers lack. Just like with code coverage, one particular type of test coverage alone doesn't explain much about the testing you have done.
Exploratory testing coverage should be layered—features, configuration, variables, pages, workflows, and so on—to show a more complete picture of coverage. You can do this by creating some light documentation with a mind map, test charters, or a whiteboard diagram. But without some sort of reference point, it’s difficult to tell the story of your test coverage.
Testing by developers works in tandem with exploration by someone in a testing role—one informs the other. Testers debriefing with developers on the automation that was built suggests the areas to explore. Bugs found during exploration will suggest areas of the code base that lack coverage, have inappropriate coverage, or have bad tests.
Good testing is layered, and so the conversation about test coverage should reflect those layers. Test coverage is a story of your tech stack and how approachable each part of it is, as well as the different ways you can inventory your product. Having a test coverage conversation that delves through the layers helps maintain a holistic view and keeps that coverage model alive. It also makes those important conversations around missed bugs much easier to have.
The title of this article is misleading.
It should be "Test coverage in Exploratory Testing".