Your Tools Are Not Your Tests
In "Musings on Test Design," Alan Page tells us that "automated testing" and "human testing" can't be considered in isolation from each other and that we need to take a holistic approach to our overall testing strategy. I think this applies not only to automation in the usual sense we think of it but also to test tools as a whole.
Testing tools are a mixed blessing. The obvious and important upside is that testing tools enable tests you wouldn't otherwise be able to do. Thinking of testing tools as a means to "hit fast-forward" on human testing is a mistake—they are more like binoculars or a microscope in that they extend our powers of observation.
The downside of our tools is that they tend to constrain our test thinking to the kind of testing they perform. This is true across all fields of endeavor, not just software testing, hence the common saying "When all you have is a hammer, everything looks like a nail." We have a tendency to try to fit the tests we do to the tools we have to do them. Unfortunately, the tools we have may or may not take us down the avenues we need to explore.
I think about this via this confessedly fanciful metaphor. Say we need to enter an abandoned industrial building. If we have a Geiger counter, we may find that the area is not radiologically contaminated, but that's a foolish basis for trudging merrily on in. There could be chemical hazards, exposed electrical wires, or unsafe structures that are prone to collapse. Knowing some of the history of the site would let us figure out what kind of assessment tools would be most useful and what approach we should take when using them.
The same goes for test design. We need to determine what the risks inherent in the product are and plan what we want to test—then see how we can use tools to enable that testing. We may need to find or build appropriate testing tools, or we may legitimately find that the tools we have are a good match to the risks we need to assess. In any case, the key point is to try to fit the tools to our testing—not the other way around.
