Reasons to Consider Software Tests as Products
With DevOps, automated tests have become a crucial necessity. Tests need to be thorough, and their automation should be stable. In fact, tests have to meet quality and robustness criteria that are similar to the application under test, but tests seldom get the attention and investments that the applications get. Where sources and components of applications are considered products that are designed and developed, tests play a mere supporting role. In Scrum projects you will not see tests specified in the backlog. Rather, they are seen as a part of the production for the user stories.
There are some valid reasons to consider tests as products. Well-defined tests can help quality and velocity. In particular, test design is a major driver for automation success, something I frequently talk about in my tutorials at the STAR testing conferences. Tests have qualities like correctness, completeness, and efficiency that can be planned for and reviewed when done.
In our projects, we follow a modularized approach called Action Based Testing (ABT). Tests are organized into test modules that we manage as the products of test development. The test modules, in turn, contain objectives and test cases to match them. The test cases are written as a sequence of actions, spreadsheet format lines each with an action keyword and zero or more arguments.
To get things right, ABT defines an important initial task called high level test design, in which the test modules are identified. Compare it to a table of contents in a book: you identify chapters in an early stage, which you then fill with text later on. If the scope of a chapter is clear, it is relatively easy to understand what the text should be. The high level test design is not meant as a framework to organize tests. It can change over time when tests become clearer.
An example of important criteria when defining and creating test modules is to keep tests for the business separate from tests for the application interaction. A business test might have actions like rent car and check invoice total but should never do things like checking whether clicking the submit button shows a confirmation dialog. Such low-level details can make tests vulnerable for application changes and can make them hard to read, in particular for outsiders like domain experts and auditors.
I also believe that there is such a thing as DevOps for automated tests. Tests, in the ABT approach at least, have a development phase, which can be much earlier than the implementation of the actions they use. And when tests are correct and their automation is working well, they can be used for production, which is their execution as part of the development and deployment of the application under test (or its components). In particular, I believe it can make sense to have separate test modules to verify the more complex actions and interface mappings, before they're executed in large-scale functional tests.
Another way of looking at tests as products is from a point of view of company value. If tests are well-designed with a clear scope and the right level of actions matching that scope, they can be reused across many generations of the application they're testing. This is particularly true for business-level tests. For example, tests for mortgage calculations are dependent on business rules, not on the details of a UI or non-UI they're running against. In fact, multiple banks could use the same set of mortgage tests. Having tests like that have value in themselves. They can, in theory, be on the assets side of a balance sheet. Allowing teams to develop them is an investment for the future.
The key of a product oriented approach is, in my view, that via a backlog a team knows and accepts that certain test modules need to be produced, not only because it helps in the development, but also because the organization wants them.
