Test Design for Automation: Anti-Patterns
A lot of what I write and talk about pertains to the role of test design in efficient and scalable automated testing. I have learned that success in automation is not as much a technical challenge as it is a test design challenge.
In our “Action Based Testing” method we organize tests into easy to manage “test modules” that look like spreadsheets, which we edit and manage in our tool, TestArchitect. The test modules contain test cases that are written as sequences of “actions,” spreadsheet lines with an action keyword and zero or more action arguments.
Since test design can have a big impact on automation, several people have asked if there might be “anti- patterns,” situations to look for in a test that potentially could be harmful for their maintainability and scalability.
Wikipedia describes an anti-pattern as “a common response to a recurring problem that is usually ineffective and risks being highly counterproductive.” The term was coined by Andrew Koenig, based on the well-known notion of “design patterns.”
Just like with design patterns, anti-patterns can benefit from a short and catchy name to make them easy to remember and talk about. Below is a list of typical situations I have seen in tests that I think can harm automation and the names that I have come up with for them so far. It is a work in progress and your feedback is welcome. The anti-patterns often occur in combination and may have some overlap as well.
Enter Enter Click Click: Test steps are too detailed. Many tests, even automated ones, have been designed on a detailed step-by-step basis. This makes it difficult to manage and maintain those tests. For example, it will be hard to factor the impact of changes in the application under test into the tests. Identifying common sequences of steps and putting them in actions or functions will relieve this problem.
Interaction Heavy: Not having many business tests. A main distinction I recommend that testers make is between “business tests” that focus on business objects, rules and processes on one hand, and “interaction tests” that focus on the interaction with the application. However, I’ve seen in many projects that testers focus only on the interaction. This makes the tests shallow and misses potential business level problems.
Lifeless: Missing life cycle steps of business objects. Most applications work on “business objects,” like orders, invoices, products, customers, etc. These objects have their life-cycles in the application, like creation, update, retrieval and closure, and also operations like copy, move, export, import, etc. However, the tests for such basic operations in applications are often scattered and as a result, hard to find and incomplete.
Lame: No depth or variety, no testing techniques used. Time pressure and other factors often result in shallow test cases that don’t challenge the application much. Try to think as a tester, somebody who wants to break things. Applying testing techniques and interaction with various stakeholders can help you in this process.
Clueless: No clear scope for the tests. A very common situation is lack of scope for tests. The tests then are hard to find and assess, and may do work that is also done in other tests.
Over-Checking: Checks not relevant for the scope. Since test designers often follow an approach of steps with an expected result for each step, tests do many checks that do not fit the scope of such tests. Such checks are unnecessary and probably over-lapping similar checks elsewhere. They then clutter up result statistics (e.g. they create too many “passes”), and aggravate the impact of changes in the application under test.
Cocktail: Interaction tests mixed with business tests. Even if tests are testing business functionalities, like business object life cycles and business rules, calculations and processes, they are often mixed with tests on interaction details, resulting in a convoluted and hard to maintain mix. A common example is to describe a log-in process in detail in all tests that start with a log in.
Sneaky Checking: Checks hidden in actions. Even though it is good to have business level actions that hide unneeded details for many of the tests, try to avoid hiding too much. In particular, checks should be explicit and visible in the main test (the test modules), at the appropriate level of detail. An outsider should be able to understand what is being tested by just looking at the test module, without a need to inspect how actions are implemented.
These are the anti-patterns I have for now. If you see any of these in a test, you may need to consider modifying them. When determining if your tests need modification, a good first step is to look at the overall test design. Make a list of your tests and see if you can determine test modules with clear, unambiguous and differentiated scopes, then within each of these test modules make sure to stay within that scope.
Being attentive in recognizing test design choices that can complicate automation (and possibly testing quality) can help to make tests more concise, effective and easy to maintain. The anti-patterns described in this blog may help spot issues earlier, in combination with a well thought through design and organization of the tests.
