Performing Effective Automated Acceptance Testing
Software development teams that want to automate their regression testing suites may wonder if there is any way to see the forest for the trees regarding their numerous story tests. Having all of the tests focused on specific stories is certainly helpful, but as more and more stories are completed, a regression suite running all of those tests can easily become very slow and difficult to maintain.
This problem has led some experienced developers, such as Jim Shore, to stop automating their acceptance tests because the benefits did not outweigh the costs.
Jim ran into two issues: (1) Customers weren’t interested in writing the tests themselves. (2) They viewed acceptance testing as being intended to cover end-to-end integration tests, which quickly become slow and brittle.
Gojko Adzic and George Dinwiddie both shared their perspectives on the first issue, which can be summed up as “don’t expect customers to write their own acceptance tests.” Rather, include a customer (or customer proxy) in a discussion with a tester and a developer using acceptance tests as the means to guide the conversation. This technique is sometimes referred to as the Three Amigos.
An interesting response to the second issue is to create tests that look at a “user journey.” In a presentation at Agile 2012, Jez Humble and Badrinath Janakiraman discussed the idea of a journey test and how it can help with automated acceptance testing. The idea of journey tests is that these plain text tests describe a common path through the system and represent several steps in the overall process.
These tests are not intended to exercise all the possible paths through the system but rather to exercise a common path. A small set of journey tests that represent the main paths through the application make up an acceptance test suite, which minimizes the run time for the acceptance suite and allows a team to run it frequently when changes are made to the system.
When a new story is delivered, it generally represents an enhancement to an existing journey rather than an entirely new one, so you can modify the journey test rather than having to create an entirely new acceptance test. Using this approach does not imply that negative tests and edge case tests are abandoned; they are placed in a separate regression test suite that can be run at a separate time and less frequently.
Both of these ideas point out important considerations for a team that is looking to continuously improve its processes. Instead of trying something and immediately throwing it out when it does not seem to work, team members should reflect back on what they were really trying to accomplish with their techniques and figure out if they should use a different approach.
