7 Essential Quality Attributes for Your Test Automation Framework

A common problem in software is that developers and designers tend to concentrate on pure functionality and neglect quality attributes. These attributes are the famous “-abilities”: usability, reliability, portability, and so on.

This statement is valid for test automation frameworks as well. But there is an additional issue here: People often forget that a test automation framework is still software. Moreover, it is very specific software that can be compared with medical diagnosis software, because test automation frameworks are used to diagnose the health status of the software under test.

Considering this perspective, then stability, reliability, user-friendliness, communication, portability, and other quality attributes are crucial for test automation frameworks. Without them, it's dead, like a body without a soul.

There are some of the signs that something is wrong with your test automation framework and it needs some CPR:

• Low ROI
• People in the company refuse to use it
• The number of automated tests remains low
• Quality is not improved
• There is no trust in test results

If you find that your testing framework is suffering, you might want to check whether the framework has these seven quality attributes:

1. Reliability: The test framework should be more reliable than the software under test. It should be tested with the aim of preventing it from crashing or throwing unhandled exceptions. Think about network delays, environments that are not available, and lost database connections, and ensure that the framework handles these cases.

2. Usability: To be widely adopted, the framework should be easy to use, starting from installation and ending with user-friendly test reports.

3. Communication: All the changes in the framework should be clearly communicated to the teams that use it. And don't forget about the release notes.

4. Test data: The ability to create test data is functionality, but how easy and fast we can do it is a quality attribute. Provide tools for users that allow them to handle the management of test data in a simple and fast way. 

5. Portability: Different members of the team might use different OSes, and all of them should be able to run tests on their machines, in the cloud, and in the build pipeline. The framework must be usable in all these environments, or else its value significantly decreases.

6. Reporting: Reports are one of the most important parts of a test framework’s functionality. Test reports that are unreadable, uninformative, hard to find, or vague, that do not provide clear information about test failures, make even the best test cases useless. 

7. Integration: It should be easy and fast to integrate the test automation framework with other tools. Think about build/deploy pipelines, APIs, and a convenient command-line interface.

Of course, there are many other factors that impact the quality of test automation frameworks. There is no question that your framework should have all the required functionality as well. However, remembering that the framework is still software—more specifically, a “diagnosis software”—and that it must be “alive” will help you build a robust automation solution, as well as save a lot of time and money in the end.

Iryna Suprun is presenting the session Destroying the Horcruxes of Full-Stack Automation at STARCANADA 2019, October 20–25 in Toronto, Ontario.

• Login or Join to add your comment