Test Automation—A Non-Expert’s Perspective
I have been involved in IT for four decades now. Starting as a systems programmer in the 70s, through database management, network management, project management, and now I specialize in testing. Over the last five years, I have noticed a strong trend in many organizations to push toward automating test execution.
I am not a tools expert, and I do see value in the application of automation to testing. However, I notice that many organizations fail to get real value from their automation efforts. Looking at automating test execution, I see two distinct areas of focus, testing at the technical level (component and technical integration) and testing at the functional level (system or acceptance). Each level has different requirements and focus.
Technical level test automation seems to have had the greater success, but it still has room for improvement. I attribute this to the fact that developers already have the skills and knowledge to effectively use their tools and that the structure of code and interfaces lend themselves to automation more easily than at the functional level. In development, highly automated testing is essential to success, especially in iterative/agile projects. The issue here is the quality of the automated tests. If the tests are essentially assertion tests (true/false tests), then the structure is not being effectively tested, and the likelihood of defects getting to higher levels of testing increases. Good technical test automation first requires the creation of strong tests.
Most of the issues I see arise at the functional levels of test automation. This can be attributed to several factors:
- The tester’s lack of technical knowledge and skills
- Constantly changing aspects of the user interface (UI), which complicates the automation process
- Unrealistic expectations from management
The first two issues can be addressed through the use of test automation frameworks implementing keyword (action word) and data-driven scripting. One or two developers can be used to build the necessary test scripts overcoming the lack of technical skills on the part of the testers. However, this requires strong cooperation between the test analyst defining the business level keywords and the developers doing the script creation. I see many problems in this one area. Testers tend to be business focused, and developers are focused more on technology. Here testers need to develop a little tool knowledge to be more effective in working with the developers.
The last area is where I see most problems occurring, and it is the most difficult to overcome. The problems I see most often in this area include:
- Automate everything: The tool (if commercial) cost a lot, so use it. In reality, some tests should never be automated.
- You have a tool, so why is testing not done yet? The problem here is that tools do not come preloaded with the necessary scripts, keywords, data, etc. All of this has to be created, and depending on the skills involved, takes time. Good test suites take time to evolve.
In today’s fast-paced development world, test automation is essential to long-term success. Whether at the technical or functional level, to be successful at automation takes time and requires an understanding of what can and cannot be done. If you are going to automate testing, take the time to do it right, or don't bother wasting resources.
