When You Can’t Manually Test All Possibilities, Turn to AI and Automation
When you’re putting together a focused, small-scale application that a few hundred or thousand people aim to use, testing it isn’t exactly rocket science. The number of different scenarios is limited, and the load you can expect from day to day is something you can plan for.
However, when you move to everyone’s favorite example, Niantic Software’s Pokémon Go, the variables you’re dealing with are eons beyond what a normal testing team can manage. Millions upon millions of players are experiencing the application all over the world and putting it through paces that the development team might have never dreamed of.
How do you manage a user base of that magnitude without constant crashes, numerous bugs, and inconsistent quality that forces a large percentage of your audience to uninstall the app? Jonathon Wright, the director of software engineering at CA Technologies, doesn’t think it’s practical to only do so with manual testing.
“How do you test these huge environments? You can try to do it manually with hundreds of gamers that spend all their time playing every scenario,” Wright explained to StickyMinds. “But if you want to be thorough, you can't physically test all the possibilities.”
The same can be said for a video game project with as many concurrent online players as Overwatch. The first-person shooter developed by Blizzard Entertainment has sold more than 20 million copies since its launch earlier this year, and in order to keep servers stable and maintain a high level of quality from game to game, teams have to take advantage of artificial intelligence, automation, and different systems that help pinpoint issues.
“We're starting to see these systems that can actually go through and exhaustively test through the application, especially when you start taking video analytics and image recognition, which is what we were talking about with Google APIs,” Wright said. “If that's a simple call, it can understand where I am in Minecraft by looking at it and identifying, OK, I'm stuck in front of a tree, so I can't physically go through, whether or not the tree is a virtual tree or a picture of a tree.”
The means by which we can improve (and hopefully guarantee) quality for applications and games at this scale are only growing more sophisticated, too. Sure, the manual testing side of things will always be important, but the closer we can get to autonomously monitoring certain aspects of this complex process, the better.
It’s not easy keeping a massively popular application with millions of users running smoothly. But the more we take advantage of AI, automation, and ever-evolving systems built to properly test every possible situation, the better the user experience will be.
