Test Automation Gone Wild
The modern stock market could not function without software, but it also seems that it can no longer function with it.
Automated trading software at Knight Capital Group, a brokerage firm that handles more than 10 percent of all trades for US companies, recently went berserk and cost the company more than $440 million in less than an hour. News of the loss sent the company’s stock plunging from more than $10 to less than $3.
It’s not just the financial loss, although that loss is almost four times the company’s entire profit for last year. It’s also the loss of confidence caused by such glitches in the market. Just recently, the Facebook IPO was bungled by NASDAQ. In 2010 automated trading systems triggered a so-called “flash crash” that vaporized a trillion dollars from the market in a single day.
“There are increasingly more problems, and we haven’t been able to get this right,” warned the chief executive of a financial research and advisory firm.
This craziness is driving individual investors out of the market. What's the point in carefully doing your research, developing your strategy, and monitoring your portfolio if a conspiracy of computers can wipe you out in the blink of an eye for no apparent reason? It is not implausible that in the not-too-distant future the stock market could devolve into a competition between coders instead of traders.
As a QA professional, it is tempting to lay the blame on inadequate testing, but based on experience, I can assure you that most financial firms do not scrimp on test automation as compared to other industries. Yet, in an industry that relies on highly complex calculations driven by a multitude of real-time factors from sources around the globe, it may not even be possible to predict—let alone validate—every possible scenario.
It does raise an interesting question: Is there a place for testing in production?
Historically, testing has been relegated to a tightly controlled environment that hopefully screens software before it enters production. My past attempts to sneak a few tests into production have been met with horror at the very idea. And yet, there are cases where it is virtually impossible to recreate realistic conditions in a simulated platform.
Think about it. What if every automated trade had to run through some type of test before its release to the market—say, one that verified the expected outcome and checked it against a set of filters such as projected income or loss that is cumulative for the day? In Knight Capital’s case, there was a human override available to shut the system down, but it, too, failed to act in time. Perhaps, with some real-time testing, they could have prevented such a stunning loss in such a short time.
Of course, there are risks with running tests in production. But, if these tests were limited strictly to validation, maybe the risk could be offset by the reward. If automation can generate millions of trades in the blink of an eye, maybe we could allow for an extra blink or two to get them right.
So, could production be the next testing frontier?
