Is it safe to trust the machines?
Wilcockson: have you checked your algos?
Algorithms used in trading may appear to be well-tested – but the process is open to any number of unexpected flaws, says Steve Wilcockson, industry manager at big data specialist firm MathWorks.
“A lot of bank testing is comprehensive, but from many years ago, and it might not fit every geography,” he says. “Testing may superficially offer a lot of capability, but below the surface it can be slow and inconsistent and add risk. There’s a lot of focus on the model, but not necessarily the software coverage.”
Trading algorithms have been widely by banks and asset managers for many years, as a convenient way to load orders into the market. The more sophisticated ones also attempt to minimise market impact while seeking out liquidity from multiple sources, such as dark pools and MTFs. However, ‘rogue’ algorithms feeding on human error can sometimes trigger disasters such as the flash crash of May 2010, in which $1 trillion was wiped off the value of the US stock market by an algo-led race to the bottom.
There have been numerous other similar incidents. In August 2012 broker Knight Capital suffered a catastrophe in which a problem with its high-frequency market making software caused a loss of $440 million that forced the firm to recapitalise and seek new backers. Knight eventually merged with rival Getco four months later. In August 2013, Goldman Sachs suffered a trading error that cost the firm an estimated $100 million. That incident was caused by a computer error in which the bank’s automated trading systems accidentally sent indications of interest as real orders to be filled at the exchanges, resulting in a number of erroneous options trades that disrupted trading across US exchanges.
Wilcockson says that part of the problem is that hundreds of people may be working on an algorithm over a period of 10 years. The original designers may have retired or moved on, and staff replacing them may not have the same level of understanding.
“With a lot of big projects, you find spaghetti code,” he said. “Transparency and documentation are essential. The sell-side has a handle on it but prop shops get it wrong. We’re moving into this big data world, but people often have specialised so much that they lack any sense of the bigger picture, and that can be dangerous. If you’re trading and there is no stop mechanism on the algo, that could be disastrous.”
His colleague Tanya Morton, senior application engineer at MathWorks, added that the task of developing trading algos is a complex one, that requires both prediction of future movements, as well as analysis of recent volumes and longer-term historical trends.
“Document its behaviour,” she said. “See what it does if you prod it. Tests make it so much easier later if you need to make a change. Sometimes a line of code isn’t used. Why isn’t it? It may not be safe. What happens then? Software tools like our ‘traffic light indicator’ can be useful. It will warn you if there’s dead code that you didn’t use. Spotting things like that can be important.”
In its automated trading guidelines, published in February 2012, European regulator ESMA said that investment firms should ensure that their algos can function effectively in stressed market conditions, that they have been tested rigorously, especially if the algorithm is deployed to a new market, and that they can be switched off. Investment firms should roll out their algos in a live environment in a controlled fashion and they should monitor the algos in real time. They should also periodically revenue and evaluate their algos, including an assessment of their impact on market integrity and resilience as well as profit and loss, and keep records for at least five years.
Other European countries have gone a step further, and put controls into law. In Germany, there are rules relating specifically to high-frequency trading, in which HFT firms have to register with the authorities and provide details of their algorithmic trading strategies to the regulator on request.
“You need to be asking, how does the algo perform over time? Will it perform the same next week as it did last week? What about over two different months, or even a year? Is it consistent? Proper testing requires a culture of critical thought, and a decent testing environment, that covers different time periods and testing scenarios.”
“It helps if you automate the process where possible,” he added. “For example, one bank we worked with to implement FX algos used to take three months to test its algos, and now has the process down to less than two weeks. You can work smarter, not just faster. Instead of manually entering tests and approving changes, now the process is streamlined and automated. Developers can test immediately, they do final checks and it goes live, that makes a difference. Thinking about it holistically led to a better situation. “
