By Raymond Russell, Chief Technology Officer & Co-Founder, Corvil
Modern trading systems operate at inhumanly fast speeds, but their behavior is difficult to predict and manage and, when they fail, they can fail catastrophically. Given the speed, complexity, and unpredictability of trading systems, the only way to manage them effectively is with other machines. Monitoring systems have been around for decades, based on periodically collecting raw counters from computers and network devices. However this approach cannot begin to provide the required insight or control into complex systems. For example, a key factor in the performance of trading system is their latency, but simply counting that a system executed 10,000 transactions in the last minute tells you nothing about how long any of them took.The only way to get a real handle on the behavior and performance of such complex systems is to capture every transaction as it happens, with an accurate record of the time at which each message effecting the transaction was exchanged. The best way to do this turns out to be on the networks that connect all the trading systems to each other; the network can copy data in flight across it without disturbing or delaying it, and we can use commodity network capture cards to capture it and timestamp it with nanosecond precision. A huge additional boon to capturing on the network is that the network itself is a key component of the trading system, and the performance of the latter depends intimately on it. Often latency arises from the interaction of software with the network, not from either component separately, and the only way to detect and resolve the problem is by capturing software messages in the context of the network connections that carry them. We have therefore developed machines that tap the network and passively capture all the trading activity taking place on it, constantly analyzing in real time the behavior of the trading systems and the networks that connect them. But, the performance of their systems is not the most pressing concern for CIOs in the trading business these days. The stability and safety is a much larger concern, in particular the levels of risk that the trading machines incur. This risk comprises both traditional trading dimensions (How big are the trading positions I have established? How much would it cost me to unwind them?) and technological dimensions (How likely is it that my trading systems misbehaves? What controls do I have in place to catch this?).