Automated Trader : Technology Workshop

Much of the underperformance of messaging systems can be attributed to inadequate measuring tools and testing conditions, according to Barry Thompson, Founder and CTO, and Dave Lauer, Senior Systems Engineer, Tervela.

Do Your Testing Methods Deliver?

---

The financial services industry is deploying so-called next generation network devices and software systems to combat the exponential increase in market data volumes and internally-generated messaging traffic. While messaging technologies, such as message-oriented middleware, have evolved, the means used to appraise them have not. This disconnect has led to ‘game-changing’ technologies failing spectacularly when moved from testing into production. Buy-side firms, in particular, are feeling the pain. For example, the ability to process and distribute market data relies heavily upon messaging systems, such as feed handlers and complex event processing (CEP) engines. For order execution, similar challenges exist for both order management systems and FIX engines. The sell side is not immune to this either and often leverages the same technology to service their buy-side clients. Though there are many components to a financial services trading infrastructure, the emphasis of this article is on the performance of messaging systems because they are so critical to other parts of the system. We also examine how replayed and live data yield completely different results when trading firms are testing competitive technology offerings.

Expectations and reality diverge

Recently, a large investment bank in New York walked us through their testing approach for new market data infrastructure products, which was created after a messaging system for market data distribution vastly underperformed against published test results. They lamented that even with their own lab they lacked the requisite methodology and framework to achieve a level of accuracy that would ensure seamless live deployment. So why does production reality not meet real world expectations given how well these technologies performed in the lab?

Low latency and high throughput are the mantras of this generation of technology, yet very little attention has been paid to properly measuring the performance of message systems, or indeed other technologies that support automated trading. In general, many products are evaluated in a development laboratory by replaying recorded data streams and examining the software or the device being tested. Firms use replayed data because they can use it as a basis to evaluate different products and observe how they perform under identical conditions. Frequently, however, systems put into a production environment show materially different performance characteristics compared to what happens in this ‘clean room’ lab environment. These real-time production performance problems cost firms profitability, but could have been uncovered with proper pre-production performance measurement. It has become increasingly apparent that replayed data does not have the same characteristics as live data and cannot accurately demonstrate or measure system performance. Ultimately, trading firms – along with their end users and customers – suffer because they can’t get the market data they need to execute a trade in a timely manner because their systems aren’t performing as they did in lab tests.

The problems inherent in using replayed data to evaluate how a system under test will perform in a live setting can be broken into two areas – measuring tools and test conditions.

Reproducing extreme volatility

Existing test tools for evaluating trading infrastructure technologies sit on commodity hardware rather than specialised equipment and typically employ non-real-time operating systems that can result in clock skew or difficulties in synchronisation. This is acceptable for evaluating legacy technologies in which latency is measured in milliseconds, but not for ‘next generation’ technologies which operate in the microsecond realm. This presents an impediment to coordinated replay of market data from multiple systems, which is critical to recreating the ‘microbursting’ characteristics of a live feed. This vast difference in granularity means that performance cannot be accurately measured for technologies operating in the sub-millisecond space. ...

Registered viewers can post questions and discussion points about this article below

---

Discussion of articles is limited to registered users. Click here to register now click here to return to the top of the page