Do Your Testing Methods Deliver?

First Published in Automated Trader Magazine Issue 08 Q1 2008

Much of the underperformance of messaging systems can be attributed to inadequate measuring tools and testing conditions, according to Barry Thompson, Founder and CTO, and Dave Lauer, Senior Systems Engineer, Tervela.

The financial services industry is deploying so-called next generation network devices and software systems to combat the exponential increase in market data volumes and internally-generated messaging traffic. While messaging technologies, such as message-oriented middleware, have evolved, the means used to appraise them have not. This disconnect has led to ‘game-changing’ technologies failing spectacularly when moved from testing into production. Buy-side firms, in particular, are feeling the pain. For example, the ability to process and distribute market data relies heavily upon messaging systems, such as feed handlers and complex event processing (CEP) engines. For order execution, similar challenges exist for both order management systems and FIX engines. The sell side is not immune to this either and often leverages the same technology to service their buy-side clients. Though there are many components to a financial services trading infrastructure, the emphasis of this article is on the performance of messaging systems because they are so critical to other parts of the system. We also examine how replayed and live data yield completely different results when trading firms are testing competitive technology offerings.

Expectations and reality diverge

Recently, a large investment bank in New York walked us through their testing approach for new market data infrastructure products, which was created after a messaging system for market data distribution vastly underperformed against publishe...