The Gateway to Algorithmic and Automated Trading

Data Distribution Challenges for Next Generation Applications

Published in Automated Trader Magazine Issue 09 Q2 2008

Angelo Corsaro, Product Marketing Manager, PrismTech Corp., outlines the advantages of a new middleware data distribution service (DDS) over messaging technologies currently deployed to support automated and algorithmic trading.

A growing number of stock exchanges, data aggregators, trading and investment firms are being challenged with the problem of distributing ever-increasing message volumes, while at the same time trying to reduce end-to-end distribution and trading latency. This problem, further exacerbated by the very nature of the financial markets - where the right answer delivered too late becomes the wrong answer, and where the difference of a millisecond could be worth millions of dollars - is motivating firms to find new and innovative ways of reducing latency while simultaneously increasing the number of messages handled per unit of time.

In fact, the need for ultra-low latency and high throughput messaging technologies has been widely accepted and is starting to receive an increase in attention within the industry. Messaging technologies that promise the extra-low latency and high throughput are already available, but a complete solution to the problem requires the provision of low latency and high throughput along with the proper level of control, stability, usability and expressiveness, thus addressing the need for increased performance as well as reduced time to market.

"… as messaging technologies don't live in isolation, it is becoming increasingly important to have seamless and optimised integration with key technologies, …"

Specifically, the real quest is for messaging technologies that can distribute from hundreds of thousands to millions of messages per second, while maintaining extra-low and predictable latency, while also providing:

• Stability in overload conditions;

• Scalability;

• High availability;

• Fairness;

• Control over the latency/throughput trade-off;

• Support for traffic engineering - such as hardware filtering, traffic shaping and traffic prioritisation;

• Data persistence; and

• Event processing - such as content-based distribution, data correlation and data aggregation.

Figure 1:The data distribution service
Figure 1:The data distribution service

Moreover, as messaging technologies don't live in isolation, it is becoming increasingly important to have seamless and optimised integration with key technologies, such as database management systems (DBMS) and complex event processing (CEP) engines. Data distribution service (DDS) for real-time systems, a standard-based publish/subscribe middleware defined by the Object Management Group1, optimally addresses these challenges by providing: a very powerful data distribution and event processing model; extremely high performance; and a rich set of non-functional properties fully tunable by means of a set of Quality of Service (QoS) policies, used to control properties such as resource usage and reservation, data prioritisation and data availability.

DDS for real-time systems

The data distribution service (DDS) for real-time systems2 is a standard data-centric, topic-based publish/subscribe middleware that has been designed to address the data distribution challenges of highly distributed real-time business- and mission-critical applications. Over the past few years, DDS has extensively and successfully proved itself in fields such as air traffic control and management, combat management systems, large-scale supervisory control and data acquisition, and financial services. 

As shown in Figure 1, DDS is a publish-subscribe technology based around the simple yet very powerful concept of a fully distributed global data space in which publishers and subscribers asynchronously produce and consume data. The global data space can be further divided into partitions in order to segregate information as well as traffic flows. When compared with other publish/subscribe technologies, DDS stands out because of: its powerful support for data modelling, event filtering and processing; and its support for a very rich set of QoS policies which allow it to configure the most important properties of data distribution, such as its temporal properties, its availability and its presentation. In particular, the ability to support data modelling, event filtering and processing, and network traffic engineering is key to DDS's successful application in the financial markets.

Data modelling

DDS perfectly blends and extends the most useful features found in real-time messaging middleware and relational databases. From real-time messaging middleware, DDS draws efficiency in data distribution, low latency, predictability and throughput. From relational databases, it draws the ability to define relational data models and operate on them via SQL92 expressions to specify content-based subscriptions, join, projection, filters and queries (see Figure 2). DDS allows the application programmer to create object views from a relational information model. This ensures that the applications logic can focus completely on the business problem, while delegating to the middleware the mechanics of data distribution. Another typical use of the object-oriented view is to reconstruct and manage relationships out of the messaging data, perform complex filtering and correlation. As shown in Figure 2, DDS can perform data filtering and correlation by relying on either SQL92 expressions or user-provided filter-objects.

All these capabilities are provided via a fully distributed architecture that ensures performance, predictability, availability and scalability, and which can be controlled and fine-tuned by a set of QoS policies that allow traffic prioritisation, traffic shaping, hardware and software filtering, and persistence.

The native support for relational information modelling makes it very efficient to seamlessly integrate DDS implementations with databases. Moreover, the combination of data filtering (by means of content-based subscriptions) and event processing (by means of filter-objects) allows the execution of arbitrary complex event filtering and correlation in the middleware, thus providing the application with only the events it is really interested in.

DDS and complex event processing

Complex event processing is being used in a growing number of financial markets applications. CEP engines are being used to implement growing portions of trading algorithms, for example, and for checking in real-time regulatory compliance. CEP engines that are integrated into trading systems are often flooded with events fed from multiple sources,

such as databases and messaging middleware, and then asked to extract complex event patterns from this incoming flood. Although this approach seems to work, the problem is that some, perhaps many, events reaching the CEP are not necessarily needed, and could be pre-filtered to improve overall throughput.

Figure 2:DDS relational and object-oriented data modelling
Figure 2:DDS relational and object-oriented data modelling

With its support for relational data modelling, object views and filtering, DDS is ideally suited to work in combination with CEP. First, the DDS is able to feed CEP engines with either relational data, or objects, or both (as shown in Figure 3). If, for example, the CEP engine works with Java or C++ objects, the DDS object-view could be used to automatically reconstruct Java or C++ objects out of the relational data streamed from publishers, or from a DBMS (alternatively, the CEP engine could be fed with relational data). Second, some of the filtering and correlation can be pushed from the CEP engine to the DDS to optimise overall resource usage, as well as to improve the total number of relevant messages processed by the CEP engine per unit of time. Third, the DDS can be used to publish complex events and present them to other applications.

Gaining control via Quality of Service

To support mission- and business-critical systems, any middleware must control and optimise its use of resources, such as network bandwidth, memory and CPU time. As such, DDS is equipped with QoS policies that provide firm control of properties such as data timeliness, priority, reliability and persistence. Two key sets of DDS QoS policies that are particularly relevant to financial markets applications are those controlling data timeliness and data availability.

Figure 3:Combining the capabilities of DDS and CEP
Figure 3:Combining the capabilities of DDS and CEP

The following QoS policies control the timeliness properties of distributed data, the trade-off between latency and throughput, and data prioritisation:

• The LATENCY_BUDGET QoS policy provides a means for applications to inform the DDS of the urgency associated with transmitted data. The latency budget specifies the time period within which the DDS must distribute the information. This time period starts from the moment the data is written by a publisher until it is available in the subscriber's data-cache ready for use by reader(s).

• The TRANSPORT_PRIORITY QoS policy allows applications to control the importance associated with a topic or with a topic instance, thus allowing a DDS implementation to prioritise more important data relative to less important data.

• The DEADLINE QoS policy allows applications to define the maximum inter-arrival time for data. DDS can be configured to automatically notify applications when deadlines are missed.

These QoS policies make it possible to ensure that the system is tuned to operate in the right latency/throughput trade-off point and that important data is always able to pre-empt less important data, thus ensuring that, even in overload condition, the temporal properties of critical data are preserved.

"… middleware must control and optimise its use of resources, such as network bandwidth, memory and CPU time."

The following QoS policies control the availability of data to subscribers:

• The DURABILITY QoS policy controls the lifetime of the data written to the global data space. Supported durability levels include:

• - (1) volatile - which specifies that once data is published it is not maintained by DDS for delivery to late-joining applications;

• - (2) transient local - which specifies that publishers store data locally so that late-joining subscribers get the last published item if a publisher is still 'alive';

• - (3) transient - which ensures that the global data space maintains the information outside the local scope of any publishers for use by late-joining subscribers; and

• - (4) persistent - which ensures that the global data space stores the information persistently so it is available to late joiners even after the shutdown and restart of the system.

• The LIFESPAN QoS policy controls the interval of time during which a data sample is valid. The default value is infinite, with alternative values being the time-span for which the data can be considered valid.

• The HISTORY QoS policy controls the number of data samples (i.e. subsequent writes of the same topic) that must be stored for readers or writers. Possible values are the last sample, the last n samples, or all samples.

These policies control the degree of time decoupling between publishing and subscribing applications, as well as the historical data that the middleware should keep on behalf of the application. Other QoS policies exist to control both the replication of publishers (thus greatly simplifying the implementation of Hot-Swap and Hot-Hot replication styles) and the use of computational resources, such as main memory and buffer sizes.

"DDS stands out for its powerful support for data modelling, event filtering and processing, …"

QoS-driven traffic engineering

In addition to the DDS QoS policies that control the temporal properties of data, it is worth investigating how these policies are leveraged by advanced DDS implementations to enforce specific network traffic properties:

• Latency/Throughput Trade-off - Advanced DDS implementations rely on the LATENCY_BUDGET QoS to bundle data across topics and applications to ensure optimal throughput and reduce CPU utilisation.

• Data Prioritisation - The TRANSPORT_PRIORITY QoS is used in combination with network channels to enforce message priority even on non-priority-preserving transports, such as the TCP/IP or UDP/IP.

• Traffic Shaping - For each channel, it is possible to define the traffic profile to ensure that the network utilisation never exceeds a user-specified value.

• Traffic Partitioning - DDS partitions are mapped to IP multicast addresses to segregate different traffic flows and rely on hardware filtering capabilities provided by modern routers, switches and network cards.

The combination of these architectural features and QoS ensures that DDS applications have full control over the characteristics of generated traffic. As a result, the system can be properly tuned and configured to operate at the right performance point, while retaining stability during overload conditions.

Data distribution through power and control

Designed to address the data distribution challenges of mission- and business-critical systems where the right answer delivered too late becomes the wrong answer, data distribution service (DDS) for real-time systems represents a leap forward with respect to existing publish/subscribe technologies. DDS stands out for its powerful support for data modelling, event filtering and processing, and its support for a very rich set of QoS policies which allow it to configure the most important properties of data distribution. DDS implementations have also proved to be extremely high performance, delivering ultra-low-latencies and extremely high throughput. DDS provides the power and control for addressing the most challenging data distribution problems faced in the financial market.

Firms that should be seriously consider deploying DDS include:

• Automated trading firms, for upgrading their internal data distribution and taking advantage of the event filtering and processing capabilities of DDS;

• Stock exchanges, for internal data distribution and QoS-enabled data distribution to aggregators and trading firms - allowing the enforcement of service level agreements;

• Feed handlers, for dealing with massing message volumes, efficiently aggregating and filtering them, while minimising latency;

• Large banks which are struggling to maintain performance in their back-ends; and

• Risk management solution providers that want to achieve better performance while at the same time relying on the powerful programming model provided by DDS's object views along with event filtering and processing capabilities.

1. OMG is an international, open membership, not-for-profit computer industry consortium that develops enterprise integration standards for a wide range of technologies.
2. See the following for more details: "Data Distribution Service for Real-Time Systems Specification" (; OMG DDS SIG Portal (; and OMG DDS Forum (