The Gateway to Algorithmic and Automated Trading

Complex Event Processing - Making sense of all your data

Real-Time Innovations (RTI) : Jens Pillgram-Larsen - 1st January 1970

The opinions expressed by this blogger and those providing comments are theirs alone, this does not reflect the opinion of Automated Trader or any employee thereof. Automated Trader is not responsible for the accuracy of any of the information supplied by this article.

So you have a distributed system and you're happily sending data between nodes in your system. The consumer applications are consuming the data your producer applications are producing, and everything is running smoothly. Now, that doesn't sound like any system you know does it? Distributed systems are by nature complex. Nodes and applications are not straight producers or consumers; they're a bit of both. And there's always some resource contention. This is where Complex Event Processing (CEP) comes on the scene. CEP allows you to run queries on streams of data in real-time, either transforming the data or triggering alerts based on data content. Let me explain by talking about a couple of use cases.

Let's say you have a large distributed system where data is being exchanged between nodes and applications. You're finding that some of your applications are getting starved for data, but everything looks fine in the network - there are only a few lost packets and plenty of bandwidth available. The next step is moving up the stack to see what's going on with the networking middleware. Now, if you're lucky enough to have a well-behaved middleware like RTI Data Distribution Service, there are ways to retrieve detailed statistics about the communication channels. Passing this information through a CEP engine you can write queries that determine when and where you have a slow producer or consumer in your system that is clogging everything up.

Another common use-case is event correlation. Seemingly disparate events may not be, and corroborating can trigger pattern based alerts. Anyone who's watched a Hollywood spy movie knows this - increased cell-phone chatter coupled with certain keywords and the change in behavior of suspected operatives means something bad is likely going to happen. Using CEP events can be monitored and corroborated in real-time and alerts sent out when the situation warrants it.

CEP is an incredibly powerful technology, and with increasing data rates it is a technology that should be watched carefully in the next couple of years.

Go check out RTI Event Processing at for an integration between RTI Data Distribution Service and Complex Event Processing.