When the notion of event-driven processing started filtering through to the commercial vendor space at the start of the millennium, there was a display of competitive semantics as vendors jockeyed for position. In their efforts to win over customers, many played down, or even dropped, the word 'complex' from the soon-to-be-established term Complex Event Processing (CEP). They preferred to use such terms as Event Stream Processing to describe their new solutions. The belief was that softer language would counteract any potential resistance from customers who may have been put off by the word 'complex.'
They had a point, although not quite the right one and not necessarily for the right reasons. There is a sense in which the 'C' in CEP is potentially misleading, especially when the kind of intelligence firms may be looking to glean from event-driven processing could be something as simple as determining whether a set of business processes is compliant or running correctly. Even the technology underpinning CEP is not necessarily that complex. "CEP is simply a database that has time," says Frédéric Ponzo, managing partner, GreySpark Partners. "It is doing what a simple database is doing; the logic is still a row of tables and columns and the philosophy behind it is still relational."
So the starting point for an enquiry into Complex Event Processing is the understanding that we won't necessarily be encountering much complexity, and we may not even be talking about Events in the commonly understood sense of the term. This is positive. CEP has evolved into a powerful tool for analysing - okay, processing - the "inner space" of an enterprise, as well as for, let's say, trend-spotting in the rendered-into-data chaos of the wider outside world. To get the inevitable mention of the "butterfly effect" out of the way early on, CEP is the tool for measuring your system's sensitivity to wing movements, before you start using it to find the right flying insect.
The two-headed butterfly
This feature started with a simple but - we thought - potentially intriguing question. How complex can you get with CEP, and still trust what it is telling you? At that point, we were thinking about the outside world. After all, not only has the volume of data to which firms apply event sets increased exponentially, but the data, the event sets, the whole trading environment seems to have become so much more complex in recent years. Recent market events may include the global meltdown, the European sovereign debt crisis, the potential collapse of the euro … but the concurrent list of non-market but market-impacting events is just as dramatic. Remember that volcano? More recently, the Arab Spring, Occupy Wall Street, China, just about any weather forecast?
We can agree, can't we, that the macro-economic world today is characterised by "black swan" events, not to mention swarms of butterflies madly flapping their wings, that present a distinctly robust environment in which to roll out your latest CEP solution? But still, that's only half the story. While all those external complexities have multiplied around us, the challenges of operating in financial markets have also become more complex. There's the ongoing pressure to automate/integrate processes across the back, middle and front office, coupled with demands for greater transparency and more detailed regulatory reporting, along with ever-increasing volumes (and types) of data that might or might not have an impact on any given trading decision.
With data these days, the issue is moving from prioritisation to safe exclusion; there's too much of it. There's also a third, less easily quantifiable element in the mix: we are changing (perhaps "evolving" is the word). As technology changes, so our use of technology changes, and the limits on what we can do with it. Most notably here, the trading strategies for which CEP was evolved a decade ago are now being applied to asset classes beyond equities - just think of all the correlations we track these days. Have we reached the limits of CEP, which began in a time of relatively simple events? To what extent has CEP evolved into a solution fit for today's complexities?
The inside track
There's the CEP you use on the inside, to fine-tune your processes, and there's the CEP you aim at the outside world. Let's start on the inside. Adam Honoré, research director, institutional securities practice, Aite Group, argues that CEP is more mature now and is no longer the "cowboy" product that - he suggests - it used to be. Indeed, in a November 2011 report Aite published on CEP, Honoré wrote: "Aite Group has actively observed CEP transform from an emerging technology to an indispensable platform in place within many capital markets core infrastructures."
Honoré pointed to a trend: firms continue to push CEP into new areas such as position-keeping, transaction cost analysis, real-time compliance and risk and automated quotation systems. Instead of just being used for a single application, CEP is now applied in a multitude of instances within the same enterprise. Aite Group's analysis suggests that, from 2009 to 2011, CEP deployments moved from an average of two deployments per customer to an average of 3.5. Further detail is added to the picture of a rapid rise in the levels of automation with findings from Automated Trader's own research. The recently published Algorithmic Trading Survey Report reveals dramatic growth in the variety of models and range of processes now being machine managed with around 60% of sell-side firms, hedge funds and proprietary trading firms already describing their trading process as being largely or fully automated. Significantly, this percentage is forecast to grow still further over the next two to three years.
Like Honoré, Frédéric Ponzo suggests specific areas where CEP is becoming significantly more critical: client-price negotiation for fixed-income markets; and automation of trade-linked activities in particular around the market-making systems used for warrants and options.
One strength of CEP technology is that it scales easily. When ConvergEx started using CEP four years ago for real-time monitoring of its US equity electronic-trading infrastructure, it started with a single message type. Joseph Weisbord, CIO, ConvergEx, says: "Now we have increased it to every single message. We take every message we can get hold of within our trading infrastructure and put that into the CEP engine.
That includes inbound messages and outbound messages to exchanges, as well as inter-system messages. We generate about 80 million messages a day, which is a pretty big number."
This is not to suggest that "inward-looking" CEP has achieved such widespread take-up because it is easy to implement. There are other reasons. Beau Alexander, head of product management for SunGard's Valdi trade-order management solution, says that the company started using CEP in the last two to three years for a number of functions on the order-management and risk-management side. Significantly, Alexander adds that this CEP take-up was driven by the accelerating compliance demands of SunGard's client base.
"CEP is being used to help clients make sophisticated and quicker compliance decisions or to act on information that could affect their ability to execute an order," says Alexander. ConvergEx, too, uses the CEP engine to monitor credit risk and for real-time compliance reporting, including all of its post-trade reporting. "We also use it to detect if customers are doing layering or spoofing," says Weisbord. And, developing a similar point, Richard Tibbetts, CTO, StreamBase, suggests that there is regulatory pressure towards real-time processing even where an end-of-day batch approach might suffice. "One thing we are seeing across asset classes is middle-and-back-office processes that could be end of day becoming more real time," says Tibbetts. There is also, of course, straightforward process-efficiency. "Clients are starting to use CEP to automate activities they are doing manually," says Ponzo. "That is not algo trading, but just automation of day-to-day trading activities."
Hmm. This is not quite the processing of complex events, isn't it?
Now, forgive me for being naïve, but are we looking at one of those new-fangled unintended consequences one hears so much about these days? You regulate to make an industry compliant and transparent, and you end up with an industry where the core service is the monitoring of compliance and transparency? Inward-looking, self-examining CEP solutions are, of course, an important tool for compliance, et cetera, but compliance isn't an asset class in the way that external complexity potentially is. You can't add to the profit column in the accounts, after all. Not twice, anyway.
This is where we move away from a match between the specifics of a given regulatory requirement and the corresponding specifics of a given CEP solution. The ongoing effort to manage/process the outside world (in the widest sense) has produced a range of solutions based on quantifying/scoring the impact of, well, the problematic. How do you score the Arab Spring, and within that question, how do you score the different [interim?] outcomes in different states? Pointless to ask, actually, or at least: the useful part of that question isn't the answer. It's the way it draws us towards a perennial human fallibility: the temptation to stick with the questions we can answer. We're talking about processing complex events, but we're doing it within a principle: trade what you can quantify.
"You can make software do lots of things," says Tibbetts, "but being able to handle complexity is more about whether you can clearly define what you want the CEP engine to do. If you can't define your business and technical objectives properly, CEP is not going to make it any better." Fair point, but this is a powerful technology coming of age in an industry where success depends on finding the edge that others haven't seen; of pushing the technology to the edge of the envelope and just past it. "CEP's application is only limited by your imagination," says Weisbord.
So let's imagine that CEP has yet to be properly stretched; let's imagine a future world in which CEP capacity is as mission-critical tomorrow as near-zero latency was last week. How do we get to there from here? The first stepping stone is the evolving concept of "Big Data". This, happily, is a complex event in its own right. Data has been big for a while now, and getting bigger, but size isn't the point; the variety of data sources has also been expanding, as has the variety of data types, data inputs, et cetera: we're getting more variety in our data and much more of it. The key perception is that traditional database data is converging with data from "multiple disparate sources" (online, sensory resources, social media), so that what we need is an effective methodology for processing and exploiting such unstructured data feeds.
Google Search Processing?
Investors should consider the frequency of Google searches for particular keywords when analysing market movements. So says a new report from the UK's Essex Business School at the University of Essex and the Norwich Business School at the University of East Anglia.
The researchers analysed a "unique dataset" of Google search frequencies for keywords related to 30 of the largest stocks traded on the NYSE and NASDAQ. Their study "confirms that the internet has indeed revolutionised not only the production, intermediation and dissemination of information but also its consumption for investment purposes". Dr Nikolaos Vlastakis from Essex Business School told our man on the campus: "We derived two new measures for information demand - one for the individual company and one for the whole market - and discovered that both have a strong association with stock return volatility and trading volume. Interestingly, we also found that the link between information demand and market activity becomes more prominent during turbulent times, such as the recent financial crisis."
The study also suggests that "in the information age, large news agencies are not the exclusive providers of new information … [both] formation and discovery may be shared between mainstream information brokers and alternative sources on the web such as blogs, local information providers, social media."
The degree to which this is a new and useful approach is exactly proportionate to the degree to which any apparently established principles of, say, "Unstructured Data Feed Processing (UDFP)" should be ignored. This is about innovating rather than finding ways to score Twitter feeds. CEP may provide a way of extracting value from "Big Data," but Frédéric Ponzo believes it is not quite ready for that yet. "CEP still requires structured data with predictable fields that can be correlated," Ponzo says. GreySpark Partners, he adds, is working with "a couple of buy-side prop traders" who want to use employment and payroll figures to build trading models to determine how the market responds to such news. This, says Ponzo, cannot be done using CEP alone. "We are having to create an interpretation layer using something like heat mapping. When it comes to 'fuzzy processing', CEP is not there yet. It is not going to get there, something else will get there, run in parallel or on top of it."
As Ponzo's summary suggests, that something else could be a data-visualisation layer. One vendor that supports "fast, effective analysis of real-time streaming feeds from CEP engines" is Panopticon. Chris Elsmore, senior vice president, says Panopticon is "helping firms visualize the output from CEP engines so that they can determine how their portfolio or position is changing and spot outlying trends or abnormalities". Visualisation enables them to spot these trends much more quickly than using a traditional spreadsheet or blotter. "We can enhance the power of CEP in terms of what is happening under the covers," says Elsmore. "It's about having the ability to not just see the data but to see how it is changing in real time." Elsmore sees CEP and data visualization as "symbiotic".
Big Data isn't the same data only more of it. Big Data is something different. Could that mean, perhaps, that complexity is different, too, so that the solutions needed to process it need to be taken out of their "silos" and remixed? "As any technology matures you start to identify a new stack. CEP platforms are part of a real-time stack, but increasingly that stack also includes analytical tools such as data visualisation, sentiment analysis and alerting," says Tibbetts. Just imagine how that's going to turn out.