There are so many end-to-end trading solutions in the marketplace today that at first glance it's far from obvious why the world needs another one. For day traders, systems such as NinjaTrader and TradeStation and SmartQuant all offer powerful out of the box options. For high frequency traders, companies like QuantHouse and Deltix offer low latency feeds and trading options, as well as a framework with which to build bespoke solutions.
However, all these systems still require a significant amount of time, effort and coding to build a fully operational system that can:
• Make use of the latest statistical and machine learning techniques to build state-of-the-art trading models, whilst also...
• ...managing the logistics of complex spread and basket trades in an intelligent manner.
All systematic trading teams learn the hard way that building the infrastructure to handle data feeds, model development and the complex business of order management and execution always takes a huge development effort, regardless of the tools being used.
In an ideal world one would combine the easy to use, high level numerical libraries of a language such as MATLAB or Python, with a low level, low latency data feed, and plug the whole thing into a smart order execution system that plugs directly into a broker. No time/effort expended on the plumbing, maximum time available for the smart stuff.
This ideal is exactly the model that Agora claims to deliver. It combines an electronic broker (currently Trading Technologies or Interactive Brokers, but extensible to others) with fully multithreaded .NET libraries for feed handling and order execution, and a direct API link into MATLAB or .NET.
This should result in a system that offers higher-level access to data, order management, trade execution, and logging designed specifically with higher-frequency futures trading in mind. As a result, traders would be able to concentrate on the business logic of their trading models, without having to build all the infrastructure first.
Multiprocessing and Multithreaded Programming
An important design consideration that lurks beneath the surface of any higher-frequency trading application is the necessity of tracking many different tasks. These include monitoring one or more data feeds for instrument ticks, synthesising baskets and spreads out of these feeds, applying the model to these instruments, creating orders, monitoring and managing the orders as they are processed by the brokerage, logging all activity, and even visualising the overall health of the system. These various tasks are all running simultaneously and not necessarily in lockstep with each other: they are asynchronous by nature.
The software design solution to this problem is the use of multiple threads and/or processes, one for each of the performed tasks. This is a practical innovation as regards speed, since multiple cores and CPUs can process threads and processes in parallel.
The use of this approach or entirely separate processes (such as in MATLAB or Python) is essential in higher frequency trading if the main modelling thread is to be kept unencumbered by bookkeeping tasks, such as monitoring a trailing stop. If it is not, the modelling calculations will be queued behind the far simpler but much more frequent auxiliary calculations. Worse still, the modelling thread can be caught in concurrency issues in a higher-frequency environment and grind to a halt.
The solution proposed by Agora avoids this by removing as many simple, frequent calculations from the main process as possible. It does this by writing them into the .NET and Java processes already being used to monitor the data, brokerage, and database connections. This underlying multiprocessing model is one of Agora's main advantages when trading tick-level data.
Data: Feed and Pre-Processing
Agora provides a data feed connection to Trading Technologies in a manner similar to other APIs. A sample, simple code snippet for the MATLAB version of Agora shown in Figure 1.
The compactness of the spread definition code is striking: in a handful of commands (starting with "theSpread", "leg1", and "leg2"), the active futures for Brent Crude and West Texas Intermediate are subtracted from each other and monitored. These spreads can be defined both in additive terms (useful for commodities futures) and in multiplicative terms (useful for FX futures).
Many existing platforms (NinjaTrader, MetaTrader, IB-MATLAB etc.) are flexible as regards order type availability and handling, but they all require a large amount of coding to handle things like multi-leg trades.
One of the most useful parts of Agora is the order management and intelligent execution system. We watched Agora deal with a three legged spread (FTSE/S&P/GBP-USD futures), then a two legged spread (EURO STOXX50/CAC40 futures) and then examined the trade logs to trace the behaviour of the execution engine.
The first thing to note is that spread definitions are contained in a configuration file and do not need to be specified for each trade. This means that only a single instruction was sent to the execution system from MATLAB, and the appropriate legs were created and submitted completely transparently. The system ranks the legs in order of priority, based on a trade-off between the potential cost of each leg, and the latency of the round trip message time to each exchange involved in the trade.
Having executed the first leg, the execution system begins an iterative process in which it continually adjusts the remaining legs to make sure that the overall target price for the spread is maintained as closely as possible. The remaining legs start off passively, sitting on the bid or offer, until either a predetermined amount of time has elapsed with no trade, or the market moves away from the target. In addition, an internal sub-process continually polls the Level 2 data and attempts to predict the next likely move of the order book. We noted dozens of order price updates as the execution system juggled the remaining legs in order to meet or beat the target it had been given, ultimately breaking even in one spread, and earning a couple of ticks on one leg in another. Overall an impressive performance.
Nevertheless, some types of behaviour - such as widening bid-offer spreads combined with sudden price gaps - caused the execution algo to cross the spread sooner than might have been warranted, though such situations are always challenging. Another point to bear in mind is that there are a large number of parameters controlling the behaviour of the execution system, contained in a couple of XML config files. Editing and managing these files and interpreting trade log files (see Figure 2 for a sample of the real time output that builds these) are not tasks for the faint hearted!
Agora's execution engine can also handle some fairly complex trade types, such as limit or stop limit orders for spreads (with a target price for the spread), iceberg orders and even trailing stop limit orders.
A final feature of interest in the order management and execution system is the ability to run end of day reconciliation. This shows all the trades that your trading model thinks you should have done, and the prices it believes they were executed at, compared with all the actual executions done and the real world prices obtained. This information is invaluable in determining the slippage of a particular trading model or if any alpha is being added by Agora's smart execution.
No algorithmic trading environment is complete without a stable, permanent logging process for every action that the model takes. Apart from monitoring assumptions about model performance, slippage and liquidity, it also facilitates recovery in situations such as a software crash.
A high-performance database is the preferred solution here. This should ideally be able to process queries from multiple processes without falling foul of the concurrency issues that are inevitable in a higher-frequency trading engine. This is clearly something poorly suited to an environment such as MATLAB, where it will tie up the main MATLAB thread while it performs its frequent update operations.
Agora avoids this by moving the logging commands entirely out of the MATLAB process and using a PostgreSQL database (see Figure 3) for its logging functionality. This has Multiversion Concurrency Control (MVCC), which allows the many processes to access and write to PostgreSQL without stepping on each other's toes. Better still, it has an open-source license.
As mentioned above, Agora is currently an alpha product, so some of the following Wrecking Crew recommendations will hopefully be included in the final cut:
• While it is an advantage that Agora leverages multiple asynchronous processes behind the scenes, its current state does expose the end user to .NET/Java objects and XML files. Ideally these could be hidden, so the end user need only be familiar with MATLAB constructs.
• The many settings for each trading brokerage API are currently stored in XML files, which risks the end user inadvertently corrupting them when making edits. Some form of GUI and/or MATLAB commands for this task that also incorporated syntax checking would be useful.
• Agora is currently being developed for both a MATLAB and a .NET-based front end. Unsurprisingly, some of the Wrecking Crew would like to see support for other high-level languages, and even more unsurprisingly Python was mentioned.
• PostgreSQL is currently the only database supported. While it is an appropriate choice, some users will already be using other databases, so some means of migration would be welcome.
Building a flexible system that combines fast data feeds, a high level modelling language and a smart execution system capable of managing many legged trades whilst minimising slippage is a tough, time-consuming job and far harder than many imagine.
Agora is a system designed and built to solve exactly that problem . The most innovative feature of the tool is the decision to keep as much of the low level book-keeping involved in managing spreads and handling messages between execution system and broker out of the modelling environment. This provides a powerful, distributed and multi-threaded tool that frees users to concentrate on building optimal trading and execution strategies. Key features of the system include:
• Fully multi-threaded .NET architecture with a MATLAB API
• Fast and configurable data feeds (from TT, IB and Bloomberg) directly into MATLAB, without using MATLAB to do any book-keeping
• Intelligent order management and execution system (linked to TT and IB, but extensible to other brokers) to squeeze out maximum execution alpha
• In depth reporting and monitoring of latency, slippage, and anticipated versus realised trading for end of day reconciliation
All told, while it is still currently at an alpha stage, it is apparent that Agora is an innovative, powerful and thoroughly worthwhile addition to the market for end-to-end trading solutions.
1. As the software currently requires some degree of customisation, traders wishing to use or trial Agora would need to get in touch with Energeia Associates, who can provide the necessary support for installing and customising the tool as required.
Smart order management and execution alpha
As anyone who has spent time looking at Level 2 data knows, there is no such thing as "the price" for a particular instrument. Everything depends on the details. Are you buying or selling? What quantity do you need? How urgent is your trade? Every potential trade has an associated cost. For example, do you cross the bid-offer spread (aggressive) or do you wait on one side to be hit (passive)? It is usually the case that a trading model will indicate a target price at which an instrument should be bought or sold. Your execution quality determines whether what happens next is slippage or execution alpha.
Minimising slippage is tricky enough for single instrument trades, but gets substantially more complex when one is trading synthetic spreads or butterflies. Trades that can have up to four or five legs (such as condors or basket stat-arb trades) can be a nightmare to execute correctly. And that's before one even begins to consider the risks and costs of being hung up on one or more legs.
When coming up with strategies for managing the complexity involved in placing multi-leg trades there is a labyrinthine decision tree to navigate. This includes cost considerations (which is the more costly leg to miss?) as well as practicalities (when trading instruments on different exchanges, which leg can be executed faster?). Then there is the question of how to minimise market impact. Finally, decisions need to be made as to what to do when it all starts to go wrong. At what point do we give up on a trade? How long can we remain passive before having to cross the spread and take the hit?
All this makes Agora's exclusion of these considerations and their associated housekeeping processes from the high level programming environment look well-considered.