The Gateway to Algorithmic and Automated Trading

Get real!

Published in Automated Trader Magazine Issue 15 Q4 2009

Historic performance data is no longer even remotely reliable as a clue to what’s going to happen next. What does that mean for system designers? Whether we’re modelling for another catastrophe, or planning for the V-shaped, W-shaped, rune-shaped or just wiggly-line recovery, we need to factor in the market’s potential to turn weird so fast that you can’t switch off the machines in time. And how do we do that? For Mick Hittesdorf, it’s all about Japanese cars.

While many segments of the financial industry still struggle to regain their footing in the aftermath of the credit crisis, the popularity and pervasiveness of automated electronic trading continues to grow. There are understandable historical reasons for this, but times are changing and the "old ways" of just a year ago may need to be upgraded to deal with post-crisis reality.

As we know, the very old model of the traditional floor trader, whose success depended on his instincts, his place in the pit and his relationships with fellow floor traders, rapidly gave way to 'black box' trading systems and sophisticated, algorithmic trading strategies. That first generation of automated, trading systems demonstrated beyond doubt that quantitative analysis of historical and real-time market data combined with high-frequency, low-latency order execution could generate considerable profits.

However, the strengths of such systems - their ability to glean complex statistical patterns from enormous volumes of market data and subsequently translate those patterns into thousands of buy and sell signals - proved also to be their Achilles heel as markets plummeted following the demise of Lehman Brothers. When the dynamics of the financial markets abruptly and dramatically changed, many automated quantitative trading systems (that had been back-tested on historical tick data) could not adapt and produced enormous trading losses.

This is a clearly a significant issue. Due to the ongoing dislocations in the financial markets, first-generation systems that rely primarily on the premise that patterns which occurred in the past can be used to identify market anomalies or predict future market behaviour are arguably obsolete. So what do we do?

Towards a new normal

First, rethink normality and apply this to system development. What once was considered normal in many cases is no longer so. The rules governing the financial markets have changed and will change further as governments around the world roll out sweeping new regulatory regimes. The old 'normal', with all its assumptions and preconceptions, is no longer reliable.

To prepare for this new world, developers of the next generation of automated trading systems must look beyond the classic methods of financial analysis and software development to embrace a broader array of techniques and technologies, some borrowed from other fields of science and engineering.
One such technique, which has a track record in the manufacturing industry, is statistical process control (SPC). This is a method by which any complex process that can be characterised by statistical measures, such as defect ratio or average temperature, can be managed so as to ensure that the outputs of the process are of consistently high quality. A process whose outputs do not violate predefined performance criteria is said to be 'in control', while a process that exhibits the converse behavior is said to be 'out of control'.

Figure 1: p Chart for Winning Trades per Day

Figure 1: p Chart for Winning Trades per Day

My purpose in this article is to describe the application of SPC to finance and automated trading systems* by reference to the Excelsior platform. Excelsior is an advanced software platform that supports the construction of next-generation automated trading systems engineered to pursue explicit performance objectives while observing specific constraints imposed by statistical process control rules. By doing this, Excelsior enables a trading system to generate consistent returns in excess of its benchmark (i.e. alpha), while avoiding the pitfalls of first-generation trading systems that cause them to collapse when market conditions deviate from those upon which the system has been trained. Excelsior originated as a graduate research project at the Illinois Institute of Technology's Stuart School of Business and is now available as open-source under the Apache software licence.

Priming the system for SPC

We start with cars. The lecturer, statistician and "quality guru" Professor W. Edwards Deming is traditionally credited with the widespread acceptance of Statistical Process Control theory as a method to improve product and process quality. Deming's work promoting SPC within Japan's auto industry during the 1950s is commonly cited as the greatest single reason Japanese cars are today among the best in the world. Out of Deming's success grew the larger Total Quality Management philosophy and the closely related Six Sigma movement pioneered at Motorola and popularised by General Electric.

What made Statistical Process Control so innovative and transformational was its emphasis on continuous data collection, analysis and proactive adaptation. Quality control that emphasised defect detection through random inspection of finished products was replaced by continuous process improvement as specified and measured by control charts.

A control chart is a simple tool that defines upper and lower specification limits for a particular process parameter and illustrates how the observed parameter value varies over time. The control limits are typically defined to be the point +/3 standard deviations from the process mean. Variation within the process manifests itself as a time series of regular observations distributed above and below the centre line of the chart. A process that is under control will exhibit few, if any, observations beyond its upper and lower control limits. Nearly all observations will fall between the two extremes.

Control charts come in several forms, the most common and popular being the p chart, which is utilised when a particular observation can be classified categorically as either conforming to or violating a specific process requirement. A requirement relevant to a trading desk might be that every trade should produce a positive return. Though next to impossible to achieve in practice, this requirement allows us to bucket every trade into one of two categories, winners and losers. Winners are trades that make money while losers are trades that do not.

Over the course of a given evaluation period, the number of winners and losers on a given trading day can be tracked and analysed by control charts to determine if the trading system is behaving normally; whether it is realising its target rate of return. A p chart depicting the proportion of winning trades over a 30 day period is shown in figure 1 above. How can p charts be incorporated into the architecture of an automated trading system? We'll get to that.

But first - a platform-architecture overview

To appreciate the role that p charts and SPC play in the architecture of an Excelsior automated trading system it is necessary to have at least a cursory understanding of the overall architecture of the Excelsior Automated Trading System platform (See figure 2). The platform consists of four primary subsystems: Core, Market Data, Portfolio and Application. Each subsystem is defined by an abstract API consisting primarily of Java interfaces and abstract classes. Each subsystem has a corresponding concrete realisation which is implemented at the level of the application subsystem in Interactive Brokers' Java API.

Figure 2: Excelsior Automated Trading Platform architecture

Figure 2: Excelsior Automated Trading Platform architecture

Interactive Brokers was chosen as the underlying execution system due to its support for many different asset classes, connectivity to every major global exchange, low transaction costs and extensive trading tools, all of which are exposed through a rich API with multiple language bindings, including Java, C++ and Excel. Excelsior takes advantage of many IB API features including real-time market-data subscriptions, position management, order execution, historical data queries, real-time P&L reporting and more.

The Core subsystem provides basic services to initialise, start and stop Excelsior's market data, portfolio and application subsystems. It also includes a number of utility classes
for logging, object identification, and reference data (e.g. currency codes). Interfaces that define Excelsior's event management API can also be found in this subsystem.

The Market Data subsystem processes real-time market-data streams utilising Esper (http://esper.codehaus.org). Esper is a free, open-source, complex-event-processing engine that enables event listeners, such as trading rules to subscribe to market data and portfolio management events that conform to specific criteria expressed in Esper Query Language (EQL). Esper is an integral part of the Excelsior platform as all market data items, such as quotes, as well as selected real-time portfolio metrics, such as profit and loss (PnL), are wrapped up in Java objects and published as Esper events.

The execution of new orders and the management of existing positions is the responsibility of the Portfolio subsystem. Excelsior's portfolio manager creates and initialises a portfolio per distinct trading strategy for which a new thread is spawned. The thread executes active portfolio management logic customised for each trading strategy. Such logic might include rules for determining when a long position should be closed or when a short position should be covered, for example.

The portfolio management thread is also responsible for continuously monitoring Excelsior's statistical process control state. The state is inspected via an overridable 'hook' that returns a boolean value to indicate whether or not the system is currently in control. If the system is found to be out of control, appropriate application-specific action can be taken to re-establish the stability of the system. Often this will mean rejecting any new trades and alerting a trader or operations desk that the system requires manual intervention.

As previously described, most of the classes defined in the application subsystem inherit from abstract intefaces defined in the Core, Portfolio or Market Data subsystems. This was done to decouple the application subsystem layer, with its dependencies on Interactive Brokers' API, from the common services required to implement any automated trading strategy. The end result is a clean separation between the application layer and the underlying platform services, which will permit Interactive Brokers' API to be swapped out for a different broker's API should the need arise in the future.

Starting with long-short equity

The first trading system built on top of Excelsior is simply named Excelsior I. It implements a classic long-short equity strategy that seeks to profit from taking long positions in companies deemed 'winners', while simultaneously taking short positions in companies deemed 'losers' as measured by recent, historical returns relative to the Russell 2000 Value Index Fund (symbol: IWN).

To both manage downside risk and leverage investment capital, Excelsior I forms its long-short equity portfolio with equity options as opposed to the direct purchase of a company's common stock. This involves taking delta-weighted long and short positions expressed as either bull call spreads or bear call spreads in such a ratio as will achieve Excelsior's performance and risk management objectives. [A bull call spread involves going long the at-the-money call option and going short a higher strike out-of-the money call option on the same underlying equity. Both options must have the same maturity date. A bear call spread is short the at-the-money call option and long the out-of-the money call option. A bear call spread trade is simply the other side of a bull call spread trade.]

Filter rules are utilised to identify candidate trades. Filters select companies to trade primarily on the basis of a company's relative value, market capitalisation and 50-day relative-return momentum. Companies with low book-to-market ratios and recent, strong positive returns (i.e. winners) are held long while companies with relatively higher book-to-market ratios and relatively weak, negative returns (i.e. losers) are sold short. Other technical trade selection filters such as stock price, open interest, volatility and volume are also employed.

The risk factors implicit in this strategy correspond to the Fama-French 4 factor model, an extension of the now-famous 3 factor model. The four factors are the broad market exposure factor (Beta), the value factor (Small Minus Big - SMB), the size/market capitalisation factor (High Minus Low - HML) and the momentum factor (Up Minus Down - UMD). Excelsior I's expected return is designed to be proportional to the weightings attached to these factors, consistent with generally accepted Arbitrage Pricing Theory (APT).

The net performance benchmark for Excelsior I is the Credit Suisse Tremont Long-Short Equity Hedge-Fund Index. As such, Excelsior I should closely track both the publicly available, monthly rate of return (ROR) and Sharpe Ratio for this index.

Excelsior I Portfolio Construction

Assume that Excelsior I's trade selection filters have derived a 'watch list' of winning and losing small-cap, value stocks from Excelsior I's investment universe, the components of the Russell 2000 Value Index. When the system is initiated at the start of the trading day, Excelsior's Market Data subsystem is instructed to create a real-time market-data subscription for each equity in Excelsior I's watch list. As the market-data stream for each subscription is received and processed by Excelsior's event engine, a QuoteEvent object is instantiated and published that encapsulates the open, close, high, low, and last trade prices for the equity as well as associated derived metrics such as the 50 day exponential moving average (EMA), 10 day EMA, and Relative Strength Index (RSI).

To open a new long position, Excelsior I submits a Bull Call Spread order when an uptrend in a winning equity's price is detected (i.e positive momentum). An uptrend pattern can be ascertained when an equity's 10 day EMA is greater than its 50 day EMA and its RSI is greater than 55. This condition can be concisely expressed in Esper Query Language (EQL) as follows:

Excelsior I will put on a short position via a Bear Call Spread when a downtrend occurs, which can be identified with a similarly formulated EQL expression:

In this fashion, Excelsior I constructs a long-short equity portfolio consisting of bull and bear call spreads on small cap value stocks that exhibit strong upward or downward price action and have shown recent positive or negative return momentum, respectively, relative to the Russell 2000 Value Index. Once the portfolio has been so constructed, Excelsior's Porfolio subsystem is enlisted to manage it.

Excelsior I Portfolio Management

A long-short equity portfolio consisting of bull and bear call spreads is subject to a number of risks, many of which can be characterised by sensitivity measures, such as the Greeks: Delta, Gamma, Vega and Theta. The Greeks are an effective way to gauge how a portfolio's value can be affected by changes in market conditions. The Greeks however do not tell a trader whether his strategy is operating normally, as designed. Often the only practical measure of proper system function is whether the system is making money. This works until the system stops making money, which can happen very suddenly and dramatically, and then it is too late to make adjustments to avoid the losses already incurred.

Statistical Process Control, on the other hand, has the advantage of providing immediate and continuous feedback on the performance of a trading strategy. Every time a new trade is done, Excelsior I's portfolio manager updates a p chart that is tracking, in realtime, the proportion of winning versus losing trades. As the thread servicing the portfolio manager loops through the portfolio management rules responsible for opening and closing positions, it also checks the control chart to determine if the system remains 'in control' or not, whether the ratio of winning to losing trades has violated the p chart's upper or lower control limits.

The pseudo-code in figure 3 below illustrates this logic.

Figure 3

Figure 3


If the system is found to be 'out of control', Excelsior I's portfolio manager prints a warning message to the console. In a production setting this message could be replaced by a systems management alert, a flashing icon on a trader's workstation or some other warning designed to allow a trading or operations desk to respond before the system begins losing vast sums of money.

And finally…

Automated trading systems, many of which execute thousands of trades per day, must avoid those six sigma events that can cause a trading strategy to 'blow up', which is exactly the fate that befell many automated trading systems when the credit crisis caused the equity and debt markets to melt down in 2008. Traditional approaches to automated trading system development and portfolio management that rely heavily on the premise of stable, historical statistical relationships are just not up to the task.

Excelsior, through Statistical Process Control, offers a better way. No need to make the dangerous assumption that the past can serve as a proxy for the present or the future. Rather, Excelsior shifts the focus from the analysis of how market inputs, such as security prices, returns and volatilities, affect a portfolio, to the analysis of trading system outputs, such as the proportion of winning versus losing trades over time.

The difference in emphasis, on system outputs rather than system inputs, is subtle but critically important. It is only by continously monitoring the manifest behaviour of a trading system, its outputs, that it can be determined whether the system is operating as designed or whether the system is spinning out of control.

About the Author

Mick Hittesdorf specializes in the analysis, design and development of automated trading and investment management systems. He has over 18 years of IT experience and an MS in Finance from the Illinois Institute of Technology's Stuart School of Business.