The Gateway to Algorithmic and Automated Trading

Open Source for Auto/Algo?

Published in Automated Trader Magazine Issue 10 Q3 2008

Open source software has made huge strides in terms of credibility in recent years. So is now the time to consider it for auto/algo trading? Graham Miller, CEO of Marketcetera, definitely thinks so.

By Graham Miller, CEO of Marketcetera
By Graham Miller, CEO of Marketcetera

Picture this: you want to build an automated trading system to take advantage of ongoing price discrepancies in US listed equity options. This system will allow you to scale up the strategy to a significant portion of your P&L, so clearly you will code this in-house with your most trusted software developers. Open source software is probably the furthest thing from your mind. However, as the project gets underway, you are surprised to find that much of the code that will ultimately make up your trading system is not proprietary, and may even be available as open source software.

The reality

Actually, not so surprising at all; a quiet move towards open source is occurring in financial services, initially at the infrastructure layer and progressively, even in finance-specific functions like FIX connectivity and basic risk-analysis. Increasingly, software adopted from open source projects can provide the flexibility of in-house code, while providing time-to-market advantages of proprietary third-party software. For example, JP Morgan based OneBench - their internal application development environment - on the open source Eclipse application framework, because it provided the efficiency, lower cost and control they required as the foundation of their front-office trading and risk system.

On the buy side, the advantage of open source is even more compelling, as the agility of trading organisations is constrained by smaller development and IT organisations. Instead of siphoning precious internal development resources off to re-implement common application infrastructure components, the buy side benefits when engineers and quants focus on the proprietary portions of your automated or algorithmic trading stack where real returns are produced from highly optimised implementations of your strategies.

Everyone now knows about the success of the Linux operating system and MySQL database server, but today, there are a variety of additional open source projects that could jumpstart development of an auto/algo trading system and provide substantial economic and time-to-market advantages. The following provides a survey of some of the most mature and promising open source initiatives available that are potentially relevant and valuable to buy side organisations. These projects generally proffer extensive documentation and maintain active developer communities, enabling thorough software evaluation and creating a rich environment to support fast implementation (see box: 'Tyre kicking').

Active MQQuickfix AboutQuickfix/J

Messaging & Middleware

Automated and algorithmic trading systems increasingly require communication across multiple applications on multiple machines with high throughput. For example, consider risk managers observing real-time trading, trading assistants entering orders for traders and clusters of pricing servers sharing data. In addition, pricing logic is increasingly separated from FIX gateway software, and executing applications must ingest blotter data from an OMS. These types of interactions lend themselves to an event driven architecture, indicating the use of messaging middleware that provides publish/subscribe support, manages latency and uses point-to-point guaranteed delivery semantics.

ActiveMQ is an open source messaging system that provides a messaging bus with an impressive array of features (http://activemq.apache.org/). Supporting several transport protocols, persistence, load-balancing, failover, and integration libraries for many languages, ActiveMQ rivals the offerings of many of its proprietary competitors. It's easy to setup and can actually be embedded directly into server applications written in Java for tighter integration and better performance.

AMQP (Advanced Message Queuing Protocol) is a messaging protocol designed to enable highly interoperable middleware messaging and replace proprietary protocols like those used by TIBCO's Rendezvous. AMQP was initially developed by JP Morgan Chase and further championed in the open source community by financial institutions such as Credit Suisse and Goldman Sachs. The Apache Foundation (the same people that developed the world's most widely used web server) is running an AMQP project called Qpid (http://cwiki.apache.org/qpid/). Apache describes Qpid as "a messaging implementation built on AMQP that allows one to implement just about any distributed or business application. Qpid provides transaction management, queuing, distribution, security, management and heterogeneous multi-platform support."

Quant stuff

While messaging infrastructure can deliver the buy side market data, transport orders and fill information across various internal systems, there are a couple of projects that can provide a model development environment for both internal pricing and trading.

The R Project (www.r-project.org) may already have a presence in your quant group. As described by the R community, "R is a language and environment for statistical computing and graphics. R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible." Quants can use R's statistical tools to develop and test trading strategies, and it can also form the basis of the runtime logic behind those strategies.

QuantLib is aimed at providing a comprehensive software framework for quantitative finance including modelling, trading, and risk management (http://quantlib.org/). In building an automated or algorithmic trading system, one may choose to implement custom proprietary pricing algorithms, or risk analytics. But, if access to implementations of well known quantitative finance tools is required, QuantLib may have what is needed. QuantLib has C++ libraries with interfaces to the majority of popular languages for a host of models and tools including pricing engines, pricers, financial instruments, yield term structures, volatility and more. Unlike proprietary code, all submissions have been peer-reviewed by an active community of academic and industry quants and statisticians.

Tyre kicking

Your organisation probably already has a process in place to evaluate commercial software for trading systems that includes phases to identify, research, compare, and analyse available options. Today, open source packages should be included in this process if there is one that meets your initial requirements.

Evaluations of open source projects, however, differ from their commercial counterparts in the amount and type of information available in comparison with proprietary competitors. For example, a structured sales process is unlikely to accompany an open source project, but you will probably receive a wealth of technical details. The technical pointers of a good open source project will help you understand the project's intent, status, history and contributors, which can guide your decision whether to entrust the software with your trades.

Some key indicators of an open source project's viability, stability and performance include:

Activity

The best indicator of success in an open source project is active usage and development across a broad community. Open source projects often give you several statistics to help judge activity:

• Frequency of versioned releases - good development organisations, both proprietary and open, release software on a regular basis. If the latest release of the project was more than a year ago, beware! This project may not be going anywhere.
• Mailing list traffic, community developed documentation - look for active participants in mailing lists and forums, and see if questions and concerns are addressed in a timely fashion. Also look for other community generated content, like Wikis and tutorials for evidence that people are using and receiving value from the product. Finance-specific projects often have lower participation rates, so look more for openness and responsiveness from the project leaders.
• Frequency of source control additions/changes - the ultimate definition of 'actively developed'; consistent and growing numbers of additions and changes in the source control system indicate the true health of the project. Tools like SourceForge and FishEye offer statistics and visualisation for this type of change.
• Multiple contributors - some smaller (and some even not-so-small) projects depend on a single individual for the majority of development work. This single point of failure may not be ideal for code
on the critical path of an organisation and should be carefully considered before moving forward.

Reliability

To judge the level of reliability of the product, evaluate the development tools and community focused on Quality Assurance. Does the project have a unit test suite? What about integration testing? Other automated testing tools? Is there a bug database? Are bug reports on the mailing list addressed in a timely fashion?

Another indicator of completeness and reliability is the version number. Open source projects are rarely anointed as Version 1.0 until the product is quite stable, though version numbers may often progress quite quickly thereafter. You may not want to deploy a fully-automated trading system on top of a pre-1.0 codebase, but look at the bug database, feature requests, and ask questions of the developers to determine future possibilities.

Commercial Support

Look for commercially offered support and/or maintenance contract options. These can lower the burden on your internal IT staff, ensure ease of future migration and often provide you with unprecedented access to the developers behind the software itself. Increasingly the organisations that already support your trading desk software are willing to support open source packages as well.

Interoperability and Standards

Look for evidence of interoperability with existing and planned future system implementations. Integration points can often be found in the data model, import/export tools, 'public' APIs, and implemented standards. On the buy side, the FIX protocol is king. Recently developed by the same organisation, the FIX ATDL (Automatic Trading Definition Language) allows you easier integration with your broker's algorithms.

Hands-on

There is no substitute for simply doing a proof-of-concept implementation, or a prototype application. Hands on experience will shake out issues around usability, completeness, and feature set that may not be obvious in discussion forums, documentation and code samples. Often you can structure this initial implementation as a performance test, generating baseline latency and throughput numbers to guide your technology adoption decision. In addition, the depth and complexity of the application you are able to produce in a time-limited proof-of-concept exercise will allow you to estimate the level of effort needed for your full development project.

Graph 1: Number of source lines of code (SLoC) in about 5000 of the most active open source projects
Graph 1: Number of source lines of code (SLoC) in about 5000 of the most active open source projects

System Implementation

With the pace of product innovation and intensity of buy side competition in trading, the need for auto/algo trading systems to reach the market quickly and inexpensively increases daily. Several open source efforts aim to increase the financial viability of rapidly launching and maintaining internally developed trading systems. These include providing open source FIX implementations and open source versions of the common components of the trading platform itself.

Graph 2: Historical shares traded by automated and algorithmic systems on the NYSE and American Stock Exchange
Graph 2: Historical shares traded by automated and algorithmic systems on the NYSE and American Stock Exchange

QuickFIX and QuickFIX/J are solid implementations of the FIX protocol in C++ and Java respectively (http://www.quickfixengine.org/, http://www.quickfixj.org/). Exchanges now integrate FIX connectivity natively into tradebook systems, providing performance meeting or exceeding proprietary protocols for order entry. As a result, FIX is becoming the lingua franca of finance and will likely supplant any other standard.

The QuickFIX libraries handle the underlying connection and session logic, providing total control of the application data communicated to counterparties. Fairly low-level, these libraries can be used to produce anything from simple drop copy loggers, all the way up to complete trading systems.

Flexible future

To stay competitive and generate alpha, trading systems must be flexible, agile and constantly adaptive to new market conditions. Trading strategies may have a half-life measured in months as increasingly sophisticated market participants quickly squeeze the edge out of even the cleverest of strategies. A trading system must, therefore, allow regular adjustment and updating, and the rapid addition of new features, strategies, and asset classes.

Using open source software, buy side organisations gain the ability to implement new trading strategies and build new features on an internally driven development schedule, without the delays inherent in relying on a third party software vendor's release cycle. Today, development teams supporting traders can build a prototype of an application on ActiveMQ in the time that it would take to negotiate a licence with a vendor for a product like IBM's MQSeries. The 'openness' of open source enables fast technology evaluation cycles, and dramatically shortens the time-to-market for developing new trading systems.

The benefits of open source will ultimately persuade all but the largest and most conventional firms on Wall Street to abandon trading systems based on wholesale internal IT efforts or proprietary systems from commercial vendors, and turn to using the timely and cost-effective strategies offered through this community-based approach. While many myths must be dispelled before wide-scale adoption occurs (see box: 'Open Source Mythology'), those that innovate early will see real returns in opportunity costs of development time and business scalability.

Open Source Mythology

Apart from its actual capabilities, open source software has unfortunately also acquired some negative and inaccurate mythology. Five of the worst culprits are:

• Open source means no secrets: many people believe the use of OSS will lead to others learning their proprietary trading information. The truth is open source does not require you to disclose anything about your trading. You can download an open source package, build a trading system, and support it - all without telling a single soul outside your organisation. Unlike commercial software, there are no sales teams to inform, no license compliance checks to complete. You choose when and how to use the software.

• Open source is unsupported: a common concern is lack of vendor support for OSS products. Increasingly open source projects are backed by one or more companies that are in business to provide support, maintenance, and upgrades. Support contracts on the open source project or a separate commercially-licenced product based on open source are commonly available. Companies such as RedHat, MySQL (Sun), and XenSource (Citrix) have been providing these services for years and building viable businesses from the effort.

• Some licenses are anti-intellectual property rights: a common misconception is that open source software subject to GPL requires any source code from derivative works to be 'given away'. The truth is an end user of GPL software is not required to disclose anything; only organisations that build and redistribute a product based on GPL code may be required to disclose source code.

• Open source software is not quality software: another common concern is that proprietary software will always be more secure, reliable and perform better than an open source alternative. This topic causes many heated debates, and will probably continue to do so in view of the commercial interests at play. However, what is clear is that good and bad software exists on both sides of the argument. Before building a trading system, or any other operational system, careful evaluation of the software is required. Happily, evaluating open source software can be easier, as until you sign on the dotted line proprietary software firms are highly protective of their products, APIs, and documentation before you sign on the dotted line.

• Open source means community service: many people believe the use of open source software requires them to 'participate' in the coding community - this is simply false. Many open source communities are built on the assumption that some fraction of the users will 'contribute' to the community by filing bugs, suggesting features, or even writing code, but there is absolutely no requirement for participation. In most cases, more than 90% of users of an open source product will simply use the product, and never participate in the community.