The Gateway to Algorithmic and Automated Trading

APIs: Real World Modelling

Published in Automated Trader Magazine Issue 21 Q2 2011

Something of a change in this issue’s software review, with a former Automated Trader First Person (see page 5 of the Q3 2010 issue) and previously anonymous Wrecking Crew member taking centre stage. Yang Wang uses multiple APIs for programming trading models and tools both for in house use at Chiron Investment LLP and for distribution to the clients of Financial Software Engineering Ltd, where he is currently Head of R & D. As a result, he is ideally placed to give his views from the sharp ends of the eSignal, CQG and STS APIs.

I should start by making clear that this isn't a software review in the usual sense. Rather than systematically stepping through and evaluating all the functionality of the eSignal1, CQG and STS (Schneider Trading System - formerly known as Stellar) APIs, the aim is to share some real life experiences (some good, some less so) with these interfaces that other programmers and traders are likely to encounter.

Horses for courses

First up, none of these APIs are suitable for high volume, high frequency trading; if your trading strategies are highly latency-sensitive you'll be better off with a dedicated infrastructure capable of supporting this requirement. The possible exception is the STS API, but only if a limited number of symbols/markets are involved; if you need to work orders on spreads involving several hundred symbols, while simultaneously scanning an even larger universe, dedicated infrastructure is the way to go.

If on the other hand your models are being applied to perhaps ten or fifteen instruments and generating trades with holding periods of minutes rather than seconds, then any one of these APIs should be perfectly capable of supporting your needs.

What do you get and how do you get it?

If you're using an API you obviously need to be able to trade and manage your orders, but you may also be looking for historical data both for simulations as well as to initialise any real time indicators your models use. On this point, both eSignal and CQG provide historical data, plus a charting and technical analysis GUI with which to display and manipulate them. Unlike eSignal and CQG, STS is a dedicated trading platform. It doesn't offer historical data - all you get are real time quotes from the markets to which you are connected, so it's a case of build or buy your own historical database.

The first major distinction to draw among these APIs is the underlying technology. Both eSignal and CQG use standard Windows COM technology to connect to the API, which certainly flattens the learning curve. STS by contrast requires writing unmanaged C++ to communicate with the API, which is a more efficient mechanism but is also considerably more demanding in terms of programming expertise and time. For example, Figures 1a to 1d illustrate an STS API data request for the current live price for GBPUSD. Comparing this with Figures 2a and 2b which show eSignal and CQG API data requests highlights the different approaches that C++ and the .NET framework take to connecting to APIs.

However, much as low latency and a highly efficient trading system sound appealing, much depends upon the exact design of the strategies and order routines, as well as the software and hardware infrastructure.

figure 1a
figure 1b
figure 1c
figure 1d

Without proper planning, efforts targeting theoretical supremacy can easily end up not only failing to deliver the desired results but incurring long development cycles and large costs as well. If you're implementing relatively simple or reasonably latency tolerant trading models, the overhead of using C++ probably won't be justified and a COM based API would probably be the better route to take, while for anything more complex it is worth investing the time to do it all using C++.

One way of reducing the C++ overhead (and one that we have adopted) is to use the managed version of the language, which is part of the .NET framework. This also makes integration with code written in other .NET languages considerably easier. In our case, we use C# for any GUI components because so many of these are available prebuilt and take a fraction of the time to implement that they would in C++.

Intriguingly, eSignal's API was originally also C++ based before migrating to COM and in fact it is still possible to use this as an alternative development path. However, it appears that eSignal is deprecating this, so obtaining C++ API support documentation may become more difficult in the future.

CQG's implementation of COM is extensive, while eSignal still has a few rough edges. For example, CQG automatically handles COM registration at API installation, but eSignal instead attempts to do this the first time the software is run. In Windows 7 this immediately bumps up against the default security measures, which block this without administrative privileges. Hardly a show stopper, in that you only have to run eSignal once as administrator to circumvent the problem, but slightly irritating nonetheless.

COM limitations

While a COM based API may be simpler to work with than C++, it has some significant practical limitations, with debugging probably topping the list. If your own programming experiences tally with the common approximation that 30% of time spent is on coding and 70% on debugging, then COM can quickly become irritating. This is because if your code triggers an error into the COM interface (rather than in your application) this will often be untraceable if the code behind COM is unmanaged code, i.e. not within .NET framework, which is the case for CQG. As a result, standard mechanisms such as using try catch loops in order to capture the error simply don't work. In fact, none of the exception handling in the .NET framework is likely to assist you - all you will get is a crash. This means that you end up trying to debug an error that you cannot actually see and about which you have no information.

One of the commonest ways of ending up in this situation is with an index out of range error. For example:

• If you have ten orders in the order book and you inadvertently try to access an eleventh order (i.e. you reference array index eleven in the code)...

• ...and the real order object is held behind COM by the API provider (which is generally the case)...

• ...the exception will occur behind COM and you will have a crash on your hands.

The main problem here is that the resulting out of range error happens behind the COM interface of your application. Under these circumstances the whole COM interface crashes and you lose connectivity to the market, but without any clear indication as to why. The net result is that you have to be scrupulous to the point of paranoia in checking the legality of every single indexing reference in your code before you pass it to any array that belongs to the COM interface.

Data

As already mentioned STS does not provide any historical data, while CQG and eSignal both do. Figures 2a and 2b show how to request the most recent 1000 bars - with live updates - for the front month of the Bund contract via the eSignal and CQG APIs respectively. The CQG code fragment is longer because it sets a number of additional data control variables (more on the granularity of CQG data control later) which can be safely omitted if you do not need to change them. (If omitted they automatically set to default values.) As can be seen, their COM interfaces mean that both eSignal and CQG are rather more succinct than the C++ that STS requires when it comes to coding standard data requests.

figure 2a
figure 2b

We have found CQG has some slightly irritating peculiarities with regard to historical data. One example is its habit of randomly refreshing its historical data; in the CQG GUI this manifests itself as all the bars disappearing from a chart and a reload message appearing, but this random reloading also affects the API in that it empties the data buffer. Having established a data channel connection you might assume that you still have the (say) 100 bars originally loaded - but you don't. Therefore if you try to reference any of these bars you get an index out of range error, because as far as COM is concerned you are trying to access a nonexistent index (which is one of the commonest reasons for unhandled system crashes among CQG API beginners).

The safest remedy for this behaviour is to buffer historical data as soon as it is initially received by copying it into a new variable, thereby effectively creating your own data buffer. Having done this, we have discovered that the best approach as regards updates is to make a separate call for two bars of data. This is because requesting just one bar runs the risk of missing any new data points that arrive while the data request is being processed.

CQG can also become hung up over delivering historical data for a single symbol both in its GUI and via its API and curiously enough this does not appear to be related to the amount of data being requested - even quite short data blocks can prove randomly problematic. If for some reason it cannot find or load the data from the remote CQG server it will simply hang with a data refresh error message. Unfortunately, restarting the CQG client application does not necessarily resolve this issue, because you will often be logged back into the original server that had a data problem in the first place. This issue can cause significant aggravation when trying to initialise models at the start of a trading session. On several occasions we have had our trading disrupted because models were unable to subscribe to the necessary CQG historical data they required in order to initialise and start trading.

Unlike CQG, restarting eSignal does help if you are experiencing data problems. But you need to be aware that the application you need to restart is not the eSignal client but the eSignal Data Manager. But by default, if you simply close the eSignal client, the Data Manager will not be closed.

CQG tick data does not suffer from these issues because it appears to use separate channels and servers for ticks, although you may still wish to maintain a local buffer copy if your strategy needs to reference historical price ticks. However, it is important to be aware that CQG never clears its tick buffer unless specifically instructed via the API. If you don't factor this behaviour into your code, the buffer will continue accumulating ticks until you run out of memory and experience a crash.

Interestingly, CQG doesn't offer any functionality to clear the tick buffer; instead, you have to destroy the data channels and reconnect them to clear the buffer. However, if all you require is the current traded price, you are not obliged to use CQG's tick channel, which is designed to provide every single tick (trade, offer, bid) going through market.

Users of CQG also have to be mindful of the way in which it handles missing data values. Instead of leaving an empty data slot or inserting a NaN (not a number) it uses a large negative number to fill the gap. You therefore need to use the IsValid function provided in the CQG API to validate all data you receive from CQG - otherwise you will find yourself inadvertently participating in some very 'interesting' trades.

As long as you remember to use the IsValid function, CQG's method of data validation is extremely reliable. On the other hand, eSignal fills empty data slots with zeros, which is a far less robust approach since zero is a valid price in many contracts. This necessitates writing a custom validation routine for checking all zeros received via the eSignal API.

However, eSignal in general provides less problematic historical data access than CQG, with far faster initial loading of bars, as well as automated clearing of its tick buffer. On the other hand, setting up eSignal's menu to control access to historical data is not very intuitive, with a multitude of options (such as session times and rollovers) that require checking (see Figure 3). Furthermore, eSignal does not allow you to configure any of these data options programmatically through the API; everything must be set up manually in advance through the eSignal client application.

figure 3

There are also practical limitations on eSignal's ability to handle large quantities of data. If for example we load ten timeframes apiece for seven symbols (70 data channels) we will almost inevitably encounter a couple of moments in each trading session (typically over key economic releases) where eSignal starts consuming well over 50% of available CPU (dual core, 2 x 3 GHz). This limitation is caused by the threading structure of eSignal which will be discussed later. To be on the safe side we have found it prudent to restrict the total number of historic data channels used by eSignal to a maximum of 15-20, and not to load all the data channels at once. By adopting this approach it is possible to prevent CPU overloads.

Instead of eSignal's rather laborious menu driven approach, CQG allows full control of all data types programmatically through its API; basically anything that can be configured in relation to data manually in the CQG client application can also be programmatically manipulated through the API. The amount of instrument information CQG provides is impressively comprehensive and includes all individual session times and tick sizes. This means that one can actually create a universal strategy where all that is required to apply it to another market is simply to input another symbol. No other alterations are required, as all the instrument-specific information can be retrieved automatically from the CQG API. This makes it possible to have a truly fully automated system that requires only minimal supervision; even contract rollovers can be unattended if the model is correctly coded. For a desktop scale data provider, this is impressive.

STS's lack of historical data means that you will either have to access this data elsewhere or build a complete data gathering application (a major undertaking) and wait for STS to populate it or use a different API completely. However, STS's real time quotes and full depth of book are extremely quick, significantly faster than either CQG or eSignal's. This has actually prompted us to consider the possibility of combining data from STS and one of the other two providers. A very simple breakout example would be for the breakout threshold to be calculated using CQG or eSignal historical data but for the actual trading trigger to be tripped by a tick from STS. As a rough estimate, this might constitute as much as a 500 millisecond execution speed advantage.

Threading

The three APIs all have different approaches to threading. STS offers true multithreading by assigning independent threads to you at the server level, which allows for complete segregation of data and order management activities. Every time you create an instance at the server it is assigned a unique key that references the actual thread at the server. The level of independence and security this approach delivers is important; if you are running multiple trading models, no one model's programmatic misbehaviour can affect anything else (as long as the server survives!). Furthermore, unlike the other two APIs, STS's approach effectively assigns you a completely segregated virtual server of your own, so the order/data resources allocated are totally dedicated to your application. This is a particularly important consideration; with the CQG and eSignal APIs you are to some extent at the mercy of other traders, such as if one (or more) of them makes a massive data request.

At the opposite end of the spectrum from STS, eSignal only gives you a single thread to work with that is also shared with the client application's main thread. This has a number of negative implications. Firstly, the number of securities that can be traded is constrained because the data and order channel share a single thread, which means that anything that causes a delay causes that delay universally within the application and API, affecting everything. Secondly, this use of a single thread also affects resilience - if any one trading model throws a serious error it can crash all other models and the entire application (client GUI and API functionality).

A further major drawback to this approach is that being single threaded, there is only one data pool, which means that all applications have to share that single common data pool.

This can cause 'data interference'; for example, if one application clears its data connection, the same data connection in all other applications is also cleared. But if one attempts to avoid this situation by not destroying data connections then memory management becomes problematic, which can result in a rather messy scheduling framework.

CQG's approach falls somewhere between STS and eSignal in that it allows multiple instances to run on independent threads that all communicate with CQG's single central main thread. Therefore, these threads are independent at the local level (i.e. at the client application level running on your workstation) with each having its own data pool. Thus if one trading model using one thread falls over, other models and the main application will be unaffected. (The same 'independence benefit' applies as regards delays.)

However, all the individual 'model threads' are managed by the central thread and are therefore dependent upon that. As a result, a problem there - such as the data retrieval delays mentioned earlier - will propagate to all the 'model threads' and cause universal disruption.

Order control

When it comes to controlling orders, we have found STS by far the most comprehensive. An additional benefit is that the API offers a range of prebuilt synthetic types. For example, order cancels order (OCO) types that are not even offered by some exchanges are available off the shelf, with all the intricacies of managing the synthetic order handled automatically on the STS server. This is a significant time saver, when you consider that writing your own OCO order from scratch would probably require a couple of pages of code. STS also offers automated icebergs and a very convenient mechanism for submitting and managing small orders as 'bucket' orders. (See Figure 4 for a listing of STS order types.)

figure 4

STS also gives you direct access to your own order book so you actually have direct control of the real order objects (or their references) on the server. This direct access means that any order updates and modifications your code makes will be processed rapidly .

The other two APIs are less powerful in this respect. For example, CQG gives you the full order book of each instance, but it is actually a copy of what is in the real order book. This means that you have to go through a couple of gateways in order to make any changes to the real order book, which increases latency - potentially by tenths of seconds. eSignal basically works in the same manner with less ease of control. This can have a significant impact if you are covering multiple markets (such as when trading spreads) and therefore makes STS's approach extremely appealing.

Time to market

In an area as competitive as trading, time to market is an important consideration and there are a number of aspects of each of these APIs that have a bearing on this. Firstly documentation; getting up to speed with an API depends heavily on the quality of the documentation provided, such as the number of examples provided and the degree of detail in explaining functions and concepts. This is an area where CQG scores strongly by providing comprehensive coverage of syntax and copious code examples (see Figure 5). eSignal comes a definite second in this respect; while it offers sufficient basics to get you started, you still have to do quite a bit of experimenting.

figure 5

STS comes a very distant third in the documentation stakes; no help files as such - just a few examples - otherwise it's a case of reading through all the header files. Therefore, STS API users will need to be capable of independent problem solving. The paucity of STS's documentation would be problematic even for a COM-based API, but is doubly so given its dependence on C++ and definitely materially slows the development process. One would normally expect to see more not less documentation for an API that used a more complex structure requiring C++. This documentation issue coupled with STS's dependence on C++ means that unless you are a highly skilled C++ programmer it will take you appreciably longer to get a model to market using it than CQG or eSignal.

Of the two COM-based APIs, CQG stands out in terms of API object-orientation. It is very well structured so everything has an object, which in the current .NET framework makes it very easy to program and easy to trace in terms of debugging. All data streams in CQG are wrapped nicely into objects with well structured iterators and attribute structures, while eSignal doesn't present data in integrated objects - instead requiring users to fetch data on a piecemeal basis. Therefore, more than often, it is necessary for eSignal API users to create custom data objects for efficient data iteration.

The other area where CQG scores in terms of productivity and in time to market is in allowing you to access custom studies you may have already written in CQG's client interface using its native macro language via the API. With both eSignal and STS everything has to be (re)coded from scratch, so this ability to immediately reuse existing custom studies could (depending on the number of such studies involved) represent an appreciable time saving. However, and especially with more complex custom studies, it is advisable only to do this with low frequency models.

Conclusion

As stated at the outset, this isn't a conventional software review and so there isn't a 'winner' or 'best' API. Which API (or possibly even combination of APIs) you choose, will obviously depend upon the exact nature of your trading. (Our organisation is slightly unusual in that it is producing trading models and applications for a wide range of users and therefore is actually using/evaluating three different APIs.)

Table 1 provides a breakdown of our view of the three API's capabilities in certain key areas that have a direct bearing on the processes of trading model development and deployment. Below Table 1 are listed some generic types of trading model and market combinations, together with the API that (based upon our own experiences) we feel would be most appropriate for each combination. In making these suggestions, we have tried to select the API that would be capable of running the models/markets concerned in a production environment, while simultaneously minimising project and software costs.

eSignal CQG STS
Typical Project Budget2 Low Reasonable Comfortable
Chart/Historical Data Access Yes Yes No
Typical Time to Model Delivery Short Flexible Long
Typical preferred minimum holding period Minutes, with occasional seconds Minutes, with occasional seconds Suitable for desktop scale high frequency
Parallel trading systems feasible? Extra data scheduling required if sharing market data Suitable Good
Typical order routine complexity Low Low High
Typical computation intensity of trading model Low Medium High

Table note: The term typical in the table assumes structuring the trading model in order to make the maximum use of the relevant API's functional capabilities, with the intention of minimising both project budget and time to market. However, this performance can of course be enhanced (possibly significantly) by writing additional extension code, should project budget and life span permit.

• Generic trading model and market combinations:

• 60 min moving average based trading model applied to a single market: eSignal

• 60 min moving average based trading model applied to multiple market: CQG

• Spread trading of currency derivatives using a midsized (50 x 50) spread matrix: CQG or STS

• Spread trading of currency derivatives using a large (100 x 100) spread matrix: STS

Dr. Yang Wang

Dr. Yang Wang

Author Biography

Dr. Yang Wang received his B.Eng. in automatic control systems from the University of Zhejiang Technology, China, in 2002. He completed his M.Sc. in digital control systems at the University of Hertfordshire, UK, in 2004 and in 2008 received his Ph.D. from the University of Hertfordshire for research into the Autonomous Learning of Appropriate Social Distance by a Mobileobot.

His research interests include machine learning, artificial neural networks, Human-Robot Interaction (HRI) and social robotics. He joined Chiron Investment LLP in late 2008 and started to utilise his academic interests in trading futures. Since then, he has been working on automated trading systems and strategy analysis and is now the Head of R&D at Financial Software Engineering Ltd.

Footnotes

1 The API discussed here relates to eSignal versions 10.6 and below; at the time of writing eSignal v11 did not yet offer an API.

2 In the context of a small desktop-scale automated trading system.