The Gateway to Algorithmic and Automated Trading

ICAP + EBS = Joined up data

Published in Automated Trader Magazine Issue 05 April 2007

The explosion in automated and algorithmic trading has been accompanied by a huge thirst for data - market participants want data, lots of data. And they want a broader range of it - the days of providing just traded prices are long gone. Today’s market wants volume, historical order book depth, and order message traffic – the works. AT takes a look at how EBS and ICAP’s latest historical data packages measure up.

From a data management perspective, one of the biggest changes brought about by automated/algorithmic trading has been participants' attitude to data collection. In the past, exchanges and others collected traded ticks and volume data and in many cases just discarded everything else. Particularly as the desire to trade at the short end of the time spectrum has grown, these trading venues have had to revisit this collection strategy. Unfortunately, some have only started to do this in the last year or so, which means that those looking for a long term "rich" dataset are going to be disappointed.

However, pretty much since their inception, some trading venues have kept virtually every possible piece of historical data - with ICAP and EBS being two prominent examples. (Although ICAP now owns EBS, their historical data offerings are currently still separately branded, so just for the purposes of this review they are treated separately.)

ICAP Information

ICAP Information supplied a DVD of US Treasury data for review, which contained a year's worth of Brokertec data (all transactions/activity in 2005) for 2, 3, 5 and 10 year Notes and 30 year Bonds. Traders and quants looking for granularity and detail won't be disappointed. Transactions are flagged as being in one of three categories - order book change, trade or workup. The first and last categories will particularly appeal to a number of quants AT has been talking to recently - much of the research in high frequency finance is currently crying out for this sort of data. These quants don't want to see just the original trade; they also want to see the additional activity (if any) that took place around it.

The granularity within each of these three categories is also pretty impressive and goes a long way beyond the usual standard dataset of bid/ask, volume and traded price. For example, the trade category includes a deal source field indicating whether it was an outright trade or a swap leg (and if so, which leg). The order book change category adds even more, with a complete table of twelve change codes that explain the reason an order was changed, the size of its change, and its position in the order book. In the case of the workup category there are additional fields for workup state (finished/private phase/public phase). A nice additional touch across all categories is that the date/time stamps in the ICAP datasets go down to the millisecond level.

EBS Data Mine

Both versions of the EBS Data Mine were supplied for review. One version simply consists of all the data in text format. The other version includes the same data, but also adds a rather neat little data viewing tool called Data Miner (see figure 1).

Figure 1 - Data Miner

The Data Miner is not intended as an analysis tool, but more as an easy way to visualise the data in various ways. Users can select from any dataset (all EBS currencies plus gold and silver are included), choose the display duration and interval and opt whether to show/hide/collapse weekends (see Figure 2).The Data Miner also includes a secondary display which can show price activity (all new bids and offers for the period), or deal activity (all new deals given, or taken for the period) or volatility.

Whenever time periods longer than one minute are being viewed in the primary display, the panel to the right of the display defaults to showing the opening bid/offer, closing bid/offer, high and low for the period (see Figure 1). However, if the time interval displayed is one minute, clicking on the primary display switches this panel to the view shown in Figure 3. This shows all the quotes for the minute centred on the selected time. Bids and offers are shown from the point of their publication to their removal as blue (bid) or green (offer) arrowheads. Given and paid quotes are represented by pink and gold boxes at the time and place they occurred.

Figure 2

The actual data in the Data Mine package is supplied as a series of files each containing all data for a week for all instruments in one file. (ICAP Information by contrast uses one file per instrument per day.) This makes for some fairly hefty file sizes, but with the latest release of Microsoft Excel™ now supporting 1m rows, it is still possible to open and manipulate the files easily with a standard desktop application.

Figure 3

Each data row consists of date, time, currency pair or precious metal symbol and a field indicating whether the data row represents a quote or an actual deal. If that field shows a deal then the next two fields represent the given/paid price, if it shows a quote they denote the bid/offer. The last two fields in the row show the volume category the deal given/paid or quote bid/offer the transaction falls into. Precise volume is not shown; instead the volume is graded into one of seven "buckets" lettered A-G, with the thresholds of these volume buckets varying according to instrument.

So for example…

…denotes a deal done in EURUSD at 1.3285 with the paid price and size showing that the taker was a buyer of somewhere between 16m and 20m (the "D" category volume bucket) at this price.


To the casual observer, a couple of data packages may seem nothing to get excited about, but nothing could be further from the truth. One critical point is that these two datasets derive from the dominant primary markets for the instruments concerned; they therefore (in a sense) represent the net global consensus for pricing in those instruments. Another is that they are providing just the sort of granularity that the marketplace is looking for. Finally, there is the fact that while fixed income and FX markets are closely related, authoritative historical datasets for them have in the past been very separate and not easily acquired.

Therefore the prospect of them both being offered by the same organisation in a detailed and increasingly homogenous form will appeal strongly to both single and (especially) multi asset class traders.