The Gateway to Algorithmic and Automated Trading

LMAX Exchange: Agile Challenge to the Status Quo

Published in Automated Trader Magazine Issue 27 Q4 2012

LMAX Exchange is an exponent of the agile development methodology, but does that agility also extend to the business as a whole? In this issue's Exchange Views, David Mercer, CEO, Andy Phillips, Head of Systems and Mike Barker, Head of Software, describe how LMAX Exchange has evolved both rapidly and sometimes in unexpected directions over the five years since its inception.

Andy Phillips (Left), David Mercer (Middle), Mike Barker (Right)Andy Phillips (Left), David Mercer (Middle), Mike Barker (Right)

What were the origins of LMAX Exchange?

David Mercer: It was born from the Betfair new ventures team in 2007, which was trying to take maximum advantage of the very strong exchange technology they had already developed. To put it in perspective, Betfair matches 7,000,000 orders a day, which is more than all the European stock exchanges put together. So it was hardly a massive leap to take the decision to look at mapping that technology to an exchange for financial trading.

LMAX Ltd was the result of this decision and LMAX Exchange was launched at the end of 2010. Initially, it didn't really get the traction it deserved, perhaps because it was directed only at a Betfair style audience, who would probably be best described as retail users. By April 2011 it was increasingly apparent that the product actually had a much broader scope than that, and that in terms of functionality wasn't a purely retail product in the same way as the Betfair exchange.

So how did you respond to that?

David Mercer: We felt that we had a very fast, robust and scalable exchange, which would be appropriate to a much broader range of clients in terms of size and sophistication. I joined LMAX Exchange at the end of April 2011, and at that point we took a close look at who the target client segments should be, which products we should focus on, and how we should be structured internally. On the basis of that re-evaluation, we re-launched in September 2011.

Although we are still relatively small, we have grown substantially, so that we now often match $3 billion in a day and $30 billion in a month. Liquidity has increased due to a growing number of General Members (in LMAX Exchange parlance market makers are General Members). We initially had three, but now have eight, and that will soon be in double figures, which we expect will deliver better top of book pricing and liquidity.

Our main traction in the marketplace currently seems to be with retail brokers. However, because of the exchange's functionality and range of markets, we are also seeing a growing number of money managers, funds and professional traders. So we would now classify ourselves as a professional exchange rather than the direct to retail-only organisation we once were.

Since late 2011 we have also been working on building out the product set, which now consists of more than 70 products. Obviously, growing the number of FX pairs has been a major part of that, but we've also added spot gold/silver and the major indices.

How do you handle the addition of new markets?

Mike Barker: We still run all markets within a single matching engine and we have spare capacity, so adding another market maker or currency pair can be done without any performance hit. While we have a lot of clustering for resilience, with one instance of the engine doing the primary matching - and multiple replicas keeping copies of that state elsewhere.

Andy Phillips: We do have the ability in the architecture to share either by account or instrument or market type across multiple engines, but we haven't had cause to use that capability as yet.

Andy Phillips (Left), David Mercer (Middle), Mike Barker (Right)

So how is your matching engine structured?

Andy Phillips:We run on Dell servers using standard Red Hat LINUX. In terms of distribution, we have real-time offsite replication for disaster recovery and as Mike mentioned we have a degree of replication within the application itself to ensure high availability. There isn't clustering in the traditional system administration sense; it is actually handled at the application layer.

What sort of functional trading performance does this deliver from a user perspective?

David Mercer: Excellent matching performance: we can match over 20,000 orders within a second, with average execution speeds of less than 3 milliseconds (measured as the roundtrip from the gateway, through the matching engine and back to the gateway).

We would regard our idling speed as some 500 to 600 messages per second. As regards total trades, we probably transact about 20,000 per day, although we have in the past matched 40,000 trades in one second.

You mentioned Dell servers earlier, why did you choose them?

Andy Phillips: On the server hardware side we've used HP in the past, but more recently we have re-

evaluated the field and have gone with Dell. This was for a variety of reasons, but mainly related to selecting the right CPU. (For us, Sandy Bridge processors from Intel offered a number of very important benefits.) When it comes down to it, servers from HP or Dell or Sun will all be very similar. The only differences that occasionally arise are if you go to someone like IBM or Cisco UCS who will have some slightly different silicon in the box that allows them to do some unusual things.

Are you CPU based for everything you do or are you using, or considering using, GPUs or FPGAs?

Mike Barker: It's an interesting question. GPUs don't really fall into the matching engine space, they would be more appropriate for the heavy duty floating point calculations needed for more complex risk modelling or algorithmic models. Matching is all about fixed point arithmetic and our risk model isn't really complicated enough to require GPUs. We can do it plenty fast enough using CPUs.

However, FPGAs are a slightly different case, because you can try to do pretty much anything with them. They are not something we've looked at yet, mainly because the development turnaround cycle can be quite slow for them and we push very hard for fast turnaround. However some of our vendors are looking at FPGAs for things like FIX parsing. The sort of activity that is really heavily commoditised is probably where we would look at them, but at the moment we have no plans.

Andy Phillips: I would echo that from previous experience of using FPGAs. If you can break a problem down into a small piece that fits well into (and can easily be coded up in) VHDL or C or another low-level language, then that is great; we're not quite at that stage yet - in many cases we are evolving our software quite rapidly. We've only been live for just over a year and a half so there are a lot of changes we still need to make.

At some point when we really start to go head-to-head with some of the equities exchanges then I think we'll get to the point where we'll begin to consider whether it makes sense to encode functionality down at the hardware level, but right now it's not really on our roadmap.

Andy Phillips (right), David Mercer (Middle), Mike Barker (left)

Andy Phillips (right), Mike Barker (middle), David Mercer (left)Andy Phillips (right), Mike Barker (middle), David Mercer (left)

LMAX Exchange is active in the open source community, but in what areas?

Mike Barker: Our involvement in open source is pretty active and we use a lot of open source technology day to day, such as our use of LINUX as a core operating system. We also use an open source web server as well as a large number of standard open source libraries, such as Spring for enterprise Java development (as mentioned earlier, Java is our main language). We tend to try and push back a lot of what we do into the community. That usually takes the form of smaller patches and changes.

Probably the most significant thing we've done in open source has been the LMAX Exchange Disruptor project (http://code.google.com/p/disruptor/). That came about because when we started looking into the performance of our software system and analysing where the costs were, the thing that jumped out at us was the cost of queuing - moving messages between threads.

The cost was probably a factor of 100 to 1 (cost of queuing versus the business logic). So we spent some time optimising that and managed to come up with a pattern that worked very well for us. It has an element of multicast in it, so you can do certain unrelated tasks in parallel. It also gives you the ability to coordinate those tasks, so you effectively have almost a graph of dependencies of actions that should occur.

In an industry usually highly concerned with protection of intellectual property, open source participation seems slightly incongruous. Do you feel it is beneficial?

Andy Phillips: Yes, you often find that there is rather a clash of cultures in the industry between the more freewheeling Internet side of a firm and people on the finance side. I think, if you look across the industry as a whole, you often find completely unnecessary, obsessive secrecy actually seriously damaging businesses. For example, consider how many times people have completely unnecessarily re-implemented FIX servers (usually poorly) - there has to be at least one per institution.

As far as we're concerned if something is not part of our core intellectual property, but we find it useful and we think others might too, then we will consider it for open sourcing. We have quite a few active contributors to various open source projects (in addition to the Disruptor project) working for us across our technology teams.

The advantages for us of open source are many fold. Firstly, we tend to get slightly better quality of coding. Also, if an engineer realises we are open sourcing a project, you would be amazed how rapidly things like intelligible documentation are added. That's a direct benefit, but there are also a lot of indirect ones as well. If something does take off, like the LMAX Exchange Disruptor has done, we find that its performance is actually increased significantly, because a lot of people contribute to it - both within the financial industry and beyond. That has expedited the progress of Disruptor, which obviously benefits us because we use it extensively.

But I would say that probably the biggest thing about open source for us has been the doors it's opened, especially when it comes down to recruitment and getting really good quality people. I participated in a conference last year in America which was completely online, and everybody there had heard of the LMAX Exchange Disruptor project. It was a real door opener to people working for companies like Amazon, Facebook and Google.

So would you say that open source involvement has helped LMAX Exchange recruitment?

Andy Phillips: Yes, it has really helped to put us on the map, which has made it very easy for us to attract the right sort of developers and technologists. You obviously need to make accommodation for people's contribution to open source projects in terms of company time, but their output has been extremely useful to us as a technology team and to the company as a whole. We will certainly be looking to continue our open source involvement, and have two or three other candidate projects that are non-core which we will probably look to open source in the next year or so.

Would you say that open source equals open mind in terms of the people you have hired as a result of this policy? Does that mean that valuable things are drawn to your attention that might otherwise be missed?

Mike Barker: Yes, I definitely think that's true. It certainly fosters a lot more discussion about tools and technologies we could probably use and adopt. In a way, we are a little bit conservative compared to the standard web company. Open source can be something of a double-edged sword, because you can end up with people rushing off to apply every single new tool that appears out-of-the-box, which results in a rather messy patchwork of different bits of open source. So we do try quite hard to have a degree of consistency in what we do. Nevertheless, we benefit from fostering discussion, looking at innovative things that are emerging and challenging ourselves. For example, if somebody wrote something faster and better than Disruptor - then we'd use it.

Has there been anything that has come from the open source community slightly out of left field that you have picked up on and are actually using? Something that you would probably never otherwise have come across?

Mike Barker: I think one of the ones that is least known that I have found extremely powerful is a Google open source product called LevelDB, which is very similar to Berkeley DB. We have managed to build one of our internal tools that we use for monitoring on top of that for its historical store and it has worked brilliantly for us, but it wouldn't have been the obvious first choice.

Andy Phillips: I would second that, but also add that you sometimes get benefits even when you don't actually decide to use the code. For example, there is a piece of open source software called StatsD which was open sourced by a company called Etsy in New York, which is a simple daemon for easy stats aggregation. Although we don't actually use it, we read through it and it sparked off quite a lot of valuable ideas about how we should actually handle data, especially monitoring data.

What about the network side?

Andy Phillips: As regards switches, we are looking at going with Arista for our technical refresh that will happen towards the end of this year, as they can provide low latency switching down in the nanosecond timescale. Up until now we have been using Brocade, because three or four years ago when we did our last evaluation they could offer some very large Foundry cut-through switches (as opposed to store and forward), which we needed.

For network cards we did have a look at Solarflare, but at the moment we've gone with Intel 10 gigabit cards. If you go with Solarflare then you obviously need to go with open onload, and at the moment our latencies are around about 1 millisecond, so the potential saving isn't really significant. Hence our decision to go with the Intel cards because they are adequate for our current needs. In any case, should it prove necessary in the future, upgrading network cards isn't a massive task and can be completed over a weekend.

What network protocols are you using?

Mike Barker: We use FIX externally, but internally it is all messaging bus-based, so it is pure multicast across-the-board. We use a product called Ultra Messaging from Informatica, plus our own internal binary format - so we don't use Java serialisation or Google protocol buffers or anything like that - we have our own marshalling so we can write our own objects directly onto the wire.

And what about preferred programming language(s)?

Mike Barker: Pretty much everything we write is Java, though we will have the odd other bits and pieces mixed in, such as HTML and JavaScript, and some of the products we use are C based (such as our messaging library). This focus on Java means that there is a fairly consistent set of skills for which we are recruiting. So having one language rather than three or four makes it quite easy to find people with the appropriate abilities and move them between teams as necessary.

Nevertheless, it is an interesting challenge to build high-performance software in Java. Sometimes you have to step outside the box and do some things that are a little bit off the wall instead of using approaches that might be considered classic best practice in Java.

For example, when it comes to high performance code, Java has a couple of niggles. One is garbage collection and the other is lack of control. Somebody who is a C or C++ developer has an advantage - not necessarily because those languages are intrinsically faster, but because of the level of control they give you over the underlying system.

One of the biggest things in terms of software performance at present is memory layout, and in Java you don't have a lot of control over that. Therefore you sometimes need to find ways to avoid garbage collection, such as by creating something upfront and keeping it around, which is atypical of most general good practice for Java. Another thing is that in order to obtain some of the benefits of improved memory layout when defining objects, you may have to use non-mainstream libraries. The underlying issue is really all about working around the inability to assign particular items to particular memory addresses. In that respect it's about being cache-friendly and stride-friendly for the CPU.

You mentioned earlier that swapping out network cards wasn't a massive task, but what about upgrades more generally?

Andy Phillips: From a technical perspective, there is nothing practical stopping us doing major system upgrades (such as adding in core servers) overnight. However, as a production site we are fairly conservative, so we tend to do that sort of work over the weekend. We can bring things up live in real time on the broker side and use standard F5 Networks' load balancing technology for that. On the internet side we use something that is a bit more custom but that allows us the ability to swing FIX servers into service live as well. I'd like to say it also allows us to swing them out of service too, but due to the nature of the FIX protocol that's not necessarily possible without cooperation from the people connected to a gateway to reconnect to a different one.

How do you work the gateway architecture in terms of allocating gateways and how many do you have per user or group of users?

Andy Phillips: That depends very much on the type of customer connecting. If they are General Members, then we like to allocate specific resources to them. If they are broker members or customers using FIX, then we essentially connect them to a small number of IP addresses and use load balancing to distribute those amongst broker FIX servers.

What about automated trading levels?

David Mercer: Over 70 percent of our trades come via API, though that includes retail brokerages who may actually be consolidating that order flow from manual traders. We certainly expect that to continue to increase, and we also notice quite a lot of that sort of automated activity at the retail end of the market where there seem to be plenty of people buying EAs (Expert Advisers) and plugging them into MetaTrader - we also offer direct MetaTrader connectivity to the exchange. LMAX Exchange is the first exchange accessed directly from MetaQuotes.

How many API options do you offer?

David Mercer: Three - or four if you include the option to write directly to our own protocol, which is available using either JSON or XML. Everything that is visible in our web user interface is also accessible via that protocol including trading, market data and also account management information, such as current position levels at both order and aggregate instrument level.

We provide two APIs in .NET and Java, which are essentially wrappers built on top of our own protocol. The .NET API is used by MultiCharts to create the user interface into our code. The final API is FIX, which was obviously required for our General Members, but is also used by the brokers, as well as some funds and high net worth individuals.

You seem to have little difficulty in attracting new market makers. Why do you think that is?

David Mercer: I think that initially it has been the flow that as General Members they can tap into from second and third tier retail brokers. This is despite the fact that our average ticket size is probably lower than they experience as market makers elsewhere because our minimum (one contract) order size is only $10,000, as opposed to the more traditional minimum of $100,000 or $1 million at the EBS level.

Another factor may be that they can see the direction the industry is going. The way that regulators are looking at issues such as pre- and post-trade transparency makes it seem likely that FX will increasingly trade on-exchange over the next decade.

Could a large buyside firm actually become a General Member of the exchange?

David Mercer: Why not? If large buyside firms stream two-way limit orders, they could be General Members as well.

It sounds like the of LMAX Exchange's business philosophy has carried over from Betfair and originally also some of the technology, but is that still the case?

David Mercer: No - although we are still owned by Betfair, our technology is now completely different and separate. This includes our hardware, and we also have a completely separate development team of 22 agile developers (soon to grow to 25). That is in addition to another 20 or so doing general technology work such as information systems, networks and connectivity.

Quite a few exchanges have struggled in managing the ratio between the number of messages users transmit and the number of trades they actually complete. How do you handle this?

David Mercer: Currently, we do not specify a rule for a ratio between the two. At present, as long as you trade with us you can send as many limit orders as you like. However, because we are relatively small, I don't think we've really suffered from people trying to abuse the system. If that eventuality should arise, then maybe we will have to introduce those ratio rules. I suspect we will have to at some point, because just about every other material venue has had to do so, but so far we haven't seen a practical need.

We can handle a lot of messages anyway - 20,000 per second - which is perhaps one reason we haven't felt particularly pressured as yet. However, because we are a regulated MTF (currently the only one in Europe for FX and CFDs) we have to have extensive real-time market monitoring and control systems in place, which allow us to track messages per second per client. Therefore, if any user started to abuse message volumes to the extent of causing disruption, we could shut them off immediately.

While on the subject of message volumes, how would you classify the typical trading activity of your buyside users? HFT?

David Mercer: No. I would say the majority would be intraday but certainly not what you would classify in equity market terms as high-frequency. We typically have participants who trade hundreds rather than thousands of times a day. Some tend to focus on country-related strategies - for example, we've seen some trading a lot of Swiss and Canadian crosses lately. We've also seen a few people starting to undertake more complex strategies involving multiple instruments simultaneously, although I wouldn't say that this was a significant part of our flow as yet.

As we stand, I think we can compete as an attractive venue, although having larger average ticket sizes wouldn't go amiss. That's why we see larger buyside participants as very much a target segment for us, and we would like to increase their participation significantly. The potential is certainly there when you consider that of the $1.5 trillion per day traded in foreign exchange, $800 billion is interbank, but some $700 billion is retail and institutional.

lmax office

How does your choice of location fit with that?

Andy Phillips: We have just moved to LD4 (Equinix in Slough), which offers cross connects into NY4 (Equinix in New York) but is also the de facto hub for FX trading in the UK. Trading participants that are latency-sensitive tend to be in there anyway and can then cross connect to us. Others, especially the larger brokers, are interested in connecting to us over one of the various financial extranets available, such as TMX Atrium or BT Radianz.

How are your users distributed globally?

David Mercer: It's interesting in that fewer than 5 percent are in the UK and Ireland. The next smallest percentage is Western Europe, followed by Eastern Europe and Asia-Pacific who together represent the vast majority (around 30 percent apiece).

I think it's probably because, being relatively small, our first growth area was in brokers connecting over API. It so happened that their clients were predominantly in Eastern Europe and Asia-Pacific. Our target is to grow by a factor of four over the next 12 to 18 months. If we achieve that, then the geographical split might well change. Nevertheless, we are still focused on those two core regions because they seem to be more open to the idea of exchange based foreign exchange trading. They seem to be ECN and exchange led, while by contrast Western Europe seems more stuck with the OTC market-making way of trading FX.

There is also an extremely strong retail FX trading culture in markets in Eastern Europe and Asia-Pacific - especially in China and India. However, Japan is obviously also hugely attractive to us when you consider that it alone represents 30 percent of the total retail market for foreign exchange. Furthermore, a recent market survey revealed that something like 2.5 percent of the Japanese population has traded FX compared with around 0.4 percent in the US.

Finally, given the way your technology has changed in such a short space of time, I'd be interested in your tech philosophy. Do you think in terms of roadmaps or is it more ad hoc than that?

Andy Phillips: We run very much with an agile methodology across the board. So we are all speaking the same language internally, be it on the commercial side of the house or technology or operations.

We divide our world up into different milestones, four milestones per year. We only really start detailed planning at the start of each milestone, so each three-month period. From there, things break down into individual two week iterations, which is when the detail of individual items of work are organised. So the way we work very much helps us both be quite reactive and nimble in terms of taking advantage of new opportunities, as well as being sure that we have several different planning horizons ranging across two weeks, three months and then the next year or so. It tends to be a bit hazy when you go above three months which is really as it should be.

Mike Barker: We tend to try to focus very much on the business as regards aligning our development. So we're trying to slot into the way the business works as regards the big technical decisions we make, such as separating out teams. These will always fall out of the business direction.

In terms of pure technology strategy, it is for me always about simpler and faster. Every time we come to a significant decision point, then it will typically involve taking something out and making it quicker, rather than adding something. It's surprising, during normal development, how complex the system becomes. Then later on you have to stand back and look at the bigger picture, and you realise how much you actually don't need in there. So for me in terms of strategy, it is all about less is more.

Andy Phillips (middle), David Mercer (left), Mike Barker (right)