The Gateway to Algorithmic and Automated Trading

Commodities on the menu

Published in Automated Trader Magazine Issue 10 Q3 2008

Hard and soft, edible and not-so-edible commodities have been a hot topic recently. But what substance is there behind the sizzle? Automated Trader has been looking into the current commodity boom, and asking, are these markets ripe for automation, and if so, what are their trading characteristics and their long-term potential for automated alpha generation?

In the first of our series of linked articles on this theme, Dr. Robert Brady, CTO of Brady PLC analyses the current use and likely future evolution of automated trading in commodities markets. This is, as Dr. Brady observes, a whole asset class in transition. Some of its trading practices are arcane, but some put it right up on the screen alongside already heavily auto-traded markets such as FX and stock indices. Shouldn’t you be getting a feed on commodities?

Dr Robert Brady, CTO, Brady plc
Dr Robert Brady, CTO, Brady plc

The commodity markets are unique because they contain such a wide range of automation practices - ranging from virtually 100% openness and automated trading in the case of the US exchanges, to partly open processes in the London Metal Exchange, right through to highly manual practices in the commodity industry that these exchanges serve. There is now a pent-up demand to extend trading automation right into the commodity industry. For example, some industry participants who have implemented internal straight-through-processing systems now report that fully 50% of their remaining back-office costs are taken up in correcting manual keying errors, a cost that will be almost eliminated with the ability to agree transactions electronically with counterparties. The existing exchange contracts are over-simple for the purposes of the industry participants because of the complexity of the "real" trades that are executed in the marketplace, away from the exchanges, and there is recognition of the need for new industry-wide standards bodies to define and agree interface standards to meet this demand. As we shall see, these standards bodies are already beginning to emerge.

Historical under-investment in IT systems and standards

Commodities are late to the automated trading party, in part because of historical under-investment in the sector. It is easy to see why the necessary investment in systems and standards needed to underpin automated trading has bypassed commodities until very recently.

Firstly, the extraction business is inherently risky. A mining company typically needs to invest hundreds of millions of dollars to develop an ore body, which will generate future production whose value, when sold, is entirely at the mercy of the market at the time. It is difficult to justify such a risky investment without appropriate risk-reduction transactions, and therefore mining companies needed to enter into financial transactions to hedge their output and to provide certainty to their business model. Fundamentally, the risks are being passed to the banks - and this has occasionally surfaced in high-profile disasters that have set the market back. To take an example, Ashanti was one of the largest gold mining companies in the world in the 1990s. In order to hedge its position, it sold large parts of its production forward (actually it did the equivalent by entering into financial derivative transactions). When the price of gold rose following the Washington accords, the banks called margin on the derivative agreements, of approximately 600 million US Dollars. Ashanti could not fund such large margin calls, and had to enter into negotiation with the banks. This arrangement resulted in a greater caution amongst banks and their regulators in trading with commodity companies, and also a significant reduction in the demand for hedging by mining companies, who perceived greater risks in their financial transactions. Although today Ashanti has come out the other end of this, the effect on the market was substantial.

Secondly, prices were inherently low. Oil was at prices that seem inconceivably low today. Even in the gold market, the cost of extraction often approached closely the price of the product. Economists developed theories about why extraction was inherently low value and would remain so - and today, school students are still taught in economics courses that 'primary' activities such as commodity extraction inevitably have low value, whilst 'secondary' or 'tertiary' activities such as car manufacture

and banking inherently add much more value. This is still often taught as a given, despite the transformation in prices in recent years, and it is at odds with predictions of computer models, such as those underlying the recent book 'The Limits to growth: the 30 year update' by D.H. Meadows, Jorgen Randers and Dennis L. Meadows (ISBN 1844071448) which envisage inherent future shortages of raw materials.

The third factor that set back trading automation in the commodities markets can be summed up in one word: Enron. For example, at the height of the dot-com boom, Enron launched an Internet-based metals contract that took, by some accounts, more that 50% of the trades from the London Metal Exchange. Some hailed this as a transformation in the industry and a harbinger of the way forward. But it later emerged that, behind the modern exterior, the system was almost entirely manual.

Worse (or better, depending on your point of view), the system allowed out-of-date prices to be traded, thus generating a guaranteed profit.

The inevitable demise of this platform coincided with that of several other Internet-based attempts at commodity trading, and has put back automation in the sector for a number of years.

But this is rapidly being reversed. The past few years have seen a complete turn-around in the esteem commanded by companies involved in extraction and the supply chain. Prices have risen - and this is generally recognised to be a long-term industrial trend, rather than a flash in the pan. Investors now recognise the benefit in diversifying at least part of their portfolios to track metal prices.

And this investor interest has ultimately driven a wave of mergers and acquisitions that have created some of the largest companies on the planet. The past few years have seen much higher investment levels in the industry and in the IT infrastructure that it relies on. Although late to the party, we are today seeing initiatives in open standards and automated trading services which will, in the long term, shake the existing structures of the commodity markets to their core.

The US exchanges

At one end of the automation spectrum are the US exchanges such as the New York Mercantile Exchange (NYMEX,, which also encompasses the COMEX contracts). This exchange offers simple, standardised products that appeal to investors and are fully automated. The interface standards for all contracts are based on the FIX protocols, and they are open to all parties. This openness and use of standards has made possible the electronic trading that has transformed the exchange.

Although the contracts on NYMEX appeal to investors because of their simplicity, they do not always match the complexities in the "real" world, where for example different qualities of oil are traded. Here the exchange focuses on 'benchmark' contracts, which are simple and can therefore attract liquidity amongst investors. The downside of this is that the benchmark products traded on the exchanges sometimes represent only a very small fraction of the material that is actually traded in the marketplace. Despite this disparity in volume, it has been found that the benchmark trades can successfully act as a useful reference point for pricing in the industry, with participants basing their prices on the benchmark plus a spread to account for items such as quality or location.

This openness and adoption of standards has led to lower transaction costs, but also to a particular culture at the exchange. Suits and ties, if not actually forbidden, appear to be deprecated. Floor traders still exist, but they wander around the building in sneakers, trading through hand-held electronic devices.

The London Metal Exchange

The leading exchange for metals around the globe is still the London Metal Exchange (LME, The cultural contrast with NYMEX could not be more stark. The LME is one of the few exchanges in the world that still maintains manual open outcry. Today, trades are still executed by gentlemen in suits and ties, sitting down in a so-called 'Ring' (trading in not permitted whilst standing up). They must talk (not shout) with their assistants behind them to get information, because electronic devices are not permitted in the Ring.

Compared to NYMEX, the LME is far less open with information, at least at this stage in its evolution. Members of the LME have access to an automated trading system (called Select) that can feed straight through into their processing systems.

However this is not available directly to end customers. The LME is also adopting standards based on the FIX protocols, in its forthcoming matching system called SMART, but again this is primarily aimed at meeting the requirements of the brokers rather than end-customers.

Behind the scenes, however, the transformation of the LME is under way. The exchange has upgraded its Select automated trading system, which now accounts for over 60% of volume in the main benchmark (3 month) copper contract. Although Select is not available to end-customers (at least at this stage until a decision is made to open up) the conflict with the traditional way of doing things has already become apparent. For example, some members have expressed concern that this electronic platform is taking away liquidity from the Ring, and some have even proposed switching off the electronic system for periods when the ring is in session in order to increase the liquidity! But such responses can only be short-term, and it is clear that the rise in electronic trading is all but inevitable, at least for the most liquid and simple of contracts.

But the LME is also - correctly - being careful not to throw the baby out with the bath-water. There is a reason that the LME is still so dominant in metal trading, despite the fact that rivals in the US such as COMEX have for a long time been offering metal contracts that are more open and fully automated, and hence more suited to the requirements of investors.

Unlike the US exchanges, which tend to start from the requirements of investors, most existing LME contracts are designed from the beginning around the requirements of producers, smelters and fabricators. These are still the major drivers in the industry, and their requirements differ considerably from those of investors. For example, a producer will typically ship several high-value batches of product each month, and consequently will wish to hedge against the average price during the month rather than the price on any particular day. The price of metal differs around the globe, and so those involved in the supply chain will require contracts that take this into account. This is why some of the most successful contracts on the LME are based on the average price for the month, or take location into account. Yet these contracts are significantly more complex than, say, a financial future, and therefore are less attractive to investors. In addition they are more complex to automate.

So the commodity markets face a challenge that is wider than that of financial contracts. Participants need simultaneously to meet the complex requirements of those involved in the supply chain, as well as the simpler and more easily automated requirements of investors who are seeking to diversify their portfolios into commodities. This poses a significant challenge for the information technology, and we shall review some of the responses (including for 'open standards') in the next section.

The jury is still out on who will successfully meet, in a single exchange or system, the requirements of investors on the one hand, and producers, smelters and fabricators on the other. The LME has launched an attempt to extend its appeal to investors with its recently announced LME Mini contract, which is designed to be simple and easy to trade and is targeted primarily at investors who seek to track underlying metal prices. Unlike most of its other contracts, which involve delivery of material, the Mini is cash-settled. If successful it will attract the investor community further into the LME's pool of liquidity which it has established in the industry. However the success of contracts is notoriously difficult to predict or gauge, even some years after their introduction, because volumes often take some years to begin to build up. Currently, volumes in LME Mini contracts are still close to zero and the field is still open for an initiative in the US to attempt this unification, such as the similar miNY contracts from NYMEX, which is described in

Industry participants

At the other end of the spectrum are the commodity trading houses and industry participants such as miners, smelters and fabricators, who cannot merely trade a benchmark but must actually deliver material. Their contracts with each other are complex and difficult to standardise because there is such a wide range of variables - such as delivery location, quality, shape, grade and so on. Even the quantities of material to be delivered are often only estimated, and provisional invoices are raised and paid, which are later adjusted. As a result of this complexity, the process of confirming the contract details with a counterparty is typically fax-based, with manual entry into the relevant systems at each end.

In recent years, many trading houses have implemented dedicated computer systems spanning their internal operations which have reduced their costs dramatically compared to their previous (usually Excel-based) processes. However some trading houses report that, as a result of this reduction in internal costs, an estimated 50% of the remaining back-office costs are now lost in manual keying errors. This error rate could be dramatically reduced through the use of interface standards to allow automated confirmation, and there is a rising demand for these to be implemented, which is driving competing organisations to come together to agree standards.

In order to understand the need for industry-wide standards that extend those used in the exchanges, let's take a look at the requirements of the end customer. A typical commodity trader will trade futures and options in order to hedge his positions. But, important as they are, these are only a part of the business that is transacted. The derivatives are traded in order to hedge "real" trades, which often means some extremely complex transactions on a trader's books. Whilst the trading of futures and options is relatively easy to automate, because they can be represented in a handful of parameters, the underlying contracts may require hundreds of parameters.

Consider for example a contract to ship or sell metal concentrate (the concentrated ore that is dug out of the ground). A typical ship-load will be valued at say $50 million, and there may be multiple shipments in one contract. In addition to the parameters that may be familiar to a futures and options trader, such as the dates and quantities (which, in the case of "real" transactions are specified in terms of acceptable ranges rather than specific dates and quantities), the contract details need to specify basic items such as the content of the load (a concentrate load may contain a mixture of several metals, such as gold and lead) and any penalties (if for example there is arsenic present). Each of these parameters, including the quantity, is provisional, and subject to amendment at a later stage for various reasons, such as moisture content. To this trade complexity, one must add further details such as those for treatment and refining costs, freight, insurance, etc. as well as charges that are dependent on the content of a particular element, or the level of an underlying price. Although there now exist straight-through transaction processing systems to handle these transactions, it is easy to see why manual errors throughout the process now account for so much of the cost and why there is a demand for interface standards to agree these details electronically.

In some sectors, there are initiatives by system suppliers who have the detailed knowledge of the complexities involved and who are able to establish provisional communication standards allowing their customers to automate the confirmation process with each other. However, by far and away the best way forward, at least in the longer term, is ratification of the interface protocols by a standards body so that all industry participants may benefit, not just those from a single supplier.

The next few years will see the emergence of new interface standards, agreed throughout the industry and policed by standards bodies, that will unify the industry and dramatically reduce costs. In some cases they will be based on provisional standards implemented by system suppliers, which have been adopted by the standards body; in other cases they will be the result of trading organisations getting together to agree standards for their in-house systems and then including the interface standards in their requirements from suppliers.

A number of standards bodies are already emerging, such as LEAP ( which aims to establish standards in some parts of the the energy sector, beginning with electricity trading. In LEAP's model, confirmations can be agreed on a peer-to-peer basis if required.

In the past few years, the XML standard has been incorporated into the FIX protocols for trade exchange, and this seems to be the standard that is receiving fastest adoption in the commodity industry. The New York Mercantile Exchange has adopted it for its trading protocols. The London Metal Exchange is still using older protocols but has announced that it will start to adopt the standard for its forthcoming SMART system for registering trades. And in the energy sector the interchange standard for electricity contracts, which is being defined by the open standards body, has also adopted and extended the standard.


The commodity markets are in a state of transition. Today, there is a spectrum of openness and standards, ranging from the US exchanges at one end, which are founded on the principle of openness of information and which are generally focussed towards the investor community; through to the end-users for whom effective standards remain to be established to cut their costs substantially, and who are beginning to establish standards bodies to achieve this. Sitting in the middle is the London Metal Exchange, with its internal automated trading methods that have yet to be opened up to end users. The next few years will see substantial redefinition of the industry, based on open standards and automated trading, which will perhaps define the markets for a generation.