The Gateway to Algorithmic and Automated Trading

Do Black Swans Swim in Dark Pools?

Published in Automated Trader Magazine Issue 09 Q2 2008

The continued rise of dark liquidity and the development of more sophisticated execution algorithms could increase regulatory obligations on brokers and/or exchanges, according to Mary Lou Von Kaenel, Managing Director, Management Consulting, Jordan & Jordan, and Greg Malatestinic, Senior Software Engineer, Jordan & Jordan.

In the wake of regulatory changes aimed at price transparency and best execution, and as market participants wade through their post-Reg NMS and MiFID implementation issues, many wonder: where has all the liquidity gone?

Securities regulation in the United States and Europe has placed a spotlight on trade price reporting and quotes, and has set the stage for competition among exchanges, multilateral trading facilities and electronic communication networks. Focused on ensuring that investors receive the benefit of fair and competitive markets, the US Securities and Exchange Commission (SEC) and the Committee of European Securities Regulators have reduced barriers to entry in support of off-exchange trading with the introduction of Reg NMS and MiFID respectively, to accomplish their objectives.

The rule makers have taken different tacks to hit their marks. MiFID is principles-based, placing the onus on market participants and their clients to agree on practices and procedures for trade execution that consider not only price but speed and likelihood of execution, as well as other costs related to execution and settlement. However, the SEC has charted a more direct, rules-based course for orders and trades, insisting under the Order Protection Rule (Rule 611 of Reg NMS) that the top-of-book displayed quote in each protected market must be hit before a transaction can be executed at an inferior price.

The SEC also requires all trades and quotes to be reported on a consolidated system in order to facilitate the exchange of information needed to support Reg NMS. In the UK and Europe, however, market participants have complained of data diffusion and increased market fragmentation. With no rules under MiFID requiring consolidated data or exchange linkages, accessibility to complete data - in addition to direct connectivity necessary to reach more market centres and a broader array of settlement venues - has increased costs and offset some of the benefits of exchange competition. Problems in Europe have also been cited with respect to trade reporting, particularly the lack of timeliness of block trade reporting and issues of duplicate reporting.

Murky waters

Despite regulatory efforts to increase transparency, the tide appears to be turning toward murkier waters. In the US, many large blocks are trading in 'dark pools' where orders are obscured from view. As orders sit in dark pools waiting to be matched, and as others may be passing through, quotations are not displayed. Thus, investors are protected from the ripple effects of information leakage. Large block orders with the potential of creating significant market impact are finding safe havens in electronic dark pools today, replacing the trusted upstairs traders in the yesteryear of floor trading.

Dark pools are proliferating with more than 40 in the US currently, although depth is critically important to become a serious source of liquidity. Electronic access to dark pools and use of algorithms by the sell side, as well as through buy-side order management systems (OMS) to find natural matches and trading opportunities, have streamed liquidity to the dark pools. Beyond simply internalising client orders in a proprietary environment, brokers have begun to actively seek partnerships and linkages with other dark pool providers in order to enhance liquidity and extend the likelihood of quality executions for their buy-side customers. The widespread use of the FIX protocol enables easy interaction across these as well as exchange trading venues. These linkages may have the cumulative effect of creating a 'super-pool' which lies outside the jurisdiction of any regulatory body.

Industry consultants estimate that, in the US, 12-15 per cent of equities trading volume is off-exchange currently, and the trend toward exchange disintermediation is predicted to reach 20 per cent in the next two years. As exchanges see liquidity and market share flowing toward alternative trading systems, the response for some has been to forge relationships and develop hybrid products that will meld the concepts of dark pools and displayed order books. For example, ISE's MidPoint Match provides the opportunity for matching before an order is publicly displayed. Similarly, BIDS and NYSE Euronext have announced their intention to develop a product that will incorporate display, reserve and dark liquidity to allow block trades to be executed through continuous order matching and negotiation.

"Despite regulatory efforts to increase transparency, the tide appears to be turning toward murkier waters."

With only half the number of dark pools in Europe compared to the US, the off-exchange market is less mature and the concept of linkages among proprietary venues has not yet taken hold. As market participants consider their options in defining best execution for different types of orders, Chi-X Europe has been successful and new market entrants including NYFIX Euro Millennium and Turquoise are expected to be well received. The hybrid model offered by Turquoise should satisfy European traders looking for alternatives that will re-aggregate liquidity, with the added bonus of centralised clearing and the opportunity for reduced settlement risk and costs.

Flow control

To the extent that current regulation is directed toward price transparency and markets are becoming more fragmented, the exchanges are less able to absorb large orders without sending signals through the market. To curtail market impact, market participants have employed algorithms to spray orders across multiple venues, slice blocks into small bits and carefully drip orders into the market. While simple VWAP and implementation shortfall algos remain popular when executing certain types of exchange strategies, they are not effective for large blocks that must get done quickly or for trading in the less liquid mid- and small-cap stocks. To achieve rapid execution of large block trades with minimal market impact, more sophisticated algorithms, with stylish names such as Covert (EdgeTrade), Sniper (Credit Suisse), Sonar (Goldman Sachs) and Ambush (Bank of America Securities), have been developed which specifically search dark pools for hidden liquidity.

“In the US, many large blocks are trading in dark pools where orders are obscured from view.”

"In the US, many large blocks are trading in dark pools where orders are obscured from view."


Algorithmic trading is becoming more complex, stretched by the need to comply with Reg NMS's Order Protection Rule and best execution requirements, and also considering market fragmentation, recent market volatility and burgeoning market data volume. In many respects, automated trading using algos and smart order routers enhances a dealer's ability to meet Reg NMS compliance obligations. The rules-based engines provide definitive documentation as to trading policies and procedures being applied under particular circumstances as required by Reg NMS. Moreover, electronic messages capture each order and trade, and provide the necessary audit trail for analysis and regulatory reviews. The SEC recommends a minimum of three days' trade and quote data be stored to ensure policies are appropriate and being followed to prevent trade-throughs. Capture of the data processed by the algo/router platform can facilitate proactive surveillance and support analysis of trade-throughs and exceptions on an ongoing basis.

Algos are frequently given by dealers to buy-side clients they may sponsor for direct access to market centres and liquidity sources. In granting direct market access (DMA), dealers retain responsibility for trade compliance, regardless of whether orders are being generated and entered manually or using automated technology. Given that algos have become mainstream, regulators are becoming increasingly concerned about their potential effects on electronic markets. Market participants are being reminded of their obligations to supervise any order that is routed to a market centre, and their responsibility for any trade that may be executed as a result. For example, Canada's Market Regulation Services Inc. issued Market Integrity Notice No. 2008-003 in January, regarding 'Supervision of Algorithmic Trading', to "provide guidance on the supervisory requirements under the Universal Market Integrity Rules of a Participant with respect to the use of an algorithmic trading system and certain limitations on the ability of Market Regulation Services Inc. to intervene to vary or cancel trades arising from a malfunctioning algorithmic trading system."

Testing the waters

The validity of regulators' concerns of algo 'malfunction' is interesting to consider, given the plethora of programmes that hit the markets with hundreds of thousands of orders each day. Algos are often developed by the sell side to handle both clients' orders and proprietary trading. In many cases, dealers provide algos to their institutional clients for use directly or integrated with the clients' order management systems. Large buy-side firms also build proprietary models of their own, and order management system vendors frequently offer their standard off-the-shelf algos as well as the opportunity to integrate proprietary models.

Typically, deployment of algos presents a time-to-market issue with resources required of both the dealer and the OMS vendor to implement algo strategies for each of their buy-side users. Every time providers want to add new algo models or refine existing strategies, they need to change their specifications and wait for the clients and their vendors to code and release new versions of the interfaces. FIX Protocol Ltd (FPL) has recently addressed the inability to deploy algos quickly with the introduction of FIX ATDL (algorithmic trading definition language), an XML-based standard used to describe algorithmic order types1. This standard was designed to expedite implementation of algos on an order management platform without requiring detailed knowledge of the underlying event processing models. This enables new models to be plugged in without extensive customisation or integration efforts.

Given the current popularity of algo trading and the relative ease with which new algos are expected to be deployed going forward, perhaps we should prepare for a flood of new development and inexperienced users. Are there adequate procedures to monitor or control algo behaviour? Can it be guaranteed that an algorithm won't malfunction or be misused? From the regulators' perspective, should there be some level of oversight of all algorithms to minimise the chances of a 'runaway' algorithm disrupting the markets?

There is generally an expectation that some type of governor is applied to each algo application that will: a) prevent orders from being over-executed; b) make sure that limit prices on orders generated are within a reasonable range based on current market conditions; and c) monitor volatility and perhaps even suspend orders when markets become too volatile. Yes, we expect that the professionals have built in alert mechanisms for overfills, watch for runaway trading conditions, monitor credit and trading limits, apply risk controls, etc. But in a world of commercially available complex event-processing engines (CEPs) where end-users write their own strategies in a vendor-created proprietary language - is there a risk that a user could mistakenly write nefarious code resulting in unintended behaviour - an infinite loop, perhaps?

"Are there adequate procedures to monitor or control algo behaviour? Can it be guaranteed that an algorithm won't malfunction or be misused?"


Older algos are comparatively simple. Fed by market data and explicit processing rules, they have no real understanding of market conditions or market colour. However, extensive modelling, research and backtesting is now underway to look at broad sections of the market to predict trading patterns and analyse behaviour of single stocks relative to their peers - across sectors and across markets and instruments (e.g. futures, bonds) - to better understand the impact of various influences on stock prices. This second generation of trade analytics and algo strategies is making use of continually updated market information and trading patterns to redefine and adapt trading strategies in a dynamic environment rather than based solely on static rules. New development is enabled by available technology to capture, store, analyse and manipulate massive amounts of data.

Making waves

To be certain that these algorithms are well-behaved and do not pose the risk of disrupting the markets, we must rely on backtesting, the same technique used to ascertain the effectiveness of an algorithm. During backtesting, historic tick, quote and depth-of-book data are fed into an algorithm with the actual results being compared to the expected results. Tests using sequences of data from multiple timeframes are run in order to assess how an algorithm would have performed under a broad range of market conditions. Many commercial CEP engines will provide access to data sets for

backtesting scenarios such as bull or bear markets; but these represent only a finite set of scenarios. Other scenarios, relevant to a particular algorithm, may be missed. Could we do better by stress testing with more scenarios? Maybe so - synthetic ones could be developed - but there are an infinite number of these and at some point the returns from continued testing begins to diminish.

A major failing of backtesting is that it can't assess the market impact of an algorithm. At the time the historic data was captured the algorithm did not yet exist, so, for example, if an algorithm were to hit a bid or take an offer it would have no effect in the subsequent tick and trade data - the top of the book would remain the same. To provide a more rigorous testing framework, a market simulator may be used in which an algorithm sends orders and receives executions. Beware, though: the simulator itself may have been created and tested using historic data and may be biased towards the same historic scenarios used in backtesting. Thus, it would be prudent for a user to be aware of the strengths and weaknesses of the simulator to ascertain its effectiveness.

"We need to be comfortable that algo applications are being built with integrity and tested within a rigorous automated framework against a comprehensive set of conditions using extensive data."


Ultimately, first-class software engineering practices should keep most poorly-behaved systems from ever reaching production, but in the event of unforeseen conditions, one would expect notification of error conditions and the opportunity for immediate intervention. We need to be comfortable that algo applications are being built with integrity and tested within a rigorous automated framework against a comprehensive set of conditions using extensive data. How confident are we?The black swan

The Black Swan

'Black Swan theory' has gained popularity recently in its general application to risk analysis and risk management following publication of Nassim Taleb's book, The Black Swan: The Impact of the Highly Improbable (Random House, May 2007). The metaphor dates back to the 17th century when black swans were found to be living in Australia. Previous to the discovery, conventional ancient wisdom dictated that black swans "could not possibly exist". Taleb uses the black swan analogy to argue that some event that is either extreme or completely unexpected, and therefore untested, will have a high impact (positive or negative) on a situation's outcome.

In case of a 'Black Swan event', it is difficult to predict how algorithmic trading applications might behave, since in theory the particular set of circumstances has never been considered in models or testing. Where does responsibility lie to ensure there is an appropriate fail-safe mechanism? Assuming the dealer exchange member remains obliged to police any orders being entered on that exchange (whether by the dealer itself, a correspondent broker or an institutional client that the dealer has sponsored for direct market access), this might suggest it is the dealers' responsibility to 'certify' any algos being employed, regardless of who developed the algo or the platform that sent the order. Or, reminiscent of the circuit breakers2 put in place at the New York Stock Exchange after the crash of 1987, is the onus ultimately on the market centre to prevent the drastic effects of a black swan event? While this is an interesting topic for debate, for the near term, we can only hope there are no black swans swimming in dark pools or displayed markets.

1.See Leader article in Q4 2007 issue of Automated Trader by John Goeller, Chair, FPL Algorithmic Trading Group

2. In response to the market breaks in October 1987 and October 1989 the NYSE instituted circuit breakers to reduce volatility and promote investor confidence. By implementing a pause in trading, investors are given time to assimilate incoming information and make informed choices during periods of high market volatility. http://www.nyse.com/press/circuit_breakers.html