Andy Webb: How did the "arrowhead" project come about?
Yuichiro Yamamoto: By 2003 it was apparent that our legacy exchange matching system for equities was experiencing capacity constraints. The TSE embarked on a series of upgrades to enhance capacity, but it was nevertheless apparent that a new trading platform would ultimately be required.
The formal planning and request for proposal (RFP) process for this new platform began in 2006, but prior to that we visited numerous exchanges around the world to assess the available exchange technology already deployed in the market place. Our objective was to identify the most cutting edge technology currently in use, but also to extrapolate that information to gain an insight into probable future developments and requirements.
We announced the RFP and vendor selection process in August 2006, before selecting Fujitsu Corporation as the developer for our next generation trading system in December 2006. Exchange staff (with input from market participants) then worked closely with Fujitsu over the next three years in developing "arrowhead", which subsequently went live on January 4th 2010.
Andy Webb: Did your selection of Fujitsu as developer imply that the TSE had already made a conscious policy decision to build rather than buy the new trading platform?
Yuichiro Yamamoto: No, not at all. We opened the door to all categories of vendor for competitive tender, but we simply took the view that Fujitsu's proposed solution offered the best overall package.
Andy Webb: What does the "arrowhead" platform consist of?
Yuichiro Yamamoto: The platform was designed as an integrated whole, with Fujitsu providing both the hardware as well as the software. (See Figure 1 for a breakdown of the hardware and middleware used in "arrowhead".) The matching engine runs on Fujitsu PRIMEQUEST mission-critical servers, which use Intel Itanium 64-bit processors.
"arrowhead" runs on Red Hat Enterprise Linux (RHEL), which we were already using for our derivatives trading system. As RHEL had delivered reliable and rapid performance in that role, we opted to use it again for the stock trading platform. A further consideration was RHEL's easy scalability, which was an important part of the design of "arrowhead".
PRIMEQUEST (mission-critical IA server)
PRIMERGY (industry standard x86 server)
ETERNUS (disk storage system for SAN environment)
IPCOM (network server)
SR-S (secure switch)
Si-R (IP access router)
Ultra-high-speed data management software
Interstage (business application platform software)
Symfoware (high-reliability database management software)
Systemwalker (enterprise system management software)
PRIMECLUSTER (clustering software)
ETERNUS SF (storage management software)
Figure 1: Overview of hardware and middleware in "arrowhead"
The platform's primary and secondary operating sites are connected to the exchange's "arrownet" optical ring network, which also links to the primary and secondary exchange offices and of course market participants. (See Figure 2 - TSE Systems Schematic Diagram.)
Andy Webb: How are system maintenance and operational roles divided between Fujitsu and TSE personnel?
Yuichiro Yamamoto: As I mentioned earlier, the system was built by the TSE in close collaboration with Fujitsu personnel. Now "arrowhead" is live, maintenance and any system enhancements (such as server cluster expansion) are the responsibility of Fujitsu, while TSE personnel handle day-to-day system operation.
Andy Webb: In view of the rate of order volume growth in modern electronic markets, scalability is presumably crucial. How flexible is the technology in this respect?
Yuichiro Yamamoto: "arrowhead" consists of four primary business functions:
- Order gateways through which market participants submit orders
- The order matching engine
- The market data distribution system
- Market surveillance
Each of these functions has its own dedicated server cluster (though the functions are linked at the software level by the PRIMESOFT middleware). These separate clusters mean that upgrades to an individual business function can be undertaken without disturbing the other functions. We maintain "arrowhead" with double the capacity required to handle predicted peak volumes. Nevertheless, if a hardware upgrade to any cluster is required, this takes at most one week - so we can respond very rapidly to any increases in activity.
Figure 2: TSE Systems Schematic Diagram
Andy Webb: How does the platform perform in terms of latency?
Yuichiro Yamamoto: It typically takes less than two milliseconds for an order to complete the round trip between a gateway server and the matching engine. (In other words, for an order message from a gateway server to hit the matching engine and for the gateway server to receive an order acknowledgement back from the matching engine.) In the case of participants using the TSE's co-location service, the typical round trip between the participant's co-location server and the gateway server is around 150 microseconds.
Andy Webb: What about redundancy of critical systems?
Yuichiro Yamamoto: "arrowhead" keeps transaction data resident in memory, which allows data access in the very low microsecond range. These memory-resident transaction data are processed in parallel on triply redundant servers, so if one server fails the switchover to another server will occur in seconds without any data loss.
Andy Webb: Apart from improving speed and resilience in the trading process, what other implications has the introduction of the new platform had on the TSE trading process?
Yuichiro Yamamoto: In addition to accelerating the distribution of market information, it has also allowed us to significantly expand the depth of market data provided to market participants. For example, the range of depth of book quotes above/below the currently traded price provided by the FLEX Standard data feed has been increased from five to eight, while the FLEX Full feed distributes all order information on all issues in real time.
We also took the opportunity with the launch of "arrowhead" to introduce a number of other innovations to the quoting/trading process. These include sequential trade quotes, implemented revisions to bid/offer price limits and special quote renewal price intervals, as well as order-matching conditions in itayose call auctions, which are all intended to promote smoother transaction flow and enhanced liquidity.
Taken as a whole, we believe that these changes - in conjunction with the implementation of "arrowhead" - also facilitate the efficient use of execution algorithms on the TSE.
Andy Webb: Would you consider selling "arrowhead" technology to other exchanges?
Yuichiro Yamamoto: Yes, we would be very happy to provide our technology to other exchanges. To date it has proven robust, with no outages or disruption since the launch date. Apart from the technology itself, the development process has been a valuable learning process for us that we could share with other exchanges. For example, as part of the project we had to develop an extremely detailed technical requirements document. While this was extremely challenging to accomplish, it ultimately resulted in robust technology that was implemented in accordance with the go live deadline.
Andy Webb: What is the exchange's attitude to high frequency trading and what controls do you have in place to counter the "message flooding" that some other exchanges have experienced?
Yuichiro Yamamoto: We welcome high frequency traders and believe that the capacity and speed of "arrowhead" combined with the introduction of our co-location services will facilitate their business. However, we do not offer sponsored access and at present have no plans to do so.
As regards control, we can monitor and manage the traffic on each participant's gateway server; this forms part of the surveillance business function of "arrowhead" referred to earlier. We also apply a fixed maximum limit to each gateway in terms of orders per second so as to prevent message flooding.
Andy Webb: Have the recent technology changes driven a significant increase in the number of foreign participants in the Tokyo market?
Yuichiro Yamamoto: It is difficult to say for certain, as it is the securities firms that have a direct connection to the exchange rather than end participants. However, the discernible change in trade and message volumes since the change of "arrowhead" might suggest that an increasing number of overseas participants are now trading on the exchange.
Andy Webb: How so?
Yuichiro Yamamoto: The number of completed trades has remained broadly speaking stable, but there has been an increase (though not dramatic) in the number of order messages being transmitted. This would be consistent with a growing number of order modifications or order slices that might suggest the presence of automated and algorithmic trading.
Andy Webb: What has been the market response to the TSE's co-location service?
Yuichiro Yamamoto: The co-location service opened to market participants in May 2009 for options and futures, with connectivity for equities, ETFs and REITs being made available as from January 4th of this year. Demand has been strong, so while we still have sufficient capacity to accommodate further participants, we are currently expanding the available co-location space to allow for future demand.
Andy Webb: Does "arrowhead" cover all the instruments traded on the TSE?
Yuichiro Yamamoto: No it only trades cash stocks, convertible bonds and exchange traded funds. For stock options we use a version of Euronext LIFFE's options trading platform that went live last October. For futures we currently use a system that was developed in house, but our intention is to make a strategy decision regarding the development of a next generation futures trading system this year. Once this is complete, it will complete the replacement of all the exchange's core trading infrastructure.