Over the last year, the trading industry has witnessed an increase in the demand for Precision Time Protocol (PTP). This has been driven partly by compliance, but also by a desire for firms to obtain a common time reference from which they can accurately deduce the precise sequence of - and hence causality within - events occurring across their distributed systems.
Corvil is a vendor with many years of experience in the timestamping and analysis of network packets, primarily for performance monitoring, and has a keen interest and expertise in the use of clock synchronisation technologies. As our customers have rolled out PTP, we have been able to help them validate accuracy and to troubleshoot the various causes of jitter and other deployment issues they have faced. In this article, we are going to explore a number of the issues with time synchronisation that we have witnessed first-hand.
Figure 01: PTP distributing a GPS time signal to a variety of end-systems
Before we start looking at some specific examples, let's first review the typical architectures in which PTP is deployed and the mechanisms that PTP uses to distribute time.
Firstly, a deployment requires an accurate clock source and GPS is the approach chosen by most users. A GPS antenna installed on the roof of the building receives signals from a constellation of satellites that are broadcasting atomic time as part of their primary function of providing global positioning. As a side note, GPS is operated by the United States government, whereas GLONASS and Galileo are the equivalent Russian and European systems.
Figure 02: Ideal deployment of PTPv1 with PTP enabled on all network devices
The GPS hardware on site typically supplies a grandmaster clock with atomic time which it uses to discipline its own internal clock. As part of this process, corrections should also be made in software to compensate for the propagation delay, as due to the length of the cable, between the antenna and the grandmaster - five nanoseconds per metre. Side note: Atomic time differs somewhat to UTC due to the fact that the speed at which the earth is revolving is changing; in fact it is slowing slightly, so periodically a leap second is added to UTC to correct for this effect. Currently UTC is 37 seconds ahead of TAI, the last correction having occurred on 31 December 2016.
The grandmaster's primary role is then to distribute accurate time to relevant downstream devices such as application servers, intrusion detection systems (IDSs), monitoring systems and network packet brokers, to name a few.
The mechanisms for distributing time from a grandmaster are typically Network Time Protocol (NTP), PTP or Pulse per Second (PPS). While NTP can synchronize systems reliably to 100μs or better, the way it exchanges timestamps is susceptible to jitter in the UDP/IP stack; this is a limiting factor in the standard implementations of NTP. In contrast, PTP uses hardware timestamping in end-systems that avoids this jitter, which allows it to achieve sub-microsecond accuracy. It is for this reason that most - but not all - implementations within electronic trading rely on PTP. PPS arguably provides the most accurate signal but, being a 1Hz square wave delivered over a dedicated coaxial cable, does not scale as an appropriate delivery mechanism beyond a handful of devices in local proximity.