Clock synchronization NIC offload

Abstract

In one embodiment, a system includes a network interface controller including a device interface to connect to a processing device and receive a time synchronization marker message from an application running on the processing device, a network interface to send packets over a network, and packet processing circuitry to process the time synchronization marker message for sending via the network interface over the network to a slave clock device, generate a time synchronization follow-up message including a timestamp indicative of when the synchronization marker message egressed the network interface, and process the time synchronization follow-up message for sending via the network interface over the network to the slave clock device.

Description

The project leading to this application has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No. 957403.

FIELD OF THE INVENTION

The present invention relates to computer networks, and in particular, but not exclusively to, clock synchronization in computer networks.

BACKGROUND

The Precision Time Protocol (PTP) is a protocol used to synchronize clocks throughout a computer network. A master clock device sends a clock synchronization marker message to a slave clock device across the network and measures when the marker message egresses the master clock device on to a network link. The master clock device then sends a clock synchronization follow-up message, which includes a timestamp indicating the time when the clock synchronization marker message egressed the master clock device. The reason for using the clock synchronization follow-up message to carry the time when the clock synchronization marker message leaves the master clock device is that is very difficult to measure when a message is leaving a device and to add a timestamp of that time in the same message. The slave clock then synchronizes its local clock based on the received messages and optionally other message interactions. The above process is known as 2-step PTP.

In one example, a user-space application running on a host device generates the clock synchronization marker message and provides the message to a kernel space driver which provides the message to a network interface controller (NIC). The NIC sends the message to the slave clock device and measures the time when the message egresses the NIC to the network, sometime known as “hitting the wire”. The NIC returns the measured time to the kernel space driver. The user-space application polls the kernel space driver for the measured time. The user-space application generates the clock synchronization follow-up message and inserts the measured time in the clock synchronization follow-up message when received from the kernel space driver. The user-space application then provides the clock synchronization follow-up message to the kernel space, which provides the clock synchronization follow-up message to NIC, which sends the clock synchronization follow-up message to the slave clock device.

SUMMARY

There is provided in accordance with an embodiment of the present disclosure, a system including a network interface controller including a device interface to connect to a processing device and receive a time synchronization marker message from an application running on the processing device, a network interface to send packets over a network, and packet processing circuitry to process the time synchronization marker message for sending via the network interface over the network to a slave clock device, generate a time synchronization follow-up message including a timestamp indicative of when the synchronization marker message egressed the network interface, and process the time synchronization follow-up message for sending via the network interface over the network to the slave clock device.

Further in accordance with an embodiment of the present disclosure the packet processing circuitry is configured to generate the time synchronization follow-up message autonomously of the application running on the processing device.

Still further in accordance with an embodiment of the present disclosure, the system includes the processing device, wherein the application is configured to generate the time synchronization marker message but not the time synchronization follow-up message.

Additionally in accordance with an embodiment of the present disclosure the packet processing circuitry is configured to duplicate the time synchronization marker message in a transmission pipeline of the packet processing circuitry as the time synchronization follow-up message for sending to the slave clock device.

Moreover in accordance with an embodiment of the present disclosure the packet processing circuitry is configured to perform a packet loopback of the time synchronization marker message in a transmission pipeline of the packet processing circuitry yielding a loop-backed packet directed towards a receive pipeline of the packet processing circuitry with the timestamp added to metadata of the loop-backed packet, and perform a hairpin operation to move the loop-backed packet from the receive pipeline to the transmission pipeline for sending to the slave clock device as the time synchronization follow-up message.

Further in accordance with an embodiment of the present disclosure the packet processing circuitry is configured to change an operation code field of the loop-backed packet from marker message to follow-up message.

Still further in accordance with an embodiment of the present disclosure the packet processing circuitry is configured to insert the timestamp into a header of the loop-backed packet responsively to the timestamp in the metadata of the loop-backed packet.

Additionally in accordance with an embodiment of the present disclosure the packet processing circuitry is configured to insert the timestamp into a preciseOriginTimestamp field of the loop-backed packet.

Moreover, in accordance with an embodiment of the present disclosure, the system includes a kernel space driver disposed between the application and the network interface controller.

Further in accordance with an embodiment of the present disclosure, the system includes the slave clock device including clock synchronization circuitry to receive the time synchronization marker message, and receive the time synchronization follow-up message while being unaware that the time synchronization follow-up message was generated by the network interface controller and not the application running on the processing device.

There is also provided in accordance with another embodiment of the present disclosure, a time synchronization method, including connecting to a processing device, receiving a time synchronization marker message from an application running on the processing device, sending packets over a network, processing by a network interface controller the time synchronization marker message for sending via a network interface over the network to a slave clock device, generating by the network interface controller a time synchronization follow-up message including a timestamp indicative of when the synchronization marker message egressed the network interface, and processing by the network interface controller the time synchronization follow-up message for sending via the network interface over the network to the slave clock device.

Still further in accordance with an embodiment of the present disclosure the generating includes generating the time synchronization follow-up message autonomously of the application running on the processing device.

Additionally in accordance with an embodiment of the present disclosure, the method includes the application generating the time synchronization marker message but not the time synchronization follow-up message.

Moreover, in accordance with an embodiment of the present disclosure, the method includes duplicating by the network interface controller the time synchronization marker message in a transmission pipeline as the time synchronization follow-up message for sending to the slave clock device.

Further in accordance with an embodiment of the present disclosure, the method includes performing by the network interface controller a packet loopback of the time synchronization marker message in a transmission pipeline yielding a loop-backed packet directed towards a receive pipeline with the timestamp added to metadata of the loop-backed packet, and performing a hairpin operation to move the loop-backed packet from the receive pipeline to the transmission pipeline for sending to the slave clock device as the time synchronization follow-up message.

Still further in accordance with an embodiment of the present disclosure, the method includes changing an operation code field of the loop-backed packet from marker message to follow-up message.

Additionally in accordance with an embodiment of the present disclosure, the method includes inserting the timestamp into a header of the loop-backed packet responsively to the timestamp in the metadata of the loop-backed packet.

Moreover, in accordance with an embodiment of the present disclosure the inserting includes inserting the timestamp into a preciseOriginTimestamp field of the loop-backed packet.

Further in accordance with an embodiment of the present disclosure, the method includes receiving by the slave clock device the time synchronization marker message, and receiving by the slave clock device the time synchronization follow-up message while being unaware that the time synchronization follow-up message was generated by the network interface controller and not the application running on the processing device.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood from the following detailed description, taken in conjunction with the drawings in which:

FIG. 1 is a block diagram view of clock synchronization system constructed and operative in accordance with an embodiment of the present invention;

FIG. 2 is a flowchart including steps in a method of operation of the system of FIG. 1;

FIG. 3 is a detailed view of packet processing circuitry in the system of FIG. 1 generating a time synchronization follow-up message; and

FIG. 4 is a detailed view of packet processing circuitry in the system of FIG. 1 generating a time synchronization follow-up message according to an alternative method.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

As previously mentioned, a user-space application running on a host device generates a clock synchronization marker message and provides the message to a kernel space driver which provides the message to a network interface controller (NIC). The NIC sends the message to the slave clock device and measures the time when the message egresses the NIC over the network, sometime known as “hitting the wire”. The NIC returns the measured time to the kernel space driver. The user-space application polls the kernel space driver for the measured time. The user-space application generates the clock synchronization follow-up message and inserts the measured time in the clock synchronization follow-up message when received from the kernel space driver. The user-space application then provides the clock synchronization follow-up message to the kernel space, which provides the clock synchronization follow-up message to NIC, which sends the clock synchronization follow-up message to the slave clock device. Generating the clock synchronization messages is a very processing intensive and is compounded by the number of slave clock devices in the network.

Embodiments of the present invention provide a system to generate clock synchronization follow-up messages in a NIC thereby offloading this task from a user-space application running on a host to the NIC. The clock synchronization marker messages are still generated by the user-space application for sending by the NIC to the slave clock devices.

In some embodiments, the clock synchronization marker message is duplicated in the transmission pipeline of the NIC as a clock synchronization follow-up message. The time that the clock synchronization marker message egresses the NIC is added to a header of the clock synchronization follow-up message and the operational code of the clock synchronization follow-up message is generally updated to “follow-up message”.

In other embodiments, a packet loopback of the clock synchronization marker message is performed towards the end of the transmission pipeline whereby the original clock synchronization marker message continues to egress the NIC whereas a copy of the clock synchronization marker message (i.e., a loop-backed packet) is directed towards a receive pipeline of the NIC with a timestamp (indicative of the time that the original clock synchronization marker message egresses the NIC) added to metadata of the loop-backed packet. The original clock synchronization marker message is included in the payload of the loop-backed packet. A hairpin operation is performed on the loop-backed packet moving the loop-backed packet into the transmission pipeline. The timestamp is taken from the metadata of the loop-backed packet and inserted into the header of the loop-backed packet and the operation code field of the loop-backed packet is changed from marker message to “follow-up message”. The above insertions and changes may be performed using steering actions in the transmission pipeline that identify a loop-backed packet with a status of “marker message”. Optionally, other fields of the loop-backed packet are modified, added or removed (e.g., optional TLV (type-length-value) fields).

In some embodiments, the user-space application may be aware that follow-up messages are being sent by the NIC (e.g., 2-step PTP where the generation of the follow-up messages is offloaded to the NIC), whereas in other embodiments, the user-space application may be unaware that the NIC is generating follow-up messages.

System Description

Reference is now made to FIG. 1, which is a block diagram view of clock synchronization system 10 constructed and operative in accordance with an embodiment of the present invention.

The system 10 may include a host device 16 (or any suitable processing device such as a central processing unit (CPU) or graphics processing unit (GPU)), a network interface controller 12 and a slave clock device 14. The host device 16 and the network interface controller 12 may also be known as a master clock device.

The host device 16 may include a processor 18 which is configured to run an application 20 and a kernel space driver 22, which is disposed logically between the application 20 and the network interface controller 12. The application 20 is configured to generate a time synchronization marker message 24, which is provided to the kernel space driver 22 and then to the network interface controller 12 for sending to the slave clock device 14 over a network 32.

The network interface controller 12 includes a device interface 26, packet processing circuitry 28, and a network interface 30. The packet processing circuitry 28 may include a physical layer (PHY) module and a MAC layer module, by way of example.

Reference is now made to FIG. 2, which is a flowchart 200 including steps in a method of operation of the system 10 of FIG. 1. Reference is also made to FIG. 1. The device interface 26 is configured to connect to the host device 16 and receive the time synchronization marker message 24 from the application 20 running on the host device 16 optionally via the kernel space driver 22 (block 202). The packet processing circuitry 28 is configured process the time synchronization marker message 24 for sending via the network interface 30 over the network 32 to the slave clock device 14 (block 204). The network interface 30 is configured to send packets (including the time synchronization marker message 24) over the network 32 to the slave clock device 14.

The packet processing circuitry 28 is configured to generate a time synchronization follow-up message 34 including a timestamp indicative of when the synchronization marker message 24 egressed the network interface 30 (block 206).

The packet processing circuitry is configured to generate the time synchronization follow-up message 34 autonomously of the application 20 running on the host device 16. In some embodiments, the application 20 running on the host device 16 is configured to generate the time synchronization marker message 24, but not the time synchronization follow-up message 34.

The packet processing circuitry 28 is configured to process the time synchronization follow-up message 34 for sending via the network interface 30 over the network 32 to the slave clock device 14 (block 208).

In practice, some or all of the functions of the packet processing circuitry 28 may be combined in a single physical component or, alternatively, implemented using multiple physical components. These physical components may comprise hard-wired or programmable devices, or a combination of the two. In some embodiments, at least some of the functions of the packet processing circuitry 28 may be carried out by a programmable processor under the control of suitable software. This software may be downloaded to a device in electronic form, over a network, for example. Alternatively, or additionally, the software may be stored in tangible, non-transitory computer-readable storage media, such as optical, magnetic, or electronic memory.

The slave clock device 14 includes: a physical hardware clock 36 to maintain a clock time; and clock synchronization circuitry 38. The clock synchronization circuitry 38 is configured to receive the time synchronization marker message 24 and the time synchronization follow-up message 34 while being unaware that the time synchronization follow-up message 34 was generated by the network interface controller 12 and not by the application 20 running on the host device 16. The clock synchronization circuitry 38 is configured to adjust the clock time of the physical hardware clock 36 responsively to receiving the messages 24, 34 and optionally other message interactions, e.g., to measure transit time between the master clock device and the slave clock device 14.

Reference is now made to FIG. 3, which is a detailed view of packet processing circuitry 28 in the system 10 of FIG. 1 generating the time synchronization follow-up message 34. Reference is also made to FIG. 2.

In some embodiments, the packet processing circuitry 28 is configured to duplicate (arrow 48) the time synchronization marker message 24 in a transmission pipeline 40 of the packet processing circuitry 28 as the time synchronization follow-up message 34 for sending to the slave clock device 14 (block 210). The packet processing circuitry 28 is configured to change an operation code field 42 of the time synchronization follow-up message 34 from marker message to follow-up message (block 212). The packet processing circuitry 28 is configured to insert a timestamp 44 (indicative of the time that the time synchronization marker message 24 egressed the network interface 30) into a header 46 of the time synchronization follow-up message 34 (block 214). In some embodiments, the packet processing circuitry 28 is configured to insert the timestamp 44 into a preciseOriginTimestamp field of the time synchronization follow-up message 34.

Reference is now made to FIG. 4, which is a detailed view of packet processing circuitry 28 in the system 10 of FIG. 1 generating the time synchronization follow-up message 34 according to an alternative method. Reference is also made to FIG. 2.

The packet processing circuitry 28 is configured to perform a packet loopback (block 54) of the time synchronization marker message 24 in the transmission pipeline 40 yielding a loop-backed packet 50 directed towards a receive pipeline 52 of the packet processing circuitry 28 with the timestamp 44 added to metadata of the loop-backed packet 50 (block 216). The time synchronization marker message 24 is typically included in a payload of the loop-backed packet 50.

The packet processing circuitry 28 is configured to perform a hairpin operation (block 56) to move the loop-backed packet 50 from the receive pipeline 52 to the transmission pipeline 40 for sending to the slave clock device 14 as the time synchronization follow-up message 34 (block 218).

In some embodiments, the packet processing circuitry 28 is configured to change an operation code field 42 of the loop-backed packet 50 (i.e., time synchronization follow-up message 34) from marker message to follow-up message (typically in the transmission pipeline 40) (block 212). In some embodiments, the packet processing circuitry 28 is configured to insert the timestamp 44 into the header 46 of the loop-backed packet 50 (i.e., the time synchronization follow-up message 34), typically in the transmission pipeline 40, responsively to the timestamp 44 in the metadata of the loop-backed packet 50 (block 214). In some embodiments, the packet processing circuitry 28 is configured to insert the timestamp into a preciseOriginTimestamp field of the loop-backed packet 50.

Various features of the invention which are, for clarity, described in the contexts of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment may also be provided separately or in any suitable sub-combination.

The embodiments described above are cited by way of example, and the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Claims

1. A system comprising a network interface controller (NIC) including: a device interface to connect to a processing device and receive a time synchronization marker message from an application running on the processing device;a network interface to send packets over a network; andpacket processing circuitry to: process the time synchronization marker message for sending via the network interface over the network to a slave clock device;generate a time synchronization follow-up message including a timestamp indicative of when the synchronization marker message egressed the network interface, thereby offloading generation of the time synchronization follow-up message from the application running on the processing device to the NIC; andprocess the time synchronization follow-up message for sending via the network interface over the network to the slave clock device.

2. The system according to claim 1, wherein the packet processing circuitry is configured to generate the time synchronization follow-up message autonomously of the application running on the processing device.

3. The system according to claim 1, further comprising the processing device, wherein the application is configured to generate the time synchronization marker message but not the time synchronization follow-up message.

4. The system according to claim 1, wherein the packet processing circuitry is configured to duplicate the time synchronization marker message in a transmission pipeline of the packet processing circuitry as the time synchronization follow-up message for sending to the slave clock device.

5. The system according to claim 1, wherein the packet processing circuitry is configured to: perform a packet loopback of the time synchronization marker message in a transmission pipeline of the packet processing circuitry yielding a loop-backed packet directed towards a receive pipeline of the packet processing circuitry with the timestamp added to metadata of the loop-backed packet; andperform a hairpin operation to move the loop-backed packet from the receive pipeline to the transmission pipeline for sending to the slave clock device as the time synchronization follow-up message.

6. The system according to claim 5, wherein the packet processing circuitry is configured to change an operation code field of the loop-backed packet from marker message to follow-up message.

7. The system according to claim 5, wherein the packet processing circuitry is configured to insert the timestamp into a header of the loop-backed packet responsively to the timestamp in the metadata of the loop-backed packet.

8. The system according to claim 7, wherein the packet processing circuitry is configured to insert the timestamp into a preciseOriginTimestamp field of the loop-backed packet.

9. The system according to claim 1, further comprising a kernel space driver disposed between the application and the network interface controller.

10. The system according to claim 1, further comprising the slave clock device including clock synchronization circuitry to: receive the time synchronization marker message; andreceive the time synchronization follow-up message while being unaware that the time synchronization follow-up message was generated by the network interface controller and not the application running on the processing device.

11. A time synchronization method, comprising: connecting to a processing device;receiving a time synchronization marker message from an application running on the processing device;sending packets over a network;processing by a network interface controller (NIC) the time synchronization marker message for sending via a network interface over the network to a slave clock device;generating by the NIC a time synchronization follow-up message including a timestamp indicative of when the synchronization marker message egressed the network interface, thereby offloading generation of the time synchronization follow-up message from the application running on the processing device to the NIC; andprocessing by the NIC the time synchronization follow-up message for sending via the network interface over the network to the slave clock device.

12. The method according to claim 11, wherein the generating includes generating the time synchronization follow-up message autonomously of the application running on the processing device.

13. The method according to claim 11, further comprising the application generating the time synchronization marker message but not the time synchronization follow-up message.

14. The method according to claim 11, further comprising duplicating by the network interface controller the time synchronization marker message in a transmission pipeline as the time synchronization follow-up message for sending to the slave clock device.

15. The method according to claim 11, further comprising: performing by the network interface controller a packet loopback of the time synchronization marker message in a transmission pipeline yielding a loop-backed packet directed towards a receive pipeline with the timestamp added to metadata of the loop-backed packet; andperforming a hairpin operation to move the loop-backed packet from the receive pipeline to the transmission pipeline for sending to the slave clock device as the time synchronization follow-up message.

16. The method according to claim 15, further comprising changing an operation code field of the loop-backed packet from marker message to follow-up message.

17. The method according to claim 15, further comprising inserting the timestamp into a header of the loop-backed packet responsively to the timestamp in the metadata of the loop-backed packet.

18. The method according to claim 17, wherein the inserting includes inserting the timestamp into a preciseOriginTimestamp field of the loop-backed packet.

19. The method according to claim 11, further comprising: receiving by the slave clock device the time synchronization marker message; andreceiving by the slave clock device the time synchronization follow-up message while being unaware that the time synchronization follow-up message was generated by the network interface controller and not the application running on the processing device.