Patent Grant
12294469
The present invention relates to computer systems, and in particular, but not exclusively to, clock synchronization.
Clock and frequency synchronization among network devices is used in many network applications. One application of using a synchronized clock value is for measuring latency between two devices. If the clocks are not synchronized the resulting latency measurement will be inaccurate.
Synchronous Ethernet (SyncE) is an International Telecommunication Union Telecommunication (ITU-T) Standardization Sector standard for computer networking that facilitates the transference of clock signals over the Ethernet physical layer. In particular, SyncE enables clock synchronization inside a network with respect to a master clock.
The actual clock value (e.g., in Coordinated Universal Time (UTC) format) is handled by higher layer standards and protocols, such as Precision Time Protocol (PTP).
There is provided in accordance with an embodiment of the present disclosure, a synchronized communication system, including a plurality of network devices, and clock connections to connect the network devices in a closed loop configuration, wherein the network devices are configured to distribute among the network devices a reference clock time from any selected one of the network devices.
Further in accordance with an embodiment of the present disclosure the reference clock time is in Coordinated Universal Time (UTC) format.
Still further in accordance with an embodiment of the present disclosure the selected network device includes a network interface to receive packets over a network from a remote clock, which outputs the reference clock time.
Additionally in accordance with an embodiment of the present disclosure the selected network device is configured to recover the reference clock time from the packets Precision Time Protocol (PTP).
Moreover, in accordance with an embodiment of the present disclosure, the system includes a controller to selectively block and unblock distribution of the reference clock time in the closed loop responsively to one of the network devices being designated as a local reference clock.
Further in accordance with an embodiment of the present disclosure the network devices include at least one of the following a data processing unit (DPU), graphics processing unit (GPU), central processing unit (CPU).
switch, network interface controller.
Still further in accordance with an embodiment of the present disclosure each of the network devices includes a network interface to transmit and receive clock synchronization messages over at least one network link, and
Additionally in accordance with an embodiment of the present disclosure a first one of the network devices includes clock synchronization circuitry to recover a remote clock time, a physical hardware clock, a clock input port connected to another clock output port of a second one of the network devices via a first one of the clock connections, and configured to receive a first clock signal indicative of the reference clock time from the second network device, and a clock output port connected to another clock input port of a third one of the network devices via a second one of the clock connections.
Moreover, in accordance with an embodiment of the present disclosure the second network device and the third network device are a same one of the network devices.
Further in accordance with an embodiment of the present disclosure the clock synchronization circuitry is configured to discipline the physical hardware clock to the reference clock time responsively to the recovered remote clock time, or the received clock signal, and output a time of the disciplined physical hardware clock in a second clock signal via the clock output port to the third network device.
Still further in accordance with an embodiment of the present disclosure the clock synchronization circuitry is configured to discipline the physical hardware clock to the reference clock time responsively to the recovered remote clock time, and output a time of the disciplined physical hardware clock in a second clock signal via the clock output port to the third network device.
Additionally in accordance with an embodiment of the present disclosure the clock synchronization circuitry is configured to ignore the first clock signal received by the clock input port.
Moreover, in accordance with an embodiment of the present disclosure, the system includes a controller to selectively block distribution of the reference clock time in the closed loop by instructing the clock synchronization circuitry to ignore the first clock signal received by the clock input port responsively to the first network device being designated as a local reference clock.
Further in accordance with an embodiment of the present disclosure the clock synchronization circuitry is configured to discipline the physical hardware clock to the reference clock time responsively to the received clock signal, and output a time of the disciplined physical hardware clock in a second clock signal via the clock output port to the third network device.
Still further in accordance with an embodiment of the present disclosure the network devices are configured to distribute the reference clock time via respective ones of the clock connections using a pulsed signal.
Additionally in accordance with an embodiment of the present disclosure a first one of the network devices includes a controller 14 to instruct clock synchronization circuitry of the first network device located immediately prior to a second one of the network devices, designated as a local reference clock in the closed loop, to not send a clock signal to the second network device.
There is also provided in accordance with another embodiment of the present disclosure, a synchronized communication method, including connecting network devices in a closed loop configuration, and distributing among the network devices a reference clock time from any selected one of the network devices.
Moreover, in accordance with an embodiment of the present disclosure the reference clock time is in Coordinated Universal Time (UTC) format.
Further in accordance with an embodiment of the present disclosure, the method includes the selected network device receiving packets over a network from a remote clock, which outputs the reference clock time.
Still further in accordance with an embodiment of the present disclosure, the method includes the selected network device recovering the reference clock time from the packets Precision Time Protocol (PTP).
Additionally in accordance with an embodiment of the present disclosure, the method includes selectively blocking and unblocking distribution of the reference clock time in the closed loop responsively to one of the network devices being designated as a local reference clock.
Moreover, in accordance with an embodiment of the present disclosure, the method includes transmitting and receiving clock synchronization messages over at least one network link, and processing the clock synchronization messages so as to recover a remote clock time.
Further in accordance with an embodiment of the present disclosure, the method includes a first one of the network devices recovering a remote clock time, connecting a clock input port of the first network device to another clock output port of a second one of the network devices via a first one of the clock connections, the first network device receiving a first clock signal indicative of the reference clock time from the second network device, and connecting a clock output port of the first network device to another clock input port of a third one of the network devices via a second one of the clock connections.
Still further in accordance with an embodiment of the present disclosure the second network device and the third network device are a same one of the network devices.
Additionally in accordance with an embodiment of the present disclosure, the method includes the first network device disciplining a physical hardware clock to the reference clock time responsively to the recovered remote clock time, or the received clock signal, and outputting a time of the disciplined physical hardware clock in a second clock signal via the clock output port to the third network device.
Moreover, in accordance with an embodiment of the present disclosure, the method includes the first network device disciplining a physical hardware clock to the reference clock time responsively to the recovered remote clock time, and output a time of the disciplined physical hardware clock in a second clock signal via the clock output port to the third network device.
Further in accordance with an embodiment of the present disclosure, the method includes the first network device ignoring the first clock signal received by the clock input port.
Still further in accordance with an embodiment of the present disclosure, the method includes selectively block distribution of the reference clock time in the closed loop by instructing clock synchronization circuitry to ignore the first clock signal received by the clock input port responsively to first network device being designated as a local reference clock.
Additionally in accordance with an embodiment of the present disclosure, the method includes the first network device disciplining a physical hardware clock to the reference clock time responsively to the received clock signal, and outputting a time of the disciplined physical hardware clock in a second clock signal via the clock output port to the third network device.
Moreover, in accordance with an embodiment of the present disclosure, the method includes distributing the reference clock time via respective ones of the clock connections using a pulsed signal.
Further in accordance with an embodiment of the present disclosure, the method includes instructing clock synchronization circuitry of a first network device located immediately prior to a second network device, designated as a local reference clock in the closed loop, to not send a clock signal to the second network device.
The present invention will be understood from the following detailed description, taken in conjunction with the drawings in which:
Clock synchronization between network devices remains a challenge in the networking industry due to continued demand for tighter clock synchronization. One solution is to use PTP clock chaining by chaining multiple PTP capable devices together so that a reference clock time is distributed from one network device at the root of the chain to other network devices in the chain. The root is defined by wiring topology.
PTP clock chaining may have some limitations including imposing a local clock hierarchy that is dictated by the physical wiring, introducing a possible “single point of failure” since the root controls the clock value of the entire chain. For example, if the network device at the root malfunctions, it then becomes impossible to distribute the clock to the remaining network devices. Therefore, if the local reference clock moves to another of the network devices, the reference clock time cannot be distributed based on the physical wiring.
One solution to the above problems is to transfer information regarding reference clock time between the root and the new local reference clock via some centralized entity, such as a software daemon running on a central processing unit (CPU). However, this solution adds complexity to the software-hardware/firmware interfaces, and to the software itself, and may add inaccuracies to the timing solution due to latencies and jitter of the control messages exchanged between the devices and the managing software. Additionally, this solution may add CPU load due to exchanging messages and performing calculations. It should be noted that CPU utilization is extremely important in many use cases, e.g., 5G use cases, where accurate clock synchronization is commonly required.
Embodiments of the present invention, solve at least some of the above problems by connecting network devices using clock connections to connect the network devices in a closed loop configuration. For example, network device 1 is connected to network device 2, which is connected to network device 3, which is connected to network device 1, forming a closed loop. The closed loop may then be used to distribute a reference clock time among the network devices from any selected one of the network devices in the closed loop by passing the reference clock time from network device to network device in the closed loop. For example, if one of the network devices is designated as a local reference clock, the reference clock time is distributed from the network device designated as the local reference clock to the other network devices via the clock connections of the closed loop. If at a later time another one of the network devices is designated as the local reference clock (for example, due to the previous network device designated as a local reference clock malfunctioning), the reference clock time is distributed from the network device of the newly designated local reference clock to the other network devices via the clock connections of the closed loop. Therefore, if one of the network devices malfunctions, it is still possible to operate another one of the network devices to distribute the reference clock time.
In some embodiments, a clock output port of one network device is connected to the clock input port of another network device with a cable or other connection (e.g., a trace on a circuit board), and so on, until all the network devices are connected together in a closed loop. For example, the clock output port of node 1 is connected to the clock input port of node 2, and so on. The clock output port of node 3 is connected to the clock input port of node 1, thereby completing the loop. Upon detecting a clock signal at its clock input port, a network device in the closed loop uses the received clock signal to discipline its physical hardware clock. A time of the physical hardware clock may then be output as a clock signal via the clock output port of that network device to the next network device in the chain, and so on.
In some embodiments, the network device designated as the local reference clock does not use the clock signal received from another network device to discipline its physical hardware clock. Instead, the network device designated as the local reference clock disciplines its physical hardware clock from a recovered remote clock. In some embodiments, software or firmware running on a controller breaks the chain of the closed loop so that the network device designated as the local reference clock does not use a clock signal received via its clock input port. Therefore, software or firmware may instruct the network device designated as the local reference clock to ignore the received clock signal at its clock input port and by default use the recovered remote clock to discipline its physical hardware clock. In other embodiments, software or firmware running on a controller breaks the chain of the closed loop so that the network device designated as the local reference clock does not receive a clock signal via its clock input port. Therefore, in some embodiments, the software or firmware running on the controller may instruct the network device, which would otherwise pass its clock via its clock output port to the network device of the designated local reference clock, to not output a clock signal to the network device of the designated local reference clock.
Each of the network devices may include clock synchronization circuitry which performs at least some of the following: recovering a remote clock and disciplining a local physical hardware clock based on the recovered remote clock, or receiving the clock signal via the chain and discipling the local physical hardware clock based on the received clock signal, and passing a clock value of its physical hardware clock in a clock signal to the next network device in the chain.
Reference is now made to
A plurality of clock connections 30 are configured to connect the network devices 12 in a closed loop configuration. For example, network device 12-1 is connected to network device 12-2, which is connected to network device 12-3, which in turn is connected to network device 12-1 via the clock connections 30 as described in more detail below.
The processing circuitry 16 may include hardwired processing circuitry and/or one or more processors on which to execute software. The software may be downloaded to the network device 12 or disposed on the network device 12 at manufacture. The processing circuitry 16 may include packet processing circuitry which may include a physical layer (PHY) chip and MAC chip (not shown). The processing circuitry 16 may include switching circuitry, and/or graphics processing unit (GPU) or any suitable processor, described in more detail with reference to
In some embodiments, the network interface 18 is configured to receive packets over a network from a remote clock 32, which outputs a reference clock time. In some embodiments, any one of the network devices 12 is configured to recover the reference clock time from the packets according to the Precision Time Protocol (PTP). In some embodiments, the network interface 18 is configured to transmit and receive clock synchronization messages over at least one network link (e.g., from the remote clock 32), and the clock synchronization circuitry 20 is configured to process the clock synchronization messages so as to recover a remote clock time from the remote clock 32. The clock synchronization messages may be received via any suitable interface via any suitable communication method and/or protocol.
The physical hardware clock 22 may be implemented as any suitable hardware clock, for example, a PTP hardware clock. The physical hardware clock 22 may be implemented as a physical clock, which is stored on the device 12. In some embodiments, the physical hardware clock 22 may be implemented as a hybrid hardware and software clock in which hardware stores a free running clock (e.g., a hardware counter which is incremented), and software holds parameters which convert the hardware clock into the actual clock time. In some embodiments, when the physical hardware clock 22 is implemented as a hybrid hardware and software clock, the PTP or PHC time and frequency adjustments do not affect the physical free running clock, but only the conversion parameters.
The clock input port 26 of one of the network devices 12 (e.g., network device 12-1) is connected to the clock output port 28 of another one of the network devices 12 (e.g., network device 12-3) via one of the clock connections 30, and configured to receive a clock signal indicative of the reference clock time from the other network device 12 (e.g., network device 12-3). The clock output port 28 of one of the network devices 12 (e.g., network device 12-1) is connected to the clock input port 26 of another one of the network devices 12 (e.g., network device 12-2) via one of the clock connections 30. The clock output port 28 of the network device 12-2 is connected to the clock input port 26 of the network device 12-3 via one of the clock connections 30.
In general, the network devices 12 are configured to distribute among the network devices 12 the reference clock time from any selected one of the network devices, for example, the computer node 12-2 designated as the local reference clock. The reference clock time may be maintained in any suitable format, for example, in Coordinated Universal Time (UTC) format.
In the example of
The network devices 12 may be configured to distribute the reference clock time via respective clock connections 30 in the form of any signal e.g., using N pulses per second, such as one pulse per second (PPS) signal(s) or 10 mega Hertz (10 MHz) signal(s).
In the network device 12-2 designated as the local reference clock, the network device 12-2 disciplines its physical hardware clock 22 to provide a clock time based on a clock time recovered from the remote clock 32. In the network device(s) 12-1, 12-3 not designated as the local reference clock, the clock signal received at the clock input port 26 is used to discipline the respective physical hardware clock 22.
In some embodiments, software or firmware running on the controller 14 breaks the chain of the closed loop so that the network device 12-2 designated as the local reference clock does not use a clock signal received at its clock input port 26 or does not receive a clock signal at its clock input port 26, as described in more detail with reference to
When the network devices 12 boot up, each network device 12 looks for a clock signal being received at its own clock input port 26 and if a clock signal is not found, the respective network device 12 uses a local clock, for example, based on an output of the oscillator 24 in that network device 12. Therefore, the first network device 12 to boot up outputs a clock signal based on its physical hardware clock 22 from its clock output port 28 to the next network device 12 in the closed loop. The next network device 12 then detects the clock signal input via its clock input port 26 and uses the received clock signal to discipline its physical hardware clock 22, and so on. When one of the network devices 12 is designated as a local reference clock, that network device 12 does not use the clock signal received at its clock input port 26, but disciplines its physical hardware clock 22 based on the remote clock 32 and outputs a clock signal indicative of the clock value of its physical hardware clock 22 via its clock output port 28 to the next network device 12 in the loop, and so on. Another option is to assign one of the network devices 12 as a default local reference clock.
Reference is now made to
In the example of
Reference is now made to
In some embodiments, the controller 14 is configured to run a software daemon which knows the topology of the system 10 (i.e., how the network devices 12 are connected in the closed loop) and which network device 12 is the local reference clock (e.g., PTP master) so that the software daemon knows where to block and unblock the closed loop. If the network devices 12 are disposed in different hosts, then the hosts may need to communicate with respect to blocking and unblocking the closed loop.
The controller 14 is configured to identify or designate one of the network devices 12 as the local reference clock (block 302). The controller 14 is configured to selectively block and unblock distribution of the reference clock time in the closed loop responsively to one of the network devices 12 being designated as a local reference clock (block 304). In some embodiments, the controller 14 is configured to instruct the clock synchronization circuitry 20 of the network device 12 designated as the local reference clock to ignore the clock signal received at its clock input port 26 responsively to that network device 12 being designated as the local reference clock (block 306). In other embodiments, the controller 14 is configured to instruct the clock synchronization circuitry 20 of the network device 12 (designated as a slave clock and) located immediately prior to the network device 12 designated as the local reference clock in the closed loop to not send a clock signal via its clock output port 28 to the network device 12 designated as the local reference clock (block 308).
In practice, some or all of the functions of the controller 14 may be combined in a single physical component or, alternatively, implemented using multiple physical components. These physical components may comprise hard-wired or programmable devices, or a combination of the two. In some embodiments, at least some of the functions of the controller 14 may be carried out by a programmable processor under the control of suitable software. This software may be downloaded to a device in electronic form, over a network, for example. Alternatively, or additionally, the software may be stored in tangible, non-transitory computer-readable storage media, such as optical, magnetic, or electronic memory.
Reference is now made to
The flowchart 400 is first traversed assuming that the network device 12-3 is designated as a slave clock.
When the network device 12-3 first boots up, the clock synchronization circuitry 20 of the network device 12-3 is configured to generate a clock responsively to an output from the oscillator 24. After a short delay, assuming there is still no clock signal received by the clock input port 26 of the network device 12-3, the clock synchronization circuitry 20 of the network device 12-3 is configured to recover a remote clock, e.g., from the remote clock 32 (block 404) and discipline its physical hardware clock 22 responsively to the recovered remote clock. The clock synchronization circuitry 20 of the network device 12-3 is configured to receive a clock signal via the clock input port 26 of the network device 12-3 (block 408) from the previous network device 12-2 in the closed loop. The clock synchronization circuitry 20 of the network device 12-3 is configured to discipline its physical hardware clock 22 to the reference clock time responsively to the received clock signal (block 410). The clock synchronization circuitry 20 of the network device 12-3 is configured to output a clock signal indicative of a time of its disciplined physical hardware clock 22 via the clock output port 28 of the network device 12-3 to the next network device 12-1 in the closed loop (block 412).
The flowchart 400 is now traversed assuming that the network device 12-3 is now designated as a local reference clock.
When the network device 12-3 first boots up, the clock synchronization circuitry 20 of the network device 12-3 is configured to generate a clock responsively to an output from the oscillator 24. The clock synchronization circuitry 20 of the network device 12-3 is configured to recover a remote clock, e.g., from the remote clock 32 (block 404). If a clock signal is received by the clock synchronization circuitry 20 of the network device 12-3 via the clock input port 26 of network device 12-3, the clock synchronization circuitry 20 of the network device 12-3 is configured to ignore the clock signal received by the clock input port 26 (block 414). The clock synchronization circuitry 20 of network device 12-3 is configured to discipline its physical hardware clock 22 to the reference clock time responsively to the recovered remote clock (recovered in the step of block 404) (block 416). The clock synchronization circuitry 20 of the network device 12-3 is then configured to perform the step of block 412.
In practice, some or all of the functions of the clock synchronization circuitry 20 may be combined in a single physical component or, alternatively, implemented using multiple physical components. These physical components may comprise hard-wired or programmable devices, or a combination of the two. In some embodiments, at least some of the functions of the clock synchronization circuitry 20 may be carried out by a programmable processor under the control of suitable software. This software may be downloaded to a device in electronic form, over a network, for example. Alternatively, or additionally, the software may be stored in tangible, non-transitory computer-readable storage media, such as optical, magnetic, or electronic memory.
Reference is now made to
The clock synchronization system 500 is substantially the same as the system 10 except that in the clock synchronization system 500 there are only two computes node 12. The clock synchronization system 500 may be compared to combining network devices 12-1, 12-3 of system 10 into the same network device 12-1, which is in a closed loop with the network device 12-2.
In the clock synchronization system 500, the clock output port 28 of network device 12-1 is connected to the clock input port 26 of network device 12-2 via one of the clock connections 30, and the clock output port 28 of network device 12-2 is connected to the clock input port 26 of network device 12-1 via another one of the clock connections 30 thereby forming the closed loop.
Reference is now made to
Various features of the invention which are, for clarity, described in the contexts of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment may also be provided separately or in any suitable sub-combination.
The embodiments described above are cited by way of example, and the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.
The present application claims benefit of US Provisional Patent Application Ser. No. 63/340,955 of Manevich, et al., filed 12 May 2022, the disclosure of which is hereby incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5392421 | Lennartsson | Feb 1995 | A |
| 5402394 | Turski | Mar 1995 | A |
| 5416808 | Witsaman et al. | May 1995 | A |
| 5491792 | Grisham et al. | Feb 1996 | A |
| 5564285 | Jurewicz et al. | Oct 1996 | A |
| 5592486 | Lo et al. | Jan 1997 | A |
| 5896524 | Halstead, Jr. et al. | Apr 1999 | A |
| 6055246 | Jones | Apr 2000 | A |
| 6084856 | Simmons et al. | Jul 2000 | A |
| 6144714 | Bleiweiss et al. | Nov 2000 | A |
| 6199169 | Voth | Mar 2001 | B1 |
| 6289023 | Dowling et al. | Sep 2001 | B1 |
| 6449291 | Burns et al. | Sep 2002 | B1 |
| 6535926 | Esker | Mar 2003 | B1 |
| 6556636 | Takagi | Apr 2003 | B1 |
| 6556638 | Blackburn | Apr 2003 | B1 |
| 6718476 | Shima | Apr 2004 | B1 |
| 6918049 | Lamb et al. | Jul 2005 | B2 |
| 7111184 | Thomas, Jr. et al. | Sep 2006 | B2 |
| 7191354 | Purho | Mar 2007 | B2 |
| 7245627 | Goldenberg et al. | Jul 2007 | B2 |
| 7254646 | Aguilera et al. | Aug 2007 | B2 |
| 7334124 | Pham et al. | Feb 2008 | B2 |
| 7412475 | Govindarajalu | Aug 2008 | B1 |
| 7440474 | Goldman et al. | Oct 2008 | B1 |
| 7447975 | Riley | Nov 2008 | B2 |
| 7483448 | Bhandari et al. | Jan 2009 | B2 |
| 7496686 | Coyle | Feb 2009 | B2 |
| 7535933 | Zerbe et al. | May 2009 | B2 |
| 7623552 | Jordan et al. | Nov 2009 | B2 |
| 7636767 | Lev-Ran et al. | Dec 2009 | B2 |
| 7650158 | Indirabhai | Jan 2010 | B2 |
| 7656751 | Rischar et al. | Feb 2010 | B2 |
| 7750685 | Bunch et al. | Jul 2010 | B1 |
| 7904713 | Zajkowski et al. | Mar 2011 | B1 |
| 7941684 | Serebrin et al. | May 2011 | B2 |
| 8065052 | Fredriksson et al. | Nov 2011 | B2 |
| 8300749 | Hadzic et al. | Oct 2012 | B2 |
| 8341454 | Kondapalli | Dec 2012 | B1 |
| 8370675 | Kagan | Feb 2013 | B2 |
| 8407478 | Kagan et al. | Mar 2013 | B2 |
| 8607086 | Cullimore | Dec 2013 | B2 |
| 8699406 | Charles et al. | Apr 2014 | B1 |
| 8824903 | Christensen | Sep 2014 | B2 |
| 8879552 | Zheng | Nov 2014 | B2 |
| 8930647 | Smith | Jan 2015 | B1 |
| 9344265 | Karnes | May 2016 | B2 |
| 9397960 | Arad et al. | Jul 2016 | B2 |
| 9549234 | Mascitto | Jan 2017 | B1 |
| 9753854 | Bao | Sep 2017 | B1 |
| 9942025 | Bosch et al. | Apr 2018 | B2 |
| 9979998 | Pogue et al. | May 2018 | B1 |
| 10014937 | Di Mola et al. | Jul 2018 | B1 |
| 10027601 | Narkis et al. | Jul 2018 | B2 |
| 10054977 | Mikhaylov et al. | Aug 2018 | B2 |
| 10095543 | Griffin et al. | Oct 2018 | B1 |
| 10148258 | Carlson et al. | Dec 2018 | B2 |
| 10164759 | Volpe | Dec 2018 | B1 |
| 10320646 | Mirsky et al. | Jun 2019 | B2 |
| 10515045 | Mattina | Dec 2019 | B1 |
| 10623173 | Geng et al. | Apr 2020 | B1 |
| 10637776 | Iwasaki | Apr 2020 | B2 |
| 10727966 | Izenberg et al. | Jul 2020 | B1 |
| 10778361 | Almog et al. | Sep 2020 | B1 |
| 10841243 | Levi et al. | Nov 2020 | B2 |
| 10879910 | Franck et al. | Dec 2020 | B1 |
| 10887077 | Ivry | Jan 2021 | B1 |
| 11070224 | Faig et al. | Jul 2021 | B1 |
| 11070304 | Levi et al. | Jul 2021 | B1 |
| 11128500 | Mentovich et al. | Sep 2021 | B1 |
| 11157433 | Lederman et al. | Oct 2021 | B2 |
| 11240079 | Kushnir et al. | Feb 2022 | B1 |
| 11303363 | Mohr et al. | Apr 2022 | B1 |
| 11336383 | Mula et al. | May 2022 | B2 |
| 11368768 | Bakopoulos et al. | Jun 2022 | B2 |
| 11379334 | Srinivasan et al. | Jul 2022 | B1 |
| 11388263 | Levi et al. | Jul 2022 | B2 |
| 11476928 | Levi et al. | Oct 2022 | B2 |
| 20010006500 | Nakajima et al. | Jul 2001 | A1 |
| 20020027886 | Fischer et al. | Mar 2002 | A1 |
| 20020031199 | Rolston et al. | Mar 2002 | A1 |
| 20040096013 | Laturell et al. | May 2004 | A1 |
| 20040153907 | Gibart | Aug 2004 | A1 |
| 20050033947 | Morris et al. | Feb 2005 | A1 |
| 20050172181 | Huliehel | Aug 2005 | A1 |
| 20050268183 | Barmettler | Dec 2005 | A1 |
| 20060109376 | Chaffee et al. | May 2006 | A1 |
| 20070008044 | Shimamoto | Jan 2007 | A1 |
| 20070072451 | Tazawa et al. | Mar 2007 | A1 |
| 20070104098 | Kimura et al. | May 2007 | A1 |
| 20070124415 | Lev-Ran et al. | May 2007 | A1 |
| 20070139085 | Elliot et al. | Jun 2007 | A1 |
| 20070159924 | Vook et al. | Jul 2007 | A1 |
| 20070266119 | Ohly | Nov 2007 | A1 |
| 20080069150 | Badt et al. | Mar 2008 | A1 |
| 20080225841 | Conway et al. | Sep 2008 | A1 |
| 20080285597 | Downey et al. | Nov 2008 | A1 |
| 20090055676 | Wiebe | Feb 2009 | A1 |
| 20090257458 | Cui et al. | Oct 2009 | A1 |
| 20100280858 | Bugenhagen | Nov 2010 | A1 |
| 20110182191 | Jackson | Jul 2011 | A1 |
| 20110194425 | Li et al. | Aug 2011 | A1 |
| 20120063556 | Hoang | Mar 2012 | A1 |
| 20120076319 | Terwal | Mar 2012 | A1 |
| 20120301134 | Davari et al. | Nov 2012 | A1 |
| 20130039359 | Bedrosian | Feb 2013 | A1 |
| 20130045014 | Mottahedin et al. | Feb 2013 | A1 |
| 20130215889 | Zheng et al. | Aug 2013 | A1 |
| 20130235889 | Aweya et al. | Sep 2013 | A1 |
| 20130294144 | Wang et al. | Nov 2013 | A1 |
| 20130315265 | Webb, III et al. | Nov 2013 | A1 |
| 20130336435 | Akkihal et al. | Dec 2013 | A1 |
| 20140085141 | Geva et al. | Mar 2014 | A1 |
| 20140153680 | Garg et al. | Jun 2014 | A1 |
| 20140185216 | Zeng et al. | Jul 2014 | A1 |
| 20140185632 | Steiner et al. | Jul 2014 | A1 |
| 20140253387 | Gunn et al. | Sep 2014 | A1 |
| 20140281036 | Cutler et al. | Sep 2014 | A1 |
| 20140301221 | Nadeau et al. | Oct 2014 | A1 |
| 20140321285 | Chew et al. | Oct 2014 | A1 |
| 20150019839 | Cardinell et al. | Jan 2015 | A1 |
| 20150078405 | Roberts | Mar 2015 | A1 |
| 20150092793 | Aweya | Apr 2015 | A1 |
| 20150127978 | Cui et al. | May 2015 | A1 |
| 20150163050 | Han et al. | Jun 2015 | A1 |
| 20150318941 | Zheng et al. | Nov 2015 | A1 |
| 20160057518 | Neudorf | Feb 2016 | A1 |
| 20160072602 | Earl et al. | Mar 2016 | A1 |
| 20160078901 | Toma | Mar 2016 | A1 |
| 20160110211 | Karnes | Apr 2016 | A1 |
| 20160140066 | Worrell et al. | May 2016 | A1 |
| 20160277138 | Garg et al. | Sep 2016 | A1 |
| 20160285574 | White et al. | Sep 2016 | A1 |
| 20160315756 | Tenea et al. | Oct 2016 | A1 |
| 20170005903 | Mirsky | Jan 2017 | A1 |
| 20170017604 | Chen et al. | Jan 2017 | A1 |
| 20170126589 | Estabrooks et al. | May 2017 | A1 |
| 20170160933 | De Jong et al. | Jun 2017 | A1 |
| 20170214516 | Rivaud et al. | Jul 2017 | A1 |
| 20170302392 | Farra et al. | Oct 2017 | A1 |
| 20170331926 | Raveh et al. | Nov 2017 | A1 |
| 20170359137 | Butterworth et al. | Dec 2017 | A1 |
| 20180059167 | Sharf et al. | Mar 2018 | A1 |
| 20180152286 | Kemparaj et al. | May 2018 | A1 |
| 20180188698 | Dionne et al. | Jul 2018 | A1 |
| 20180191802 | Yang et al. | Jul 2018 | A1 |
| 20180227067 | Hu et al. | Aug 2018 | A1 |
| 20180309654 | Achkir et al. | Oct 2018 | A1 |
| 20190007189 | Hossain et al. | Jan 2019 | A1 |
| 20190014526 | Bader et al. | Jan 2019 | A1 |
| 20190089615 | Branscomb et al. | Mar 2019 | A1 |
| 20190149258 | Araki et al. | May 2019 | A1 |
| 20190158909 | Kulkarni et al. | May 2019 | A1 |
| 20190196563 | Lai | Jun 2019 | A1 |
| 20190273571 | Bordogna et al. | Sep 2019 | A1 |
| 20190319729 | Leong et al. | Oct 2019 | A1 |
| 20190349392 | Wetterwald et al. | Nov 2019 | A1 |
| 20190379714 | Levi et al. | Dec 2019 | A1 |
| 20200162234 | Almog et al. | May 2020 | A1 |
| 20200169379 | Gaist | May 2020 | A1 |
| 20200235905 | Su et al. | Jul 2020 | A1 |
| 20200304224 | Neugeboren | Sep 2020 | A1 |
| 20200331480 | Zhang et al. | Oct 2020 | A1 |
| 20200344333 | Hawari et al. | Oct 2020 | A1 |
| 20200396050 | Perras et al. | Dec 2020 | A1 |
| 20200401434 | Thampi et al. | Dec 2020 | A1 |
| 20200403652 | Goff | Dec 2020 | A1 |
| 20210006344 | Chen et al. | Jan 2021 | A1 |
| 20210141413 | Levi et al. | May 2021 | A1 |
| 20210218431 | Narayanan et al. | Jul 2021 | A1 |
| 20210243140 | Levi et al. | Aug 2021 | A1 |
| 20210288785 | Faig et al. | Sep 2021 | A1 |
| 20210297151 | Levi et al. | Sep 2021 | A1 |
| 20210297230 | Dror et al. | Sep 2021 | A1 |
| 20210318978 | Hsung | Oct 2021 | A1 |
| 20210328900 | Sattinger et al. | Oct 2021 | A1 |
| 20210392065 | Sela et al. | Dec 2021 | A1 |
| 20210409031 | Ranganathan | Dec 2021 | A1 |
| 20220006606 | Levi et al. | Jan 2022 | A1 |
| 20220021393 | Ravid et al. | Jan 2022 | A1 |
| 20220066978 | Mishra et al. | Mar 2022 | A1 |
| 20220086105 | Levi et al. | Mar 2022 | A1 |
| 20220173741 | Ravid et al. | Jun 2022 | A1 |
| 20220191275 | Levi et al. | Jun 2022 | A1 |
| 20220191578 | Ramkumar et al. | Jun 2022 | A1 |
| 20220121691 | Mentovich et al. | Jul 2022 | A1 |
| 20220224500 | Mula et al. | Jul 2022 | A1 |
| 20220239549 | Zhao et al. | Jul 2022 | A1 |
| 20220342086 | Yoshida | Oct 2022 | A1 |
| 20220390979 | Xu | Dec 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 106817183 | Jun 2017 | CN |
| 108829493 | Nov 2018 | CN |
| 1215559 | Sep 2007 | EP |
| 2770678 | Aug 2014 | EP |
| 2011091676 | May 2011 | JP |
| 498259 | Aug 2002 | TW |
| 2012007276 | Jan 2012 | WO |
| 2013124782 | Aug 2013 | WO |
| 2013143112 | Oct 2013 | WO |
| 2014029533 | Feb 2014 | WO |
| 2014138936 | Sep 2014 | WO |
| Entry |
|---|
| Levi et al., U.S. Appl. No. 17/246,730, filed May 3, 2021. |
| Levi et al., U.S. Appl. No. 17/315,396, filed May 10, 2021. |
| Levi et al., U.S. Appl. No. 17/359,667, filed Jun. 28, 2021. |
| Wasko et al., U.S. Appl. No. 17/520,674, filed Nov. 7, 2021. |
| Wasko et al., U.S. Appl. No. 17/582,058, filed Jan. 24, 2022. |
| Levi et al., U.S. Appl. No. 17/667,600, filed Feb. 9, 2022. |
| Shapira et al., U.S. Appl. No. 17/534,776, filed Nov. 24, 2021. |
| Shapira et al., U.S. Appl. No. 17/578,115, filed Jan. 18, 2022. |
| Kernen et al., U.S. Appl. No. 17/858,236, filed Jul. 6, 2022. |
| Zhang et al., “TI BAW technology enables ultra-low jitter clocks for highspeed networks”, White paper, Texas Instruments, pp. 1-11, Feb. 2019. |
| Skywork Solutions Inc., “PCI Express 3.1 JITTER Requirements”, AN562, pp. 1-16, year 2021. |
| Intel, “Can Altera GX/GT/GZ device high speed transceivers handle Spread Spectrum Clocking (SSC), as required by PCIe or SATA/SAS protocols?”, p. 1, Sep. 11, 2012. |
| Pismenny et al., U.S. Appl. No. 17/824,954, filed May 26, 2022. |
| U.S. Appl. No. 17/871,937 Office Action dated Aug. 1, 2023. |
| U.S. Appl. No. 17/578,115 Office Action dated Apr. 26, 2023. |
| U.S. Appl. No. 17/534,776 Office Action dated Jun. 29, 2023. |
| SiTime Corporation, “Sit5377—60 to 220 MHZ, ±100 ppb Elite RF™ Super-TCXO,” Product Description, pp. 1-3, last updated Mar. 18, 2023 as downloaded from https://web.archive.org/web/20230318094421/https://www.sitime.com/products/super-tcxos/sit5377. |
| PCI-SIG, “PCI Express®—Base Specification—Revision 3.0,” pp. 1-860, Nov. 10, 2010. |
| U.S. Appl. No. 17/867,779 Office Action dated Mar. 28, 2024. |
| U.S. Appl. No. 17/549,949 Office Action dated Mar. 30, 2023. |
| Corbett et al., “Spanner: Google's Globally Distributed Database,” ACM Transactions on Computer Systems, vol. 31, No. 3, article 8, pp. 1-22, Aug. 2013. |
| U.S. Appl. No. 17/191,736 Office Action dated Jun. 26, 2023. |
| IEEE Standard 1588™-2008: “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, IEEE Instrumentation and Measurement Society, Revision of IEEE Standard 1588-2002, USA, pp. 1-289, Jul. 24, 2008. |
| Weibel et al., “Implementation and Performance of Time Stamping Techniques”, 2004 Conference on IEEE 1588, pp. 1-29, Sep. 28, 2004. |
| Working Draft Project American National Standard T10/1799-D, “Information Technology—SCSI Block Commands—3(SBC-3)”, pp. 1-220, Revision 19, May 29, 2009. |
| “Infiniband Architecture: Specification vol. 1”, pp. 1-1727, Release 1.2.1, Infiniband Trade Association, Nov. 2007. |
| Mellanox Technologies, “Mellanox ConnectX IB: Dual-Port InfiniBand Adapter Cards with PCI Express 2.0”, pp. 1-2, USA, year 2008. |
| WIKIPEDIA—“Precision Time Protocol”, pp. 1-8, Aug. 24, 2019. |
| IEEE Std 1588-2002, “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, IEEE Instrumentation and Measurement Society, pp. 1-154, Nov. 8, 2002. |
| Weibel, H., “High Precision Clock Synchronization according to IEEE 1588 Implementation and Performance Issues”, Zurich University of Applied Sciences, pp. 1-9, Jan. 17, 2005. |
| Lu et al., “A Fast CRC Update Implementation”, Computer Engineering Laboratory, Electrical Engineering Department, pp. 113-120, Oct. 8, 2003. |
| Texas Instruments, “LMK05318 Ultra-Low Jitter Network Synchronizer Clock With Two Frequency Domains,” Product Folder, pp. 1-86, Dec. 2018. |
| Dlugy-Hegwer et al., “Designing and Testing IEEE 1588 Timing Networks”, Symmetricom, pp. 1-10, Jan. 2007. |
| Mellanox Technologies, “How to test 1PPS on Mellanox Adapters”, pp. 1-6, Oct. 22, 2019 downloaded from https://community.mellanox.com/s/article/How-To-Test-1PPS-on-Mellanox-Adapters. |
| ITU-T recommendation, “G.8273.2/Y.1368.2—Timing characteristics of telecom boundary clocks and telecom time slave clocks”, pp. 1-50, Jan. 2017. |
| Wasko et al., U.S. Appl. No. 17/549,949, filed Dec. 14, 2021. |
| Ipclock, “IEEE 1588 Primer,” ip-clock.com, pp. 1-3, May 1, 2017 (downloaded from https://web.archive.org/web/20170501192647/http://ip-clock.com/ieee-1588-primer/). |
| ITU-T Standard G.8261/Y.1361, “Timing and synchronization aspects in packet networks”, pp. 1-120, Aug. 2019. |
| Levy et al., U.S. Appl. No. 17/313,026, filed May 6, 2021. |
| “Precision Time Protocol,” PTP Clock Types, CISCO, pp. 1-52, Jul. 30, 2020, as downloaded from https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/5x/system-management-configuration/cisco-apic-system-management-configuration-guide-52x/m-precision-time-protocol.pdf. |
| ITU-T Standard G.8262/Y.1362, “Timing characteristics of synchronous equipment slave clock”, pp. 1-44, Nov. 2018. |
| ITU-T Standard G.8264/Y.1364, “Distribution of timing information through packet networks”, pp. 1-42, Aug. 2017. |
| Manevich et al., U.S. Appl. No. 17/579,630, filed Jan. 20, 2022. |
| Levi et al., U.S. Appl. No. 17/871,937, filed Jul. 24, 2022. |
| Manevich et al., U.S. Appl. No. 17/867,779, filed Jul. 19, 2022. |
| Levi et al., U.S. Appl. No. 17/868,841, filed Jul. 20, 2022. |
| U.S. Appl. No. 17/579,630 Office Action dated Oct. 24, 2022. |
| U.S. Appl. No. 17/579,630 Office Action dated Jan. 12, 2023. |
| U.S. Appl. No. 17/670,540 Office Action dated Jan. 18, 2023. |
| U.S. Appl. No. 17/191,736 Office Action dated Nov. 10, 2022. |
| Levi et al., U.S. Appl. No. 17/582,058, filed Jan. 24, 2022. |
| U.S. Appl. No. 17/191,736 Advisory Action dated Feb. 16, 2023. |
| “IEEE Standard for Local and Metropolitan Area Networks—Timing and Synchronization for Time-Sensitive Applications,” IEEE Std 802.1AS-2020, IEEE Computer Society, pp. 1-421, year 2020. |
| U.S. Appl. No. 17/867,779 Office Action dated Jan. 30, 2025. |
| Number | Date | Country | |
|---|---|---|---|
| 20230370305 A1 | Nov 2023 | US |
| Number | Date | Country | |
|---|---|---|---|
| 63340955 | May 2022 | US |
