The present invention relates to computer systems, and in particular, but not exclusively to, clock syntonization.
Clock synchronization among network devices is used in many network applications. One application of using a synchronized clock value is for measuring one-way latency from one device to another device. If the clocks are not synchronized the resulting one-way latency measurement will be inaccurate.
Synchronization is typically achieved by syntonization, in which the clock frequency of two devices is aligned, and aligning the phase between the two devices.
For Ethernet, there are two complementary methods to achieve synchronization. One is Synchronous Ethernet (SyncE), which is a physical-layer protocol which achieves syntonization based on the receive/transmit symbol rate. SyncE is an International Telecommunication Union Telecommunication (ITU-T) Standardization Sector standard for computer networking that facilitates the transference of clock signals over the Ethernet physical layer. In particular, SyncE enables clock syntonization inside a network with respect to a master clock.
The other is Precision Time Protocol (PTP), which is a packet-based protocol that may be used with SyncE to align offset (e.g., in Coordinated Universal Time (UTC) format) and phase between two clocks. It should be noted that PTP may be used alone over Ethernet (without SyncE), but this is typically used for lower accuracy use cases. PTP is used to synchronize clocks throughout a computer network, and is considered to be the de facto standard for this purpose.
Time, clock, and frequency synchronization are crucial in some modern computer network applications. It enables 5G and 6G networks, and is proven to enhance the performance of data center workloads.
There is provided in accordance with an embodiment of the present disclosure, a system, including a digitally controlled oscillator (DCO) to generate a local clock having a local clock frequency, and clock synchronization circuitry to receive from a device a signal indicative of a remote clock frequency, compare measures of the remote clock frequency and the local clock frequency, generate a digital control command based on the comparison, and provide the digital control command to the DCO, wherein the DCO is to adjust the local clock frequency responsively to the digital control command.
Further in accordance with an embodiment of the present disclosure digital control command indicates a new clock frequency to which to change the local clock frequency by the DCO.
Still further in accordance with an embodiment of the present disclosure digital control command indicates a frequency adjustment to be made by the DCO to the local clock frequency.
Additionally in accordance with an embodiment of the present disclosure, the system includes a network interface to receive the signal including first packets from the device over a network and transmit a signal including second packets over the network, the clock synchronization circuitry is to compare a symbol rate of the received signal to a symbol rate of the transmitted signal to provide the comparison of the remote clock frequency and the local clock frequency.
Moreover, in accordance with an embodiment of the present disclosure, the system includes multiple chips, wherein the network interface and the clock synchronization circuitry are distributed among the multiple chips.
Further in accordance with an embodiment of the present disclosure the clock synchronization circuitry includes at least one counter in which the clock synchronization circuitry is to count symbols of the received signal and the transmitted signal in a time window, the clock synchronization circuitry is to run firmware to compute the comparison of the remote clock frequency and the local clock frequency based on at least one symbol count of the at least one counter.
Still further in accordance with an embodiment of the present disclosure, the system includes a clock input port to receive the signal from the device via a clock connection, the clock synchronization circuitry is configured to compare features of the received signal and the local clock to provide the comparison of the remote clock frequency and the local clock frequency.
Additionally in accordance with an embodiment of the present disclosure the clock synchronization circuitry includes clock chaining logic which includes at least one counter in which the clock chaining logic is to count the features of the received signal and the local clock in a time window, the clock synchronization circuitry being to run firmware to compute the comparison of the remote clock frequency and the local clock frequency based on at least one count value of the at least one counter.
Moreover, in accordance with an embodiment of the present disclosure, the system includes a clock output port to provide a clock synchronization signal to another device via another clock connection.
Further in accordance with an embodiment of the present disclosure the clock chaining logic is to proportionally change a frequency of the local clock to yield the clock synchronization signal, and provide the clock synchronization signal to the clock output port for output to the other device via the other clock connection.
Still further in accordance with an embodiment of the present disclosure clock synchronization circuitry is to run firmware to compute the comparison of the remote clock frequency and the local clock frequency.
Additionally in accordance with an embodiment of the present disclosure, the system includes a hardware clock to maintain a time-of-day value advancing at a frequency proportional to the local clock frequency of the local clock output by the DCO, wherein the DCO is connected to the hardware clock and is to output the local clock towards the hardware clock.
Moreover, in accordance with an embodiment of the present disclosure, the system includes a network interface including ports to send packets over a network at a symbol rate based on a frequency proportional to the local clock frequency, wherein DCO is connected to the network interface and is to output the local clock towards the network interface.
There is also provided in accordance with another embodiment of the present disclosure, a method, including generating a local clock having a local clock frequency with a digitally controlled oscillator (DCO), receiving from a device a signal indicative of a remote clock frequency, comparing measures of the remote clock frequency and the local clock frequency, generating a digital control command based on the comparison, and provide the digital control command to the DCO, and adjusting the local clock frequency by the DCO responsively to the digital control command.
Further in accordance with an embodiment of the present disclosure digital control command indicates a new clock frequency to which to change the local clock frequency by the DCO.
Still further in accordance with an embodiment of the present disclosure digital control command indicates a frequency adjustment to be made by the DCO to the local clock frequency.
Additionally in accordance with an embodiment of the present disclosure the receiving includes receiving the signal including first packets from the device over a network, the method further including transmitting a signal including second packets over the network, the comparing includes comparing a symbol rate of the received signal to a symbol rate of the transmitted signal to provide the comparison of the remote clock frequency and the local clock frequency.
Moreover, in accordance with an embodiment of the present disclosure, the method includes counting symbols of the received signal and the transmitted signal in a time window using at least one counter, and running firmware to compute the comparison of the remote clock frequency and the local clock frequency based on at least one symbol count of the at least one counter.
Further in accordance with an embodiment of the present disclosure the receiving includes receiving the signal from the device via a clock connection, and the comparing includes comparing features of the received signal and the local clock to provide the comparison of the remote clock frequency and the local clock frequency.
Still further in accordance with an embodiment of the present disclosure, the method includes counting the features of the received signal and the local clock in a time window using at least one counter, and running firmware to compute the comparison of the remote clock frequency and the local clock frequency based on at least one count value of the at least one counter.
Additionally in accordance with an embodiment of the present disclosure, the method includes providing a clock synchronization signal to another device via another clock connection.
Moreover, in accordance with an embodiment of the present disclosure, the method includes proportionally changing a frequency of the local clock to yield the clock synchronization signal.
Further in accordance with an embodiment of the present disclosure, the method includes running firmware to compute the comparison of the remote clock frequency and the local clock frequency.
Still further in accordance with an embodiment of the present disclosure, the method includes maintaining a time-of-day value advancing at a frequency proportional to the local clock frequency of the local clock output by the DCO.
Additionally in accordance with an embodiment of the present disclosure, the method includes sending packets over a network at a symbol rate based on a frequency proportional to the local clock frequency of the local clock output by the DCO.
The present invention will be understood from the following detailed description, taken in conjunction with the drawings in which:
A local device may transmit data including symbols to a remote device at a rate equal to, or proportional to, its local clock frequency. The local device may also receive data including symbols from the remote device at a rate equal to, or proportional to, the clock frequency of the remote device. Therefore, if the remote device is a master clock for the local device, then the local device may syntonize to the master clock by determining if its local clock is faster or slower than the master clock based on a difference between its send symbol rate and receive symbol rate from the remote device and adjust the frequency of its local clock accordingly.
There are many challenges to overcome when designing a local clock which needs to be syntonized with a master clock. Some clocks may have low phase noise but have poor drift stability over time. Other clocks, such as temperature or oven compensated clocks may have good drift stability but suffer from phase noise issues.
Embodiments of the present invention address at least some of the above challenges by providing a local clock with low phase noise and good drift stability using a digitally controlled oscillator (DCO) which is controlled by a digital control command. The DCO may include a temperature compensated oscillator and generate an output frequency of around 156.25 MHz. Clock synchronization circuitry tracks the difference between the frequency of the local clock and a remote master clock (e.g., using one or more counters to count received and transmitted symbols in a time window) and instructs the DCO, using a digital control command, to increase or decrease the rate of the local clock based on the tracked difference. The local clock is then adjusted (iteratively) until the local clock is synchronized with the remote master clock.
In some embodiments, firmware running on a firmware controller may read the counter value(s) in order to determine whether the local clock is faster or slower than the remote master clock. The firmware then generates the digital control command, which is then provided to the DCO.
In some embodiments, the remote master clock may be received from a link partner over a network. In some embodiments, clock synchronization is implemented with a network device including a network interface controller (NIC) application specific integrated circuit (ASIC). The ASIC includes ports, clock synchronization circuitry including a firmware controller, and a hardware clock. The ASIC is mounted on a printed circuit board (PCB), which also includes the DCO. The firmware controller provides digital control commands to the DCO via a digital control channel connecting the ASIC with the DCO. The DCO then provides the local clock to the ports and the hardware clock on the ASIC via another connection from the DCO to the ASIC. In some embodiments, the firmware controller may be included in an integrated circuit, which is external to the NIC ASIC.
In some cases, the identity of the master clock may not be known and only becomes known after time. Therefore, the receive symbol rates of all the ports may be tracked by the clock synchronization circuitry (e.g., using multiple counters) and when the identity of the master clock is known the firmware determines whether the local clock is faster or slower than the remote master clock (e.g., based on using the value of the relevant counter of the relevant port receiving the remote master clock).
In some embodiments, the network device may include multiple NIC ASICs, with each ASIC including ports and clock synchronization circuitry. The ASICs are mounted on a PCB with the DCO and firmware controller. The firmware controller identifies the port receiving the remote master clock from among the ports on the different ASICs and determines whether the local clock is faster or slower than the remote master clock and generates a digital control command and provides the digital control command to the DCO to adjust the frequency of the local clock. The local clock is provided to the ports on each of the ASICs and to the hardware clocks on each of the ASICs.
In some embodiments, the remote master clock may be received from a device connected via a clock connection, e.g., a clock cable or a printed circuit board (PCB) trace. Similarly, a clock synchronization signal based on the local clock may be generated and provided to another device connected to the local device via another clock connection, e.g., a clock cable or a PCB trace.
Reference is now made to
The clock synchronization system 10 includes a device 12. In the example of
The device 12 includes a digitally controller oscillator (DCO) 18 mounted on the network adapter board 14. The DCO 18 is configured to generate a local clock 46 having a local clock frequency. SiT5377 is a ±100 ppb precision MEMS Super-TCXO and is suitable for use as DTO 18. SiT5377 is commercially available from SiTime Corporation.
The network adapter board 14 (or each network adapter board 14) may include any one or more of the following: a core phase-locked loop (PLL) 20, clock synchronization circuitry 22, a hardware clock 24, a network interface 26, clock chaining logic 28, a firmware controller 30, a digital control channel 32, and a serializer/deserializer (SerDes) PLL 34. The network interface 26 may include one or more ports 36 to receive and/or transmit packets over a network 38 from/to one or more remote devices 40 (only one shown for the sake of simplicity). The network interface 26 may also include one or more counters 42 in which to count symbols sent and received by the ports 36 described in more detail with reference to
The DCO 18 is connected to the hardware clock 24 via the core PLL 20. The DCO 18 is configured to output a local clock 46 towards the hardware clock 24. The hardware clock 24 is configured to maintain a time-of-day value advancing at a frequency proportional to the local clock frequency of the local clock 46 output by the DCO 18. The core PLL 20 is configured to proportionally change the local clock 46 provided by the DCO 18 to a frequency (e.g., to a higher frequency by multiplying the local clock by a given value) used by the hardware clock 24. The time-of-day value of the hardware clock 24 may be used to provide time values for timestamping performed by the network interface 26.
The time-of-day of the hardware clock 24 may be synchronized to a remote master time-of-day or using any suitable method, for example, PTP. In some cases, the hardware clock 24 may be a master clock for time-of-day for another device. The device 12 may include a pulse per second (PPS) input port 48 and a PPS output port 50 for receiving and sending time-of-day synchronization data, respectively.
The DCO 18 is connected to the network interface 26 via the SerDes PLL 34. The DCO 18 is configured to output the local clock 46 towards the network interface 26. The SerDes PLL 34 is configured to proportionally change the local clock 46 provided by the DCO 18 to a frequency (e.g., to a higher frequency by multiplying the local clock by a given value) used by the network interface 26. The ports 36 of the network interface 26 are configured to send packets over the network 38 at a symbol rate based on a frequency proportional to the local clock frequency.
In some embodiments, the network interface 26 is configured to count symbols received by one or more of the ports 36 and count symbols sent by one or more of the ports 36. Separate counters 42 may be maintained for sent and received symbols per port 36 or a combined counter may be maintained for both sent and received symbols per port 36 so that a difference between symbols sent and received in a time window is tracked by a counter per port 36, for example, by incrementing a counter for received symbols and decrementing that counter for sent symbols, or vice-versa.
The clock synchronization circuitry 22 may be implemented to include the clock chaining logic 28, the firmware controller 30, and the counters 42 of the network interface 26. The values of the counter(s) 42 and/or the counter(s) 44 are exposed to the firmware controller 30 to enable the firmware controller 30 to determine whether the local clock provided by the DCO 18 is faster or slower than a remote master clock, as described in more detail with reference to
In some situations, the remote master clock may be received from one of the remote devices 40 over the network 38. The symbol rate of the data received from that remote device 40 is counted by the network interface 26 using the counters 42. Controlling the DCO 18 based on the remote master clock being received from the remote device 40 is described in more detail with reference to
In some embodiments, the device 12 may include a peripheral device communication data bus interface (e.g., PCIe interface) (not shown) in addition to, or instead of the network interface 26. The communication data bus interface may be configured to count symbols sent and received over a high-speed link disposed between the device 12 and one of the remote devices 40. The clock synchronization circuitry 22 may be configured to synchronize the local clock 46 generated by the DCO 18 to a remote master clock received from one of the remote devices 40 via the communication data bus interface.
The device 12 may also include a clock input port 56. In some situations, the clock chaining logic 28 may be configured to receive the remote master clock as a clock synchronization signal from a chained device 52 via a clock connection 54 (e.g., a clock cable or PCB trace) connected to the clock input port 56. The DCO 18 may also be connected to the clock chaining logic 28 and be configured to provide the local clock 46 to the clock chaining logic 28. Features (e.g., pulses of the signals or other identifiable sections of the signal waveform such as rising edges of the waveform) of the clock synchronization signal and local clock 46 are counted by the clock chaining logic 28 using the counters 44. As the local clock 46 and the clock synchronization signal may be at different base frequencies, for example, the local clock 46 may be around 156.25 MHz whereas the clock synchronization signal may be around 10 MHz, either the counting method of the features of the local clock 46 and the clock synchronization signal need to be adjusted, and/or the computation of the frequency difference needs to take into account the difference between the base frequencies, or some other method may be employed to equate the differences in the base frequencies. The device 12 may also include a clock output port 60. The clock chaining logic 28 may multiply or divide the local clock 46 by a given factor to yield a clock synchronization signal for output to another chained device 58 via a clock connection 62 (e.g., a clock cable or PCB trace) connected to the clock output port 60.
When the device 12 includes multiple network adapter chips 16, the network interface 26 and the clock synchronization circuitry 22 may be distributed among the multiple chips 16.
In practice, some, or all of these functions of the clock synchronization circuitry 22 may be combined in a single physical component or, alternatively, implemented using multiple physical components. These physical components may comprise hard-wired or programmable devices, or a combination of the two. In some embodiments, at least some of the functions of the clock synchronization circuitry 22 may be carried out by a programmable processor under the control of suitable software. This software may be downloaded to a device in electronic form, over a network, for example. Alternatively, or additionally, the software may be stored in tangible, non-transitory computer-readable storage media, such as optical, magnetic, or electronic memory.
Reference is now made to
One of the ports 36 (designated as receiving the master clock) of the network interface 26 is configured to receive a signal including data (e.g., first packets) from the device 40 over the network 38 and transmit a signal including data (e.g., second packets) over the network 38 (block 202).
The clock synchronization circuitry 22 is configured to receive the signal from the device 40, the signal being indicative of a remote clock frequency (block 204). In some embodiments, the part of the network interface 26 included in the clock synchronization circuitry 22 is configured to receive from the device 40 the signal indicative of the remote clock frequency (block 204). The data rate, e.g., symbol rate, of the received signal is typically indicative of the remote clock frequency. The remote clock frequency is the frequency of a remote clock which is remote to the local clock 46 generated by the DCO 18. The remote clock may be external to the clock synchronization system 10. The remote clock may be internal to the clock synchronization system 10 but not generated by the DCO 18. For example, the network adapter board 14 may include another DCO which generates the remote clock for example. This latter example may be more relevant to the embodiment of
In some embodiments, the part of the network interface 26 included in the clock synchronization circuitry 22 is configured to count symbols of the received signal and the transmitted signal in a time window using the counter(s) 42 (block 206). The firmware controller 30 may be configured to send a command to the network interface 26 to initiate counting of the symbols. The symbols of the received signal may be counted on one of the counters 42 and the symbols of the transmitted signal may be counted on another one of the counters 42. In some embodiments, the symbols of the received signal and the transmitted signal may be counted on the same counter 42 such that the counter 42 is incremented for every received symbol and decremented for every transmitted symbol, or vice-versa.
The clock synchronization circuitry 22 is configured to compare measures of the remote clock frequency and the local clock frequency (block 208). In some embodiments, the firmware controller 30 of the clock synchronization circuitry 22 is configured to run firmware to compute the comparison of the remote clock frequency and the local clock frequency.
In some embodiments, the clock synchronization circuitry 22 is configured to compare the symbol rate of the received signal to a symbol rate of the transmitted signal to provide the comparison of the remote clock frequency and the local clock frequency (block 210). In some embodiments, the firmware controller 30 of the clock synchronization circuitry 22 is configured to run the firmware to compute the comparison of the remote clock frequency and the local clock frequency based on the symbol count of the counter(s) 42 (block 212). For example, the comparison may be equal to the difference between the symbol count of received symbols and symbol count of the transmitted symbols in the time window.
The firmware controller 30 of the clock synchronization circuitry 22 is configured to run the firmware to generate a digital control command based on the comparison (block 214). For example, if the symbol count of received symbols is greater than symbol count of the transmitted symbols in the time window, then the remote clock is faster than the local clock and therefore the speed of the local clock should be increased, and vice-versa. The adjustment to the speed of the local clock may be proportional to the difference between the symbol count of received symbols and symbol count of the transmitted symbols in the time window.
In some embodiments, the digital control command indicates a new clock frequency to which to change the local clock frequency by the DCO 18. For example, the digital control command may take the following form: “set your frequency to your initial frequency multiplied by X/Y”, where X and Y could be any suitable values such as 100 million plus 1 and 100 million, or 100 billion plus 1 and 100 billion. The interface may also be a single number, which multiplies the initial frequency, such as 1.00000001.
In some embodiments, the digital control command indicates a frequency adjustment to be made by the DCO 18 to the local clock frequency. For example, if the current frequency of the local clock 46 is 156.25 MHz and the local clock 46 is slow by 1 Hz, the digital control command may indicate that the new clock frequency should be increased by 1 Hz.
The firmware controller 30 is configured to provide the digital control command to the DCO 18 via the digital control channel 32 (block 216). The DCO 18 is configured to adjust the local clock frequency responsively to the digital control command received from the firmware controller 30 (block 218). The DCO 18 is configured to output the local clock 46 towards the hardware clock 24 and the network interface 26 via the core PLL 20 and the SerDes PLL 34, respectively (block 220).
Reference is now made to
The clock input port 56 is configured to receive a first clock synchronization signal via clock connection 54 from chained device 52 (block 302). The clock chaining logic 28 is configured to receive the first clock synchronization signal from the clock input port 56 (block 304). The first clock synchronization signal is indicative of the remote clock frequency of the clock running on the chained device 52. The first clock synchronization signal is generally an analogue signal having a frequency proportional to the frequency of the clock running on the chained device 52. For example, the clock running on the chained device 52 may have a frequency around 156.25 MHz, while the clock synchronization signal has a frequency around 10 MHz. The remote clock frequency is the frequency of a remote clock which is remote to local clock generated by the DCO 18. The remote clock may be external to the clock synchronization system 10. The remote clock may be internal to the clock synchronization system 10 but not generated by the DCO 18. For example, the network adapter board 14 may include chained device 52 which includes another DCO which generates the remote clock.
In some embodiments, the clock chaining logic 28 is configured to count features of the received first clock synchronization signal and the local clock 46 in a time window using the counter(s) 44 (block 306). The features may include pulses of the signals or other identifiable sections of the signal waveform, e.g., rising edges of the waveform. The firmware controller 30 may be configured to send a command to the clock chaining logic 28 to initiate counting of the features. The features of the received first clock synchronization signal may be counted on one of the counters 44 and the features of the local clock 46 may be counted on another one of the counters 44. In some embodiments, the features of the first clock synchronization signal and the local clock 46 may be counted on the same counter 42 such that the counter 44 is incremented for every feature of the first clock synchronization signal and decremented for every feature of the local clock 46 (assuming the signals have compatible frequencies), or vice-versa.
The clock synchronization circuitry 22 is configured to compare measures of the remote clock frequency and the local clock frequency (block 308). In some embodiments, the firmware controller 30 of the clock synchronization circuitry 22 is configured to run firmware to compute the comparison of the remote clock frequency and the local clock frequency. In some embodiments, the clock synchronization circuitry 22 is configured to compare a number of features of the first clock synchronization signal to a number of features of the local clock 46 to provide the comparison of the remote clock frequency and the local clock frequency (block 310). In some embodiments, the firmware controller 30 of the clock synchronization circuitry 22 is configured to run the firmware to compute the comparison of the remote clock frequency and the local clock frequency based on the count value(s) of the counter(s) 44 (block 312). For example, the comparison may be equal to the difference between the feature count of the first clock synchronization signal and the feature count of the local clock 46 in the time window.
The firmware controller 30 of the clock synchronization circuitry 22 is configured to run the firmware to generate a digital control command based on the comparison (block 314). For example, if the feature count of the first clock synchronization signal is greater than the feature count of the local clock 46 in the time window, then the remote clock (as indicated by the first clock synchronization signal) is faster than the local clock 46 and therefore the speed of the local clock 46 should be increased. The adjustment to the speed of the local clock 46 may be proportional to the difference between the feature count of the first clock synchronization signal and the feature count of the local clock 46 in the time window.
In some embodiments, the digital control command indicates a new clock frequency to which to change the local clock frequency by the DCO 18. For example, the digital control command may take the following form: “set your frequency to your initial frequency multiplied by X/Y”, where X and Y could be any suitable values such as 100 million plus 1 and 100 million, or 100 billion plus 1 and 100 billion. The interface may also be a single number, which multiplies the initial frequency, such as 1.00000001.
In some embodiments, the digital control command indicates a frequency adjustment to be made by the DCO 18 to the local clock frequency. For example, if the current frequency of the local clock 46 is 156.25 MHz and the local clock 46 is slow by 1 Hz, the digital control command may indicate that the new clock frequency should be increased by 1 Hz.
The firmware controller 30 is configured to provide the digital control command to the DCO 18 via the digital control channel 32 (block 316). The DCO 18 is configured to adjust the local clock frequency responsively to the digital control command received from the firmware controller 30 (block 318). The DCO 18 is configured to output the local clock 46 towards the hardware clock 24 and the network interface 26 via the core PLL 20 and the SerDes PLL 34, respectively (block 320).
The clock chaining logic 28 may be configured to: proportionally change (e.g., step up or step down but with a constant proportion of the base frequency of the local clock 46) a frequency of the local clock 46 to yield a second clock synchronization signal; and provide the second clock synchronization signal to the clock output port 60 for output to the chained device 58 via the clock connection 62 (clock 322). The clock output port 60 is configured to provide the second clock synchronization signal to the chained device 58 via the clock connection 62.
Various features of the invention which are, for clarity, described in the contexts of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment may also be provided separately or in any suitable sub-combination.
The embodiments described above are cited by way of example, and the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.
Number | Name | Date | Kind |
---|---|---|---|
5392421 | Lennartsson | Feb 1995 | A |
5402394 | Turski | Mar 1995 | A |
5416808 | Witsaman et al. | May 1995 | A |
5491792 | Grisham et al. | Feb 1996 | A |
5564285 | Jurewicz et al. | Oct 1996 | A |
5592486 | Lo et al. | Jan 1997 | A |
5896524 | Halstead, Jr. et al. | Apr 1999 | A |
6055246 | Jones | Apr 2000 | A |
6081143 | Ho | Jun 2000 | A |
6084856 | Simmons et al. | Jul 2000 | A |
6144714 | Bleiweiss et al. | Nov 2000 | A |
6199169 | Voth | Mar 2001 | B1 |
6289023 | Dowling et al. | Sep 2001 | B1 |
6449291 | Burns et al. | Sep 2002 | B1 |
6535926 | Esker | Mar 2003 | B1 |
6556636 | Takagi | Apr 2003 | B1 |
6556638 | Blackburn | Apr 2003 | B1 |
6718476 | Shima | Apr 2004 | B1 |
6918049 | Lamb et al. | Jul 2005 | B2 |
7111184 | Thomas, Jr. et al. | Sep 2006 | B2 |
7191354 | Purho | Mar 2007 | B2 |
7245627 | Goldenberg et al. | Jul 2007 | B2 |
7254646 | Aguilera et al. | Aug 2007 | B2 |
7334124 | Pham et al. | Feb 2008 | B2 |
7412475 | Govindarajalu | Aug 2008 | B1 |
7440474 | Goldman et al. | Oct 2008 | B1 |
7447975 | Riley | Nov 2008 | B2 |
7483448 | Bhandari et al. | Jan 2009 | B2 |
7496686 | Coyle | Feb 2009 | B2 |
7535933 | Zerbe et al. | May 2009 | B2 |
7623552 | Jordan et al. | Nov 2009 | B2 |
7636767 | Lev-Ran et al. | Dec 2009 | B2 |
7650158 | Indirabhai | Jan 2010 | B2 |
7656751 | Rischar et al. | Feb 2010 | B2 |
7750685 | Bunch et al. | Jul 2010 | B1 |
7904713 | Zajkowski et al. | Mar 2011 | B1 |
7941684 | Serebrin et al. | May 2011 | B2 |
8065052 | Fredriksson et al. | Nov 2011 | B2 |
8072313 | Drucker | Dec 2011 | B2 |
8300749 | Hadzic et al. | Oct 2012 | B2 |
8341454 | Kondapalli | Dec 2012 | B1 |
8370675 | Kagan | Feb 2013 | B2 |
8407478 | Kagan et al. | Mar 2013 | B2 |
8607086 | Cullimore | Dec 2013 | B2 |
8693506 | Hadzic | Apr 2014 | B2 |
8699406 | Charles et al. | Apr 2014 | B1 |
8824903 | Christensen | Sep 2014 | B2 |
8879552 | Zheng | Nov 2014 | B2 |
8930647 | Smith | Jan 2015 | B1 |
9344265 | Karnes | May 2016 | B2 |
9397960 | Arad et al. | Jul 2016 | B2 |
9549234 | Mascitto | Jan 2017 | B1 |
9753854 | Bao | Sep 2017 | B1 |
9942025 | Bosch et al. | Apr 2018 | B2 |
9979998 | Pogue et al. | May 2018 | B1 |
10014937 | Di Mola et al. | Jul 2018 | B1 |
10027601 | Narkis et al. | Jul 2018 | B2 |
10054977 | Mikhaylov et al. | Aug 2018 | B2 |
10095543 | Griffin et al. | Oct 2018 | B1 |
10122527 | Fischer | Nov 2018 | B1 |
10148258 | Carlson et al. | Dec 2018 | B2 |
10164759 | Volpe | Dec 2018 | B1 |
10320646 | Mirsky et al. | Jun 2019 | B2 |
10515045 | Mattina | Dec 2019 | B1 |
10637776 | Iwasaki | Apr 2020 | B2 |
10727966 | Izenberg et al. | Jul 2020 | B1 |
10778361 | Almog et al. | Sep 2020 | B1 |
10778406 | Gaist et al. | Sep 2020 | B2 |
10785015 | Rada | Sep 2020 | B1 |
10841243 | Levi et al. | Nov 2020 | B2 |
10879910 | Franck et al. | Dec 2020 | B1 |
10887077 | Ivry | Jan 2021 | B1 |
10908635 | Ranganathan et al. | Feb 2021 | B1 |
11070224 | Faig et al. | Jul 2021 | B1 |
11070304 | Levi et al. | Jul 2021 | B1 |
11128500 | Mentovich et al. | Sep 2021 | B1 |
11157433 | Lederman et al. | Oct 2021 | B2 |
11240079 | Kushnir et al. | Feb 2022 | B1 |
11303363 | Mohr et al. | Apr 2022 | B1 |
11336383 | Mula et al. | May 2022 | B2 |
11368768 | Bakopoulos et al. | Jun 2022 | B2 |
11379334 | Srinivasan et al. | Jul 2022 | B1 |
11388263 | Levi et al. | Jul 2022 | B2 |
11476928 | Levi et al. | Oct 2022 | B2 |
11606157 | Wasko et al. | Mar 2023 | B1 |
11706014 | Manevich et al. | Jul 2023 | B1 |
20010006500 | Nakajima et al. | Jul 2001 | A1 |
20020027886 | Fischer et al. | Mar 2002 | A1 |
20020031199 | Rolston et al. | Mar 2002 | A1 |
20040096013 | Laturell et al. | May 2004 | A1 |
20040153907 | Gibart | Aug 2004 | A1 |
20050033947 | Morris et al. | Feb 2005 | A1 |
20050172181 | Huliehel | Aug 2005 | A1 |
20050268183 | Barmettler | Dec 2005 | A1 |
20060109376 | Chaffee et al. | May 2006 | A1 |
20060171496 | Nakamuta et al. | Aug 2006 | A1 |
20070008044 | Shimamoto | Jan 2007 | A1 |
20070072451 | Tazawa et al. | Mar 2007 | A1 |
20070104098 | Kimura et al. | May 2007 | A1 |
20070124415 | Lev-Ran et al. | May 2007 | A1 |
20070139085 | Elliot et al. | Jun 2007 | A1 |
20070159924 | Vook et al. | Jul 2007 | A1 |
20070266119 | Ohly | Nov 2007 | A1 |
20080069150 | Badt et al. | Mar 2008 | A1 |
20080225841 | Conway et al. | Sep 2008 | A1 |
20080285597 | Downey et al. | Nov 2008 | A1 |
20090167443 | Liu et al. | Jul 2009 | A1 |
20090257458 | Cui et al. | Oct 2009 | A1 |
20100280858 | Bugenhagen | Nov 2010 | A1 |
20110110360 | Fenwick et al. | May 2011 | A1 |
20110182191 | Jackson | Jul 2011 | A1 |
20110194425 | Li et al. | Aug 2011 | A1 |
20120063556 | Hoang | Mar 2012 | A1 |
20120076319 | Terwal | Mar 2012 | A1 |
20120301134 | Davari et al. | Nov 2012 | A1 |
20130039359 | Bedrosian | Feb 2013 | A1 |
20130045014 | Mottahedin et al. | Feb 2013 | A1 |
20130215889 | Zheng et al. | Aug 2013 | A1 |
20130235889 | Aweya et al. | Sep 2013 | A1 |
20130294144 | Wang et al. | Nov 2013 | A1 |
20130315265 | Webb, III et al. | Nov 2013 | A1 |
20130336435 | Akkihal et al. | Dec 2013 | A1 |
20140085141 | Geva et al. | Mar 2014 | A1 |
20140153680 | Garg et al. | Jun 2014 | A1 |
20140185216 | Zeng et al. | Jul 2014 | A1 |
20140185632 | Steiner et al. | Jul 2014 | A1 |
20140253387 | Gunn et al. | Sep 2014 | A1 |
20140281036 | Cutler et al. | Sep 2014 | A1 |
20140301221 | Nadeau et al. | Oct 2014 | A1 |
20140321285 | Chew et al. | Oct 2014 | A1 |
20150019839 | Cardinell et al. | Jan 2015 | A1 |
20150078405 | Roberts | Mar 2015 | A1 |
20150092793 | Aweya | Apr 2015 | A1 |
20150127978 | Cui et al. | May 2015 | A1 |
20150131766 | Chen | May 2015 | A1 |
20150163050 | Han et al. | Jun 2015 | A1 |
20150318941 | Zheng et al. | Nov 2015 | A1 |
20160057518 | Neudorf | Feb 2016 | A1 |
20160072602 | Earl et al. | Mar 2016 | A1 |
20160110211 | Karnes | Apr 2016 | A1 |
20160140066 | Worrell et al. | May 2016 | A1 |
20160277138 | Garg et al. | Sep 2016 | A1 |
20160285574 | White et al. | Sep 2016 | A1 |
20160315756 | Tenea et al. | Oct 2016 | A1 |
20170005903 | Mirsky | Jan 2017 | A1 |
20170017604 | Chen et al. | Jan 2017 | A1 |
20170126589 | Estabrooks et al. | May 2017 | A1 |
20170160933 | De Jong et al. | Jun 2017 | A1 |
20170214516 | Rivaud et al. | Jul 2017 | A1 |
20170302392 | Farra et al. | Oct 2017 | A1 |
20170331926 | Raveh et al. | Nov 2017 | A1 |
20170359137 | Butterworth et al. | Dec 2017 | A1 |
20180059167 | Sharf et al. | Mar 2018 | A1 |
20180152286 | Kemparaj et al. | May 2018 | A1 |
20180188698 | Dionne et al. | Jul 2018 | A1 |
20180191802 | Yang et al. | Jul 2018 | A1 |
20180227067 | Hu et al. | Aug 2018 | A1 |
20180309654 | Achkir et al. | Oct 2018 | A1 |
20190007189 | Hossain et al. | Jan 2019 | A1 |
20190014526 | Bader et al. | Jan 2019 | A1 |
20190089615 | Branscomb et al. | Mar 2019 | A1 |
20190149258 | Araki et al. | May 2019 | A1 |
20190158909 | Kulkarni et al. | May 2019 | A1 |
20190196563 | Lai | Jun 2019 | A1 |
20190220300 | Rosenboom | Jul 2019 | A1 |
20190265997 | Merrill et al. | Aug 2019 | A1 |
20190273571 | Bordogna et al. | Sep 2019 | A1 |
20190319729 | Leong et al. | Oct 2019 | A1 |
20190349392 | Wetterwald et al. | Nov 2019 | A1 |
20190379714 | Levi et al. | Dec 2019 | A1 |
20200044657 | Pi et al. | Feb 2020 | A1 |
20200162234 | Almog et al. | May 2020 | A1 |
20200235905 | Su et al. | Jul 2020 | A1 |
20200287557 | Horovitz | Sep 2020 | A1 |
20200304224 | Neugeboren | Sep 2020 | A1 |
20200331480 | Zhang et al. | Oct 2020 | A1 |
20200344333 | Hawari et al. | Oct 2020 | A1 |
20200396050 | Perras et al. | Dec 2020 | A1 |
20200401434 | Thampi et al. | Dec 2020 | A1 |
20210006344 | Chen et al. | Jan 2021 | A1 |
20210141413 | Levi et al. | May 2021 | A1 |
20210175991 | Neugeboren | Jun 2021 | A1 |
20210218431 | Narayanan et al. | Jul 2021 | A1 |
20210243140 | Levi et al. | Aug 2021 | A1 |
20210288785 | Faig et al. | Sep 2021 | A1 |
20210297151 | Levi et al. | Sep 2021 | A1 |
20210297230 | Dror et al. | Sep 2021 | A1 |
20210318978 | Hsung | Oct 2021 | A1 |
20210328900 | Sattinger et al. | Oct 2021 | A1 |
20210392065 | Sela et al. | Dec 2021 | A1 |
20210409031 | Ranganathan et al. | Dec 2021 | A1 |
20220006606 | Levi et al. | Jan 2022 | A1 |
20220021393 | Ravid et al. | Jan 2022 | A1 |
20220066978 | Mishra et al. | Mar 2022 | A1 |
20220086105 | Levi et al. | Mar 2022 | A1 |
20220173741 | Ravid et al. | Jun 2022 | A1 |
20220191275 | Levi et al. | Jun 2022 | A1 |
20220121691 | Mentovich et al. | Jul 2022 | A1 |
20220224500 | Mula et al. | Jul 2022 | A1 |
20220239549 | Zhao et al. | Jul 2022 | A1 |
20220261032 | Bateni | Aug 2022 | A1 |
20220342086 | Yoshida | Oct 2022 | A1 |
20220352998 | Levi et al. | Nov 2022 | A1 |
20220357763 | Levy et al. | Nov 2022 | A1 |
20220360423 | Levi et al. | Nov 2022 | A1 |
20220385598 | Pismenny et al. | Dec 2022 | A1 |
20220416925 | Levi et al. | Dec 2022 | A1 |
20230076889 | Rabinovich et al. | Mar 2023 | A1 |
20230185600 | Wasko et al. | Jun 2023 | A1 |
20230236624 | Levi et al. | Jul 2023 | A1 |
20230251899 | Levi et al. | Aug 2023 | A1 |
20230361900 | Levi et al. | Nov 2023 | A1 |
20230362096 | Manevich et al. | Nov 2023 | A1 |
Number | Date | Country |
---|---|---|
101059825 | Oct 2007 | CN |
101388741 | Mar 2009 | CN |
106817183 | Jun 2017 | CN |
108667547 | Oct 2018 | CN |
108829493 | Nov 2018 | CN |
1215559 | Sep 2007 | EP |
2770678 | Aug 2014 | EP |
2011091676 | May 2011 | JP |
498259 | Aug 2002 | TW |
2012007276 | Jan 2012 | WO |
2013124782 | Aug 2013 | WO |
2013143112 | Oct 2013 | WO |
2014029533 | Feb 2014 | WO |
2014138936 | Sep 2014 | WO |
Entry |
---|
U.S. Appl. No. 17/313,026 Office Action dated Dec. 19, 2023. |
Zhang et al., “TI BAW technology enables ultra-low jitter clocks for highspeed networks”, White paper, Texas Instruments, pp. 1-11, Feb. 2019. |
Skywork Solutions Inc., “PCI Express 3.1 Jitter Requirements”, AN562, pp. 1-16, year 2021. |
Intel, “Can Altera GX/GT/GZ device high speed transceivers handle Spread Spectrum Clocking (SSC), as required py PCle or SATA/SAS protocols?”, p. 1, Sep. 11, 2012. |
Shapira et al., U.S. Appl. No. 17/534,776, filed Nov. 24, 2021. |
Shapira et al., U.S. Appl. No. 17/578,115, filed Jan. 18, 2022. |
Kernen et al., U.S. Appl. No. 17/858,236, filed Jul. 6, 2022. |
Manevich et al., U.S. Appl. No. 17/885,604, filed Aug. 11, 2022. |
Manevich et al., U.S. Appl. No. 18/067,767, filed Dec. 19, 2022. |
“IEEE Standard for Local and Metropolitan Area Networks—Timing and Synchronization for Time-Sensitive Applications,” IEEE Std 802.1AS-2020, IEEE Computer Society, pp. 1-421, year 2020. |
U.S. Appl. No. 17/871,937 Office Action dated Aug. 1, 2023. |
U.S. Appl. No. 17/578,115 Office Action dated Apr. 26, 2023. |
U.S. Appl. No. 17/534,776 Office Action dated Jun. 29, 2023. |
SiTime Corporation, “Sit5377—60 to 220 MHz, ±100 ppb Elite RF™ Super-TCXO,” Product Description, pp. 1-3, last updated Mar. 18, 2023 as downloaded from https://web.archive.org/web/20230318094421/https://www.sitime.com/products/super-tcxos/sit5377. |
PCI-SIG, “PCI Express®—Base Specification—Revision 3.0,” pp. 1-860, Nov. 10, 2010. |
U.S. Appl. No. 17/191,736 Office Action dated Jun. 26, 2023. |
Corbett et al., “Spanner: Google's Globally Distributed Database,” ACM Transactions on Computer Systems, vol. 31, No. 3, article 8, pp. 1-22, Aug. 2013. |
Wasko et al., U.S. Appl. No. 18/111,916, filed Feb. 21, 2023. |
Manevich et al., U.S. Appl. No. 18/349,976, filed Jul. 11, 2023. |
IEEE Standard 1588™-2008: “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, IEEE Instrumentation and Measurement Society, Revision of IEEE Standard 1588-2002, USA, pp. 1-289, Jul. 24, 2008. |
Weibel et al., “Implementation and Performance of Time Stamping Techniques”, 2004 Conference on IEEE 1588, pp. 1-29, Sep. 28, 2004. |
Working Draft Project American National Standard T10/1799-D, “Information Technology—SCSI Block Commands—3 (SBC-3)”, pp. 1-220, Revision 19, May 29, 2009. |
“Infiniband Architecture: Specification vol. 1”, pp. 1-1727, Release 1.2.1, Infiniband Trade Association, Nov. 2007. |
Mellanox Technologies, “Mellanox ConnectX IB: Dual-Port InfiniBand Adapter Cards with PCI Express 2.0”, pp. 1-2, USA, year 2008. |
Wikipedia—“Precision Time Protocol”, pp. 1-8, Aug. 24, 2019. |
IEEE Std 1588-2002, “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, IEEE Instrumentation and Measurement Society, pp. 1-154, Nov. 8, 2002. |
Weibel, H., “High Precision Clock Synchronization according to IEEE 1588 Implementation and Performance Issues”, Zurich University of Applied Sciences, pp. 1-9, Jan. 17, 2005. |
Lu et al., “A Fast CRC Update Implementation”, Computer Engineering Laboratory, Electrical Engineering Department, pp. 113-120, Oct. 8, 2003. |
Texas Instruments, “LMK05318 Ultra-Low Jitter Network Synchronizer Clock With Two Frequency Domains,” Product Folder, pp. 1-86, Dec. 2018. |
Dlugy-Hegwer et al., “Designing and Testing IEEE 1588 Timing Networks”, Symmetricom, pp. 1-10, Jan. 2007. |
Mellanox Technologies, “How to test 1PPS on Mellanox Adapters”, pp. 1-6, Oct. 22, 2019 downloaded from https://community.mellanox.com/s/article/How-To-Test-1PPS-on-Mellanox-Adapters. |
ITU-T recommendation, “G.8273.2/Y.1368.2—Timing characteristics of telecom boundary clocks and telecom time slave clocks”, pp. 1-50, Jan. 2017. |
Evi et al., U.S. Appl. No. 17/871,937, filed Jul. 24, 2022. |
Ipclock, “IEEE 1588 Primer,” ip-clock.com, pp. 1-3, May 1, 2017 (downloaded from https://web.archive.org/web/20170501192647/http://ip-clock.com/ieee-1588-primer/). |
ITU-T Standard G.8261/Y.1361, “Timing and synchronization aspects in packet networks”, pp. 1-120, Aug. 2019. |
Levi et al., U.S. Appl. No. 17/868,841, filed Jul. 20, 2022. |
“Precision Time Protocol,” PTP Clock Types, Cisco, pp. 1-52, Jul. 30, 2020, as downloaded from https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/5x/system-management-configuration/cisco-apic-system-management-configuration-guide-52x/m-precision-time-protocol.pdf. |
ITU-T Standard G.8262/Y.1362, “Timing characteristics of synchronous equipment slave clock”, pp. 1-44, Nov. 2018. |
ITU-T Standard G.8264/Y.1364, “Distribution of timing information through packet networks”, pp. 1-42, Aug. 2017. |
Manevich et al., U.S. Appl. No. 17/867,779, filed Jul. 19, 2022. |
CN Application # 202210456108.6 Office Action dated Aug. 30, 2024. |
U.S. Appl. No. 18/111,916 Office Action dated Jun. 5, 2024. |
Obleukhov et al., “Simple Precision Time Protocol at Meta,” pp. 1-15, Feb. 7, 2024, as downloaded from https://engineering.fb.com/2024/02/07/production-engineering/simple-precision-time-protocol-sptp-meta/. |
Behn, “meinberg-sync / flashptpd,” Meinberg Funkuhren GmbH & Co. KG, pp. 1-6, year 2023. |
Arnold et al., “Enterprise Profile for the Precision Time Protocol with Mixed Multicast and Unicast Messages,” Internet Engineering Task Force (IETF), Tictoc Workgroup, Internet Draft, pp. 1-13, Nov. 23, 2023. |
Wikipedia, “Network Time Protocol,” pp. 1-20, last edited Mar. 31, 2024. |
Silicon Laboratories Inc., “Temperature-Compensated Oscillator Example,” AN365, Rev. 0.1, pp. 1-8, Nov. 9, 2009. |
Sutton et al., “Reinforcement Learning,” Chapter 6.6—“Actor-Critic Methods,” The MIT Press, e-book, pp. 1-3, Jan. 4, 2005, as downloaded from incompleteideas.net/book/first/ebook/node66.html. |
ITU-T Recommendation, “G.8262.1/Y.1362.1—Series G: Transmission Systems and Media, Digital Systems and Networks—Packet over Transport Aspects—Synchronization, Quality and Availability Targets Series Y: Global Information Infrastructure, Internet Protocol Aspects, Next-Generation Networks, Internet of Things and Smart Cities—Timing Characteristics of Enhanced Synchronous Equipment Slave Clock,” pp. 1-26, Jan. 2019. |
Number | Date | Country | |
---|---|---|---|
20250055668 A1 | Feb 2025 | US |