The present invention relates generally to time synchronization in communication networks, and particularly to network adapters providing time services to hosts.
Computer and communication networks may use various schemes and protocols for synchronizing network nodes to a common time-base. One common example of such a protocol is the Precision Time Protocol (PTP) defined in IEEE standards 1588-2002 and 1588-2008, and later versions thereof. PTP is used to synchronize clocks throughout a computer network and may achieve sub-microsecond accuracy.
U.S. Pat. No. 8,370,675 describes a method for clock synchronization which includes computing an offset value between a local clock time of a real-time clock circuit and a reference clock time, and loading the offset value into a register that is associated with the real-time clock circuit. The local clock time is then summed with the value in the register so as to give an adjusted value of the local clock time that is synchronized with the reference clock.
U.S. Patent Application Publication 2020/0162234 describes an apparatus including a shared device in communication with a plurality of computing machines external to the shared device. The shared device includes at least one PTP domain coefficient storage area, the at least one PTP domain coefficient storage area receiving a PTP coefficient from a computing machine having a PTP client, and providing the PTP coefficient to a computing machine not having a PTP client.
An embodiment of the present invention that is described herein provides a network adapter including a network port for communicating with a communication network, a hardware clock, and circuitry. The circuitry is coupled to receive from the communication network, via the network port, one or more time-protocol packets that convey a network time used for synchronizing network devices in the communication network, to align the hardware clock to the network time conveyed in the time-protocol packets, and to make the network time available to one or more time-service consumers running in a host served by the network adapter.
In some embodiments, the circuitry is coupled to communicate with the host over a peripheral bus, and to make the network time available to the time-service consumers using a peripheral-bus device exposed on the peripheral bus. In an example embodiment, the peripheral-bus device is a read-only bus device that transfers information only unidirectionally from the peripheral bus to the host.
In a disclosed embodiment, the circuitry includes a processor coupled to run software that aligns the hardware clock. In an embodiment, the time-protocol packets comply with a time-synchronization protocol, and the processor is coupled to run a software stack of the time-synchronization protocol, the software stack aligning the hardware clock to the network time.
In another embodiment, the circuitry is coupled to apply to the time-protocol packets timestamps having a defined timestamping accuracy, and the processor is coupled to align the hardware clock with an accuracy that matches the timestamping accuracy. In yet another embodiment, the circuitry is coupled to isolate the time-protocol packets from the host. In still another embodiment, the time-protocol packets comply with a time-synchronization protocol, and the circuitry is coupled to isolate a control plane of the time-synchronization protocol from the host.
In an embodiment, the circuitry is coupled to calculate, based on the received time-protocol packets, one or more corrections that align the hardware clock to the network time, and to apply the calculated corrections to the hardware clock. In an embodiment, the circuitry is coupled to align both a frequency and a time offset of the hardware clock to the network time, so as to make an offset-free value of the network time available to the host. In another embodiment, the circuitry is coupled to make available a value of the network time that is usable by the time-service consumers without translation or conversion.
Typically, the circuitry is further coupled to provide networking services to the host, concurrently with making the network time available to the host.
There is additionally provided, in accordance with an embodiment of the present invention, a compute node including a host and a network adapter. The host includes one or more processors for running one or more time-service consumer processes. The network adapter includes a hardware clock and is coupled to receive, from a communication network, one or more time-protocol packets that convey a network time used for synchronizing network devices in the communication network, to align the hardware clock to the network time conveyed in the time-protocol packets, and to make the network time available to the time-service consumer processes running in the host.
In various embodiments, the time-service consumer processes running in the host are of at least one process type selected from a group of types consisting of a user application, a Virtual Machine (VM), and operating-system process and a container.
In some embodiments, the host and the network adapter are coupled to communicate with one another over a peripheral bus, and the host is coupled to obtain the network time from the network adapter using a peripheral-bus device that is queried for the network time by the time-service consumer processes running in the host. In an embodiment, the peripheral-bus device is a read-only bus device that transfers information only unidirectionally from the peripheral bus to the host.
There is also provided, in accordance with an embodiment of the present invention, a method in a network adapter. The method includes receiving in the network adapter, from a communication network, one or more time-protocol packets that convey a network time used for synchronizing network devices in the communication network. A hardware clock in the network adapter is aligned to the network time conveyed in the time-protocol packets. The network time is made available to one or more time-service consumers running in a host served by the network adapter.
There is further provided, in accordance with an embodiment of the present invention, a method is a compute node. The method includes running one or more time-service consumer processes in one or more processors of a host. In a network adapter that serves the host, one or more time-protocol packets, which convey a network time used for synchronizing network devices in a communication network, are received from the communication network. A hardware clock in the network adapter is aligned to the network time conveyed in the time-protocol packets. The network time is made available to the time-service consumer processes running in the host.
The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:
Embodiments of the present invention that are described herein provide improved network adapters that, in addition to connecting their hosts to a network, provide the hosts with an accurate time service.
In the present context, the term “time service” refers to a service in which a network adapter provides a host with the current network time. The term “network time” refers to a common time base that is used for synchronizing network devices (e.g., compute nodes, network adapters, network switches and the like) in the network to one another. The network devices typically synchronize to the network time by exchanging time-protocol packets of a suitable time-synchronization protocol, such as PTP.
In some embodiments, the network adapter communicates with the host over a peripheral bus, e.g., a Peripheral Component Interconnect express (PCIe) bus. The host comprises one or more processors that may run various “time-service consumers,” e.g., user applications, Virtual Machines (VMs), operating system processes, or any other suitable software that uses the network time. At least some of the time-service consumers may require a highly accurate indication of the network time, to carry out various time-sensitive applications. In order to provide the time service to the time-service consumers running in the host, the network adapter comprises the following:
When using the disclosed technique, the host is typically completely isolated from the control plane of the time-synchronization protocol. Time-protocol packets arriving from the network are handled by the software stack of the time-synchronization protocol that runs in the network adapter, and typically do not reach the host. The interaction of the host with the time service is typically limited to querying the dedicated device for the current network time. The time-service consumers are not required to support the time-synchronization protocol (e.g., PTP). The time-service consumers need not even be aware of (i.e., have information as to) the fact that such a protocol is being used.
The isolated time service solution described herein provides considerable advantages. For example, implementation of the time service is agnostic to the hardware and software configuration of the host, the host operating system and the software the host may run. As such, configuration changes in the host, e.g., hardware or software upgrades, have no effect on the time service. The accuracy of the time service remains constant, regardless of the load conditions and the hardware and software configuration or topology of the host.
In addition, since the time service is implemented entirely within the network adapter, the software stack of the time-synchronization protocol can be designed and optimized for the specific network adapter hardware. The network adapter can be provided “out of the box” with pre-configured time-synchronization protocol hardware and software. For example, the software stack can be optimized for the precision of the specific hardware clock used in the network adapter. In an example embodiment, the software stack may avoid aligning the hardware clock with a resolution that is finer than the hardware clock's precision or timestamping resolution (e.g., 400 nSec). Offloading the time-synchronization protocol stack to the network adapter also reduces the computational load of the host.
Moreover, isolating the host from the time service provides a high degree of security. Since the host is exposed only to a dedicated device having specific functionality over the peripheral bus, the “surface of attack” that might be exploited by malware is very small. Vulnerability to attack is further reduced by the fact that the exposed device is typically a read-only device, meaning that the flow of time-related information is unidirectional from the network adapter to the host.
The disclosed solution is useful in a wide variety of applications and use-cases. One typical example is a server in a “bare metal cloud” data center, in which a client (“tenant”) is provided with full access to the host. Another typical example is a server in a multi-tenant cloud-based data center (sometimes referred to as “virtualization cloud”). In both use-cases, the service provider has little or no control over the end-user's instance (which may comprise the entire host, a VM, a container, or any other suitable entity). A time service that is isolated and self-contained within the network adapter is highly effective in such environments.
When using the disclosed technique in a cloud architecture, the time service can be provided to a tenant in a transparent manner that is easy to integrate with the tenant's system, regardless of the operating system and software used by the tenant. At the same time, the entire time-synchronization protocol infrastructure and configuration remain hidden from the tenant, thereby isolating and protecting the cloud provider network from possible attacks originating from the tenant's software. The cloud provider may also serve multiple isolated tenants (in a bare-metal configuration or otherwise) using a single time-synchronization protocol infrastructure.
Provisioning of Accurate Time Services by Network Adapter
Network adapter 32 may comprise any suitable type of network adapter. In the present example, network adapter 32 is a high-functionality device that comprises one or more integral CPUs 40 and processing hardware 44. Such a network adapter is also referred to as a Data Processing Unit (DPU) or “Smart Network Interface Controller (Smart-NIC)”. In alternative embodiments, network adapter 32 may comprise, for example, an Ethernet NIC, an Infiniband™ Host Channel Adapter (HCA), a network-enabled Graphics Processing Unit (GPU), a Field-Programmable Gate Array (FPGA), or any other suitable type of device capable of network communication. The description that follows refers to a DPU, by way of example.
Host 24 and DPU 32 communicate with one another over a peripheral bus 36. In the present example bus 36 comprises a Peripheral Component Interconnect express (PCIe) bus. Alternatively, any other suitable type of peripheral bus can be used.
DPU 32 connects host 24 to a packet network 28. Depending on the applicable use-case, network 28 may comprise, for example, a Local-Area Network (LAN) of a data center, a Wide-Area Network (WAN) such as the Internet, or any other suitable type of network or combination of networks. Communication over network 28 may be performed using any suitable communication protocol, e.g., Ethernet or Infiniband. DPU 32 comprises one or more network ports 52 (a single port in the example of
In some embodiments, DPU 32 is synchronized to the network time of network 28. Typically, the network devices of network 28 synchronize to the network time by exchanging time-protocol packets of a suitable time-synchronization protocol. In the present example the time-synchronization protocol is PTP, and the time-protocol packets are referred to as PTP packets. Alternatively, the time-synchronization protocol may comprise, for example, Network Time Protocol (NTP) or Synchronous Ethernet (SyncE). (Aspects of synchronization using SyncE are addressed, for example, in U.S. patent application Ser. No. 16/920,772, filed Jul. 6, 2020, which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference.) The description that follows refers to PTP, by way of example.
In the embodiment of
When the operating system of CPUs 40 is Linux, PTP daemon 60 may comprise, for example, the PTP4L software. Alternatively, PTP daemon 60 may comprise any other suitable software that carries out the protocol stack of the time-synchronization protocol being used. In one example embodiment, CPUs 40 run a Linux operating system, and the PTP4L software runs in a “hardware aware” mode for high stability and accuracy. In an embodiment, CPUs 40 also maintain a PTP queue 64 for queuing incoming PTP packets that are pending for processing by PTP daemon 60.
In some embodiments, DPU 32 provides an isolated, self-contained time service to host 24 using a dedicated PTP device 68 that is exposed on PCIe bus 36. The present example refers to a single PTP device 68. In alternative embodiments, multiple PTP devices 68 may be exposed on the PCIe bus.
Various time-service consumers 48 that run on host 24 (e.g., user applications, virtual machines or operating system processes) can request and receive the current network time from PTP device 68. Time-service consumers 48 may interact with device 68 directly, or indirectly, e.g., via a suitable Application Programming Interface (API). Host 24 comprises one or more processors (e.g., CPUs—not shown in the figure). Time-service consumers 48 may run on any of the processors in the host. For simplicity, software that runs on any of the processors of host 24 are sometimes referred to as “running in host 24”.
As part of the “isolation” provided by the disclosed techniques, in some embodiments the PTP packets that arrive from network 28 typically remain confined within DPU 32 and do not reach host 24. By contrast, non-PTP packets arriving from the network are delivered to host 24 and are queued in one or more non-PTP queues 72 (as opposed to PTP queue 64 that is part of DPU 32).
The configurations of network adapter (e.g., DPU) 32 and host 24 shown in
In various embodiments, the synchronized hardware PTP clock function may be exposed to time-service consumers 48 on host 24 in various ways. In some embodiments, host 24 operating systems may run native software drivers to identify the synchronized PTP function over the network adapter's PCIe bus 36, and expose the synchronized PTP function as a system PTP device 68. In other embodiments, some host operating systems may require suitable middleware, which would provide an API for the time-service consumers to interact with the synchronized PTP function over the PCIe bus.
The embodiment of
The various elements of network adapter (e.g., DPU) 32 and host 24 may be implemented using suitable hardware, such as in one or more Application-Specific Integrated Circuits (ASIC) or Field-Programmable Gate Arrays (FPGA), using software, or using a combination of hardware and software elements. In some embodiments, any of CPUs 40 of network adapter 32 and/or any CPU of host 24 may comprise a general-purpose processor, which is programmed in software to carry out the functions described herein. The software may be downloaded to the processor in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.
At a protocol stack running step 80, CPUs 40 of DPU 32 run PTP daemon 60, which implements the PTP software stack. At a packet reception step 84, DPU 32 receives packets from network 28 via port 52. At a checking step 88, hardware 44 checks classifies the received packets into PTP packets and non-PTP packets. At a non-PTP forwarding step 92, hardware 44 of DPU 32 sends any non-PTP packet via PCIe bus 36 to queue(s) 72 of host 24. At a PTP forwarding step 96, hardware 44 sends any PTP packet to PTP queue 64, for processing by PTP daemon 60. The PTP packets are typically timestamped by PHC 56 with the current network time, before being forwarded to queue 64.
At an alignment step 100, PTP daemon 60 aligns PHC 56 based on the timestamped PTP packets. PTP daemon 60 typically calculates, based on the received PTP packets, corrections that align PHC 56 to the network time, and applies the calculated corrections to the PHC. The method loops back to step 80 above.
In parallel with steps 84-100, PTP daemon 60 exposes dedicated PTP device 68 to host 24 over PCIe bus 36, at a device exposing step 104. At a time-service provisioning step 108, any time-service consumers running in host 24 may query device 68 and thus obtain the current network time. The method loops back to step 80 above.
In some embodiments, the alignment of PHC 56 (at step 100) comprises both a frequency alignment and a time-offset alignment. With such an alignment of PHC 56, PTP device 68 can expose an offset-free value of the current network time to time-service consumers 48, obviating the need for any translations or conversions in host 24.
In various embodiments, time-service consumers 48 may use the network time information in any suitable way and for any suitable purpose. The host may use time-sensitive semantics and operations, such as, for example, ‘send packet X at 8:00 am’ or ‘forward packet Y to queue Z based on the arrival time of the packet’. Techniques of this sort are addressed, for example, in U.S. Patent Application Publication 2020/0162234, cited above, and in U.S. patent application Ser. No. 16/782,075, filed Feb. 2, 2020, which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference.
In an embodiment, host 24 may use the time service to synchronize its own clock to PHC 56 of DPU 32 (and thus to the network time) without having to implement any of the PTP stack. Any time-service consumer 48 (e.g., a VM) may similarly synchronize its clock to the network time. Multiple different software entities in host 24 can be synchronized in this manner. Aspects of such synchronization are also addresses in U.S. Patent Application Publication 2020/0162234, cited above, whose disclosure is incorporated herein by reference. Linux methods and services, such as PHC2SYS, can use the time service to synchronize to the network time with little or no software overhead.
Although the embodiments described herein mainly address various general cloud computing architectures, the methods and systems described herein can also be used in other applications, such as in industry-specific cloud architectures (such as cloud architectures oriented at telecom, media & entertainment, finance cloud and the like). The architecture in question may be implemented in the core or at the edge of the network. Further alternatively, the disclosed techniques can be used in standard bare-metal architectures having no cloud/virtualization, which may be used for various industry-oriented applications. In one example, an enterprise customer may use the disclosed technique to run time-sensitive applications on a bare-metal operating system that does not support any time-protocol software stack.
It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.
Number | Name | Date | Kind |
---|---|---|---|
5392421 | Lennartsson | Feb 1995 | A |
5402394 | Turski | Mar 1995 | A |
5416808 | Witsaman et al. | May 1995 | A |
5491792 | Grisham et al. | Feb 1996 | A |
5564285 | Jurewicz et al. | Oct 1996 | A |
5592486 | Lo et al. | Jan 1997 | A |
5896524 | Halstead, Jr. et al. | Apr 1999 | A |
6055246 | Jones | Apr 2000 | A |
6084856 | Simmons et al. | Jul 2000 | A |
6144714 | Bleiweiss et al. | Nov 2000 | A |
6199169 | Voth | Mar 2001 | B1 |
6289023 | Dowling et al. | Sep 2001 | B1 |
6449291 | Burns et al. | Sep 2002 | B1 |
6535926 | Esker | Mar 2003 | B1 |
6556636 | Takagi | Apr 2003 | B1 |
6556638 | Blackburn | Apr 2003 | B1 |
6718476 | Shima | Apr 2004 | B1 |
6918049 | Lamb et al. | Jul 2005 | B2 |
7111184 | Thomas, Jr. et al. | Sep 2006 | B2 |
7191354 | Purho | Mar 2007 | B2 |
7245627 | Goldenberg et al. | Jul 2007 | B2 |
7254646 | Aguilera et al. | Aug 2007 | B2 |
7334124 | Pham et al. | Feb 2008 | B2 |
7412475 | Govindarajalu | Aug 2008 | B1 |
7440474 | Goldman et al. | Oct 2008 | B1 |
7447975 | Riley | Nov 2008 | B2 |
7483448 | Bhandari et al. | Jan 2009 | B2 |
7496686 | Coyle | Feb 2009 | B2 |
7535933 | Zerbe et al. | May 2009 | B2 |
7623552 | Jordan et al. | Nov 2009 | B2 |
7636767 | Lev-Ran et al. | Dec 2009 | B2 |
7650158 | Indirabhai | Jan 2010 | B2 |
7656751 | Rischar et al. | Feb 2010 | B2 |
7750685 | Bunch et al. | Jul 2010 | B1 |
7904713 | Zajkowski et al. | Mar 2011 | B1 |
7941684 | Serebrin et al. | May 2011 | B2 |
8065052 | Fredriksson et al. | Nov 2011 | B2 |
8300749 | Hadzic et al. | Oct 2012 | B2 |
8341454 | Kondapalli | Dec 2012 | B1 |
8370675 | Kagan | Feb 2013 | B2 |
8407478 | Kagan et al. | Mar 2013 | B2 |
8607086 | Cullimore | Dec 2013 | B2 |
8699406 | Charles et al. | Apr 2014 | B1 |
8879552 | Zheng | Nov 2014 | B2 |
8930647 | Smith | Jan 2015 | B1 |
9344265 | Karnes | May 2016 | B2 |
9397960 | Arad et al. | Jul 2016 | B2 |
9549234 | Mascitto | Jan 2017 | B1 |
9942025 | Bosch et al. | Apr 2018 | B2 |
9979998 | Pogue et al. | May 2018 | B1 |
10014937 | Di Mola et al. | Jul 2018 | B1 |
10027601 | Narkis et al. | Jul 2018 | B2 |
10054977 | Mikhaylov et al. | Aug 2018 | B2 |
10164759 | Volpe | Dec 2018 | B1 |
10320646 | Mirsky et al. | Jun 2019 | B2 |
10637776 | Iwasaki | Apr 2020 | B2 |
10727966 | Izenberg et al. | Jul 2020 | B1 |
10908635 | Ranganathan et al. | Feb 2021 | B1 |
11379334 | Srinivasan et al. | Jul 2022 | B1 |
11388263 | Levi et al. | Jul 2022 | B2 |
20010006500 | Nakajima et al. | Jul 2001 | A1 |
20020027886 | Fischer et al. | Mar 2002 | A1 |
20020031199 | Rolston et al. | Mar 2002 | A1 |
20040096013 | Laturell et al. | May 2004 | A1 |
20040153907 | Gibart | Aug 2004 | A1 |
20050033947 | Morris et al. | Feb 2005 | A1 |
20050268183 | Barmettler | Dec 2005 | A1 |
20060109376 | Chaffee et al. | May 2006 | A1 |
20060171496 | Nakamuta et al. | Aug 2006 | A1 |
20070008044 | Shimamoto | Jan 2007 | A1 |
20070072451 | Tazawa et al. | Mar 2007 | A1 |
20070104098 | Kimura et al. | May 2007 | A1 |
20070124415 | Lev-Ran et al. | May 2007 | A1 |
20070139085 | Elliot et al. | Jun 2007 | A1 |
20070159924 | Vook et al. | Jul 2007 | A1 |
20070266119 | Ohly | Nov 2007 | A1 |
20080069150 | Badt et al. | Mar 2008 | A1 |
20080225841 | Conway et al. | Sep 2008 | A1 |
20080285597 | Downey et al. | Nov 2008 | A1 |
20090167443 | Liu et al. | Jul 2009 | A1 |
20090257458 | Cui et al. | Oct 2009 | A1 |
20100280858 | Bugenhagen | Nov 2010 | A1 |
20110110360 | Fenwick et al. | May 2011 | A1 |
20110182191 | Jackson | Jul 2011 | A1 |
20110194425 | Li et al. | Aug 2011 | A1 |
20120063556 | Hoang | Mar 2012 | A1 |
20120076319 | Terwal | Mar 2012 | A1 |
20120301134 | Davari et al. | Nov 2012 | A1 |
20130039359 | Bedrosian | Feb 2013 | A1 |
20130045014 | Mottahedin et al. | Feb 2013 | A1 |
20130215889 | Zheng et al. | Aug 2013 | A1 |
20130235889 | Aweya et al. | Sep 2013 | A1 |
20130294144 | Wang et al. | Nov 2013 | A1 |
20130315265 | Webb, III et al. | Nov 2013 | A1 |
20130336435 | Akkihal et al. | Dec 2013 | A1 |
20140085141 | Geva et al. | Mar 2014 | A1 |
20140153680 | Garg et al. | Jun 2014 | A1 |
20140185216 | Zeng et al. | Jul 2014 | A1 |
20140185632 | Steiner et al. | Jul 2014 | A1 |
20140253387 | Gunn et al. | Sep 2014 | A1 |
20140281036 | Cutler et al. | Sep 2014 | A1 |
20140301221 | Nadeau et al. | Oct 2014 | A1 |
20140321285 | Chew et al. | Oct 2014 | A1 |
20150019839 | Cardinelli et al. | Jan 2015 | A1 |
20150078405 | Roberts | Mar 2015 | A1 |
20150092793 | Aweya | Apr 2015 | A1 |
20150127978 | Cui et al. | May 2015 | A1 |
20150163050 | Han et al. | Jun 2015 | A1 |
20150318941 | Zheng et al. | Nov 2015 | A1 |
20160057518 | Neudorf | Feb 2016 | A1 |
20160072602 | Earl et al. | Mar 2016 | A1 |
20160110211 | Kames | Apr 2016 | A1 |
20160112182 | Karnes | Apr 2016 | A1 |
20160140066 | Worrell et al. | May 2016 | A1 |
20160277138 | Garg et al. | Sep 2016 | A1 |
20160285574 | White et al. | Sep 2016 | A1 |
20160315756 | Tenea et al. | Oct 2016 | A1 |
20170005903 | Mirsky | Jan 2017 | A1 |
20170017604 | Chen et al. | Jan 2017 | A1 |
20170126589 | Estabrooks et al. | May 2017 | A1 |
20170160933 | De Jong et al. | Jun 2017 | A1 |
20170214516 | Rivaud et al. | Jul 2017 | A1 |
20170302392 | Farra et al. | Oct 2017 | A1 |
20170331926 | Raveh et al. | Nov 2017 | A1 |
20170359137 | Butterworth et al. | Dec 2017 | A1 |
20180059167 | Sharf et al. | Mar 2018 | A1 |
20180152286 | Kemparaj et al. | May 2018 | A1 |
20180188698 | Dionne et al. | Jul 2018 | A1 |
20180191802 | Yang et al. | Jul 2018 | A1 |
20180227067 | Hu et al. | Aug 2018 | A1 |
20180309654 | Achkir et al. | Oct 2018 | A1 |
20190007189 | Hossain et al. | Jan 2019 | A1 |
20190014526 | Bader et al. | Jan 2019 | A1 |
20190089615 | Branscomb et al. | Mar 2019 | A1 |
20190149258 | Araki et al. | May 2019 | A1 |
20190158909 | Kulkarni et al. | May 2019 | A1 |
20190196563 | Lai | Jun 2019 | A1 |
20190220300 | Rosenboom | Jul 2019 | A1 |
20190265997 | Merrill | Aug 2019 | A1 |
20190273571 | Bordogna et al. | Sep 2019 | A1 |
20190319729 | Leong et al. | Oct 2019 | A1 |
20190349392 | Wetterwald et al. | Nov 2019 | A1 |
20190379714 | Levi et al. | Dec 2019 | A1 |
20200044657 | Pi et al. | Feb 2020 | A1 |
20200162234 | Almog et al. | May 2020 | A1 |
20200169379 | Gaist et al. | May 2020 | A1 |
20200235905 | Su et al. | Jul 2020 | A1 |
20200304224 | Neugeboren | Sep 2020 | A1 |
20200331480 | Zhang et al. | Oct 2020 | A1 |
20200344333 | Hawari et al. | Oct 2020 | A1 |
20200396050 | Perras et al. | Dec 2020 | A1 |
20200401434 | Thampi | Dec 2020 | A1 |
20210006344 | Chen et al. | Jan 2021 | A1 |
20210141413 | Levi et al. | May 2021 | A1 |
20210175991 | Neugeboren | Jun 2021 | A1 |
20210218431 | Narayanan et al. | Jul 2021 | A1 |
20210297230 | Dror et al. | Sep 2021 | A1 |
20210318978 | Hsung | Oct 2021 | A1 |
20210409031 | Ranganathan et al. | Dec 2021 | A1 |
20220066978 | Mishra et al. | Mar 2022 | A1 |
20220239549 | Zhao et al. | Jul 2022 | A1 |
20220261032 | Bateni | Aug 2022 | A1 |
20220342086 | Yoshida | Oct 2022 | A1 |
20230076889 | Rabinovich et al. | Mar 2023 | A1 |
Number | Date | Country |
---|---|---|
101059825 | Oct 2007 | CN |
101388741 | Mar 2009 | CN |
106817183 | Jun 2017 | CN |
108667547 | Oct 2018 | CN |
108829493 | Nov 2018 | CN |
1215559 | Sep 2007 | EP |
2770678 | Aug 2014 | EP |
2011091676 | May 2011 | JP |
2012007276 | Jan 2012 | WO |
2013124782 | Aug 2013 | WO |
2013143112 | Oct 2013 | WO |
2014029533 | Feb 2014 | WO |
2014138936 | Sep 2014 | WO |
Entry |
---|
U.S. Appl. No. 17/549,949 Office Action dated Mar. 30, 2023. |
Corbett et al., “Spanner: Google's Globally Distributed Database,” ACM Transactions on Computer Systems, vol. 31, No. 3, article 8, pp. 1-22, Aug. 2013. |
U.S. Appl. No. 17/191,736 Office Action dated Jun. 26, 2023. |
IPclock, “IEEE 1588 Primer,” ip-clock.com, pp. 1-3, May 1, 2017 (downloaded from https://web.archive.org/web/20170501192647/http://ip-clock.com/ieee-1588-primer/). |
U.S. Appl. No. 16/900,931 Office Action dated Apr. 28, 2022. |
U.S. Appl. No. 16/683,309 Office Action dated Mar. 17, 2022. |
U.S. Appl. No. 16/779,611 Office Action dated Mar. 17, 2022. |
U.S. Appl. No. 17/120,313 Office Action dated Mar. 28, 2022. |
U.S. Appl. No. 17/191,736 Office Action dated Apr. 26, 2022. |
EP Application # 21214269 Search Report dated May 2, 2022. |
U.S. Appl. No. 17/148,605 Office Action dated May 17, 2022. |
U.S. Appl. No. 17/191,736 Advisory Action dated Feb. 16, 2023. |
“IEEE Standard for Local and Metropolitan Area Networks—Timing and Synchronization for Time-Sensitive Applications,” IEEE Std 802.1AS-2020, IEEE Computer Society, pp. 1-421, year 2020. |
EP Application # 22151451.6 Search Report dated Jun. 17, 2022. |
U.S. Appl. No. 16/779,611 Office Action dated Jun. 24, 2022. |
“Precision Time Protocol,” PTP Clock Types, CISCO, pp. 1-52, Jul. 30, 2020, as downloaded from https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/5x/system-management-configuration/cisco-apic-system-management-configuration-guide-52x/m-precision-time-protocol.pdf. |
U.S. Appl. No. 17/120,313 Office Action dated Aug. 29, 2022. |
ITU-T Standard G.8262/Y.1362, “Timing characteristics of synchronous equipment slave clock”, pp. 1-44, Nov. 2018. |
ITU-T Standard G.8264/Y.1364, “Distribution of timing information through packet networks”, pp. 1-42, Aug. 2017. |
ITU-T Standard G.8261/Y.1361, “Timing and synchronization aspects in packet networks”, pp. 1-120, Aug. 2019. |
U.S. Appl. No. 17/871,937 Office Action dated Aug. 1, 2023. |
U.S. Appl. No. 17/578,115 Office Action dated Apr. 26, 2023. |
U.S. Appl. No. 17/534,776 Office Action dated Jun. 29, 2023. |
SiTime Corporation, “Sit5377-60 to 220 MHZ, ±100 ppb Elite RF™ Super-TCXO,” Product Description, pp. 1-3, last updated Mar. 18, 2023 as downloaded from https://web.archive.org/web/20230318094421/https://www.sitime.com/products/super-tcxos/sit5377. |
PCI-SIG, “PCI Express®—Base Specification—Revision 3.0,” pp. 1-860, Nov. 10, 2010. |
U.S. Appl. No. 17/579,630 Office Action dated Oct. 24, 2022. |
U.S. Appl. No. 17/579,630 Office Action dated Jan. 12, 2023. |
U.S. Appl. No. 17/670,540 Office Action dated Jan. 18, 2023. |
U.S. Appl. No. 17/191,736 Office Action dated Nov. 10, 2022. |
U.S. Appl. No. 16/683,309 Office Action dated Sep. 17, 2021. |
U.S. Appl. No. 16/920,722 Office Action dated Aug. 12, 2021. |
IEEE Standard 1588™—2008: “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, IEEE Instrumentation and Measurement Society, Revision of IEEE Standard 1588-2002, USA, pp. 1-289, Jul. 24, 2008. |
Weibel et al., “Implementation and Performance of Time Stamping Techniques”, 2004 Conference on IEEE 1588, pp. 1-29, Sep. 28, 2004. |
Working Draft Project American National Standard T10/1799-D, “Information Technology—SCSI Block Commands—3 (SBC-3)”, pp. 1-220, Revision 19, May 29, 2009. |
“Infiniband Architecture: Specification vol. 1”, pp. 1-1727, Release 1.2.1, Infiniband Trade Association, Nov. 2007. |
Mellanox Technologies, “Mellanox ConnectX IB: Dual-Port InfiniBand Adapter Cards with PCI Express 2.0”, pp. 1-2, USA, year 2008. |
Wikipedia—“Precision Time Protocol”, pp. 1-8, Aug. 24, 2019. |
Levi et al., U.S. Appl. No. 16/782,075, filed Feb. 5, 2020. |
Weibel, H., “High Precision Clock Synchronization according to IEEE 1588 Implementation and Performance Issues”, Zurich University of Applied Sciences, pp. 1-9, Jan. 17, 2005. |
Lu et al., “A Fast CRC Update Implementation”, Computer Engineering Laboratory, Electrical Engineering Department, pp. 113-120, Oct. 8, 2003. |
Levi et al., U.S. Appl. No. 16/799,873, filed Feb. 25, 2020. |
Dlugy-Hegwer et al., “Designing and Testing IEEE 1588 Timing Networks”, Symmetricom, pp. 1-10, Jan. 2007. |
Mellanox Technologies, “How to test 1PPS on Mellanox Adapters”, pp. 1-6, Oct. 22, 2019 downloaded from https://community.mellanox.com/s/article/How-To-Test-1PPS-on-Mellanox-Adapters. |
ITU-T recommendation, “G.8273.2/Y.1368.2—Timing characteristics of telecom boundary clocks and telecom time slave clocks”, pp. 1-50, Jan. 2017. |
Texas Instruments, “LMK05318 Ultra-Low Jitter Network Synchronizer Clock With Two Frequency Domains,” Product Folder, pp. 1-86, Dec. 2018. |
Ravid et al., U.S. Appl. No. 16/920,772, filed Jul. 6, 2020. |
Sela et al., U.S. Appl. No. 16/900,931, filed Jun. 14, 2020. |
Levi et al., U.S. Appl. No. 17/120,313, filed Dec. 14, 2020. |
Mula et al., U.S. Appl. No. 17/148,605, filed Jan. 14, 2021. |
IEEE Std 1588-2002, “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”, IEEE Instrumentation and Measurement Society, pp. 1-154, Nov. 8, 2002. |
Sattinger et al., U.S. Appl. No. 17/191,736, filed Mar. 4, 2021. |
U.S. Appl. No. 18/111,916 Office Action dated Jun. 5, 2024. |
CN Application # 202210456108.6 Office Action dated Aug. 30, 2024. |
Number | Date | Country | |
---|---|---|---|
20220357763 A1 | Nov 2022 | US |