The present disclosure deals with processes and apparatus for synchronizing clock devices in distributed networks, such as those found among the nodes (substations) in power distribution systems. Power distribution systems, also known as electrical power grids, are used to transmit power from power generators to consumers. Over time, power distribution systems have become increasingly complex and more difficult to govern, resulting in increased monitoring needs. Power system monitors and disturbance monitoring equipment (DME) rely upon the correct correlation of data to time (i.e., accurate timestamping) to perform their respective functions.
A common source of timestamping in electric distribution systems is a Global Positioning System (GPS)-based substation clock (including one or more oscillators) equipped with GPS receivers to provide accurate timestamping, and GPS substation clocks have become foundational to disturbance monitoring equipment. These systems are susceptible to a variety of threats that can disrupt accurate timestamping, from equipment or component failures, network outages, GPS loss-of-lock, multipath interference, or natural interference (e.g., solar flares), to active cyber measures, such as denial-of-service, GPS jamming, GPS spoofing, or interruptions to spectrum management. Many times, either a single GPS antenna, or a single GPS substation clock is used to provide timestamping to multiple devices at a substation. Disruption to the timestamping function of any given substation clock can disrupt normal operations of the grid and compound the complexity of event and/or failure analysis.
For example, a distribution system may employ several generators, each of which can be adjusted independently. This can lead to phase differences between a given generator and the wider distribution system, whereby the generator will attempt to correct itself such that it is synchronized (in phase) with the rest of the distribution system. The power used to synchronize the generator may flow from the wider system and therefore stress and/or overload the individual generator in some situations. Likewise, other parts of the network that are subject to multiple generator loads which are significantly out-of-phase may be damaged. Therefore, it is important to synchronize each generator with the rest of the power distribution system. This is generally done through enhanced phase measurement units (PMUs), sometimes known as synchrophasors, which reports the amplitude, frequency, and phase information of electricity flowing through the system at a particular location to a remote site for analysis. However, the monitoring and analysis of the system information requires reporting data with accurate and reconcilable timestamps, such that the various measurement devices have synchronized clock information. An erroneous clock offset in a substation introduces error into the phase calculations used to manage the entire grid, which may cascade from localized equipment failures into broader blackouts. Instability of power grids from fluctuating supply and demand is compounded by such events and the corresponding recovery and synchronization actions.
One protocol that has been developed for use in applications requiring precise time synchronization is the Precision Time Protocol (PTP), as defined in IEEE 1588-2008, which is incorporated herein by reference. In the PTP, each node determines whether it is master or a slave clock through monitoring messages known as “Announce Messages” which contain information on each clock's management priority and time source(s). In essence, the classical BMCA operates as shown in
One problem with the PTP standard is the potential critical point of failure of a master clock. For example, if the first clock has a loss-of-lock, the second clock will be elected as MC. When the first clock re-obtains a lock to the GPS signal, the first clock will be re-elected as the MC. The process of electing and selecting a new MC requires a certain amount of time during which the various slave clocks are not synchronized. During this time, the slave clocks are running disparately and their relative drift will prevent the correct correlation of data to time needed to establish the baseline against which precise monitoring, control, and event correlation can be applied. A further problem is the reliance on one master clock that is not validated: the slave/client nodes have no way to check the supposed accuracy of the self-proclaimed grand master clock. A compromised clock may therefore be able to masquerade as the grand master and introduce timestamping errors throughout the network.
Thus, in order to provide accurate monitoring, analysis, and control of the entire grid, there is a need for improved, robust methods and systems for time synchronization among the many clocks in the distribution network.
The present disclosure relates to methods and systems for synchronizing multiple clocks in a network such as a power distribution system. The method comprises assigning clock devices to groups, selecting a best clock from each group, electing a grand master clock from the various group best clocks, and synchronizing the network clocks to the grand master clock. The selection of best clocks within each group and the selection of the grand master clock from among the group best clocks may be performed using the same, similar, or different processes.
The disclosure also relates to methods of selecting a grand master clock from among multiple clock devices. Preferably, the clock devices are in a communication network. The method comprises receiving clock data from the clock devices, grouping the clock devices into groups, selecting group best clock for each group using a first selection process, and selecting a grand master clock from the group best clocks using a second selection process. The method may further comprise sending the clock data of the grand master clock to the other clock devices and/or synchronizing the clock devices to the grand master clock. In some embodiments, the first selection process and the second selection process are different. The methods may be but are not necessarily implemented in one processing device. In some embodiments, multiple processing devices may be used. For example, one device in a given device group may select that group's group best clock and report the corresponding clock data to another device for a further selection process. Further options include using redundant systems to execute the grand master clock selection processes, with comparison and/or tie-breaking criteria implemented in the event of disagreement among the results from the redundant systems.
The disclosure also relates to implementation of the above methods in power distribution systems and/or systems capable of or configured to perform such methods.
As illustrated in
In step 14, a group election logic is selected for each clock group. The group election logic may be, but is not necessarily, the same for each clock group. The selection may be based various factors, including but not limited to a preset sequence of algorithms, the time and/or computing resources available, the size of the group, known communication delays in the group, prior selection/election as the master clock group and/or grand master clock (discussed further below), and/or combinations thereof. In some embodiments, multiple election logics are chosen and the results of those processes may be compared and evaluated as part of the application in step 15.
In step 15, K group best clocks are selected from among the devices in each group. In one embodiment, step 15 employs an N-input voting algorithm (NIVA), such as the algorithm described in A. Karimi, F. Zarafshan, A. Ramli, “A Novel N-Input Voting Algorithm for X-by-Wire Fault Tolerant Systems,” Scientific World Journal (2014), which is incorporated herein by reference. Alternatively, step 15 may employ the Best Master Clock Algorithm (BMCA) as described in IEEE 1588-2008 and above in connection with
In another embodiment of step 15, method 15b seen in
However, returning to
A grand master clock is elected from the group best clocks in election process 16. In election process 16, the first step 17 is selection of a master election logic. In election step 18, a preliminary grand master is selected from the group best clocks in accordance with the selected election logic. In step 19, the reasonableness of the preliminary grand master is evaluated. If the evaluation in step 19 is acceptable, the method 10 continues to optional authentication step 20, formal grand master designation step 21, and network synchronization step 22, as discussed further below. If the evaluation in step 19 indicates that the preliminary grand master is unacceptable, the process 10 may return to selecting the election logic at 17, or to an earlier step such as reshuffling the groupings in step 13, or even return to start step 11 and recollect new clock data from the P clock devices or a subset thereof. The grand master election of steps 17-19 can therefore be iterated until an acceptable preliminary grand master is elected. A particular implementation of grand master election is discussed further in connection with
At authentication step 20, an integrity protection mechanism utilizes a Message Authentication Code (MAC) and symmetric encryption to verify that messages (including time synchronization packets) have not suffered unauthorized modification in transit, similar to and as suggested by Annex K of IEEE 1588-2002. If the authentication between the preliminary grand master and the selector device (and/or a slave clock) fails, the slave clock(s) will not accept the new master clock information. At that point, the authentication step 20 can be attempted again, or the method 10 can return to some prior step, even as far back as start step 11. If the authentication step 20 is successful, the preliminary grand master is formally designated as the grand master clock in designating step 21. Then, in step 22, the clock devices of the network synchronize to the grand master clock, for example using conventional techniques such as those described in PTP and IEEE 1588-2008. Optionally and preferably, another integrity protection mechanism and/or security protocol is utilized prior to the syndication of the grand master clock information to the rest of the network in step 22.
Synchronization methods 10 may be implemented in one or more devices. For example, the clock groups could select their group best clocks independently and report only the group best clock to a central time controller device for further election of the grand master clock in process 16. Such an implementation may reduce network resources required to transmit the full set of various clock data to a central controller. Alternatively, a central controller can receive all of the clock data and perform the intermediate steps before engaging in any communications with the separate networked devices in steps 20 and/or 22.
Further variations and extensions of method 10 are within the scope of this disclosure, such as a further nesting of selection criteria and/or subgrouping of clocks, as shown in
In a specific embodiment 60 as seen in
In some cases, the grand master clock will also be the clock which produced the central value 67. However, this is not necessarily true. For example, similar to the discussion above in connection with
The above methods may be implemented in the specific context of power distribution systems. In any particular power system substation, the actual implementation of the GPS timestamping can take a variety of forms. As one example,
In
The implementations described in
Further descriptions relating to certain embodiments may be found in the inventor's work, Chan S. (2020), A Potential Cascading Succession of Cyber Electromagnetic Achilles' Heels in the Power Grid (in: Arai K., Bhatia R. (eds), FICC 2019: Advances in Information and Communication, Lecture Notes in Networks and Systems, vol 70. Springer), which is incorporated herein by reference. Although the primary examples described herein relate to clock synchronization in power distribution systems, it is understood that these inventive, robust time-synchronization methods may be applied to other applications with distributed device networks. For example, precise time synchronization may improve alignment of the access and activity logs in a computer network, thereby providing valuable advantages in cybersecurity intrusion detection, monitoring, or analysis.
This application is a continuation of U.S. patent application Ser. No. 18/304,404, filed Apr. 21, 2023, entitled “Multi-Clock Synchronization in Power Grids,” which is a divisional of U.S. patent application Ser. No. 17/189,826, filed Mar. 2, 2021, entitled “Multi-Clock Synchronization in Power Grids,” now U.S. Pat. No. 11,650,620, which application is a continuation of International Patent Application No. PCT/US2019/033455, filed May 22, 2019, entitled “Multi-Clock Synchronization in Power Grids,” all of which are hereby incorporated by reference.
Number | Name | Date | Kind |
---|---|---|---|
5377206 | Smith | Dec 1994 | A |
5422915 | Byers et al. | Jun 1995 | A |
6041066 | Meki | Mar 2000 | A |
6631483 | Parrish | Oct 2003 | B1 |
6665316 | Eidson | Dec 2003 | B1 |
6675307 | Heitkamp | Jan 2004 | B1 |
7111195 | Berkcan et al. | Sep 2006 | B2 |
7535931 | Zampetti et al. | May 2009 | B1 |
7730230 | Kondapalli | Jun 2010 | B1 |
8108558 | Kirsch et al. | Jan 2012 | B2 |
8630314 | York | Jan 2014 | B2 |
8664351 | Hamasaki et al. | Feb 2014 | B2 |
9130687 | Webb, III et al. | Sep 2015 | B2 |
9219562 | Chen | Dec 2015 | B2 |
9225344 | Jones | Dec 2015 | B2 |
9331805 | Steiner et al. | May 2016 | B2 |
9544079 | Dalvi et al. | Jan 2017 | B2 |
9548833 | Bui et al. | Jan 2017 | B2 |
9667370 | Tzeng et al. | May 2017 | B2 |
9670907 | Bengtson et al. | Jun 2017 | B2 |
9671761 | Dougan et al. | Jun 2017 | B2 |
9749972 | Bin Sediq et al. | Aug 2017 | B2 |
9759816 | Achanta | Sep 2017 | B2 |
9813175 | Bottari et al. | Nov 2017 | B2 |
9948420 | Wright | Apr 2018 | B2 |
9973601 | Spada et al. | May 2018 | B2 |
9983555 | Bengtson | May 2018 | B2 |
10019333 | Regev | Jul 2018 | B2 |
10020905 | Farra et al. | Jul 2018 | B2 |
10033593 | Kumbhari et al. | Jul 2018 | B2 |
10078312 | Bengtson et al. | Sep 2018 | B2 |
20030079054 | Miller | Apr 2003 | A1 |
20030158972 | Friedli et al. | Aug 2003 | A1 |
20040158759 | Chang et al. | Aug 2004 | A1 |
20050071703 | Lee et al. | Mar 2005 | A1 |
20050141565 | Forest et al. | Jun 2005 | A1 |
20060250160 | Chang et al. | Nov 2006 | A1 |
20080250185 | Clark et al. | Oct 2008 | A1 |
20090222589 | Kirsch et al. | Sep 2009 | A1 |
20090241073 | Ellavsky et al. | Sep 2009 | A1 |
20100180143 | Ware et al. | Jul 2010 | A1 |
20110109360 | Dennis et al. | May 2011 | A1 |
20110158120 | Hamasaki et al. | Jun 2011 | A1 |
20130227008 | Yang | Aug 2013 | A1 |
20140003199 | Dougan et al. | Jan 2014 | A1 |
20160170437 | Aweya | Jun 2016 | A1 |
20160211936 | Bottari et al. | Jul 2016 | A1 |
20160301416 | Milijevic et al. | Oct 2016 | A1 |
20170013293 | White et al. | Jan 2017 | A1 |
20170222743 | Ruffini et al. | Aug 2017 | A1 |
20170273044 | Alsina et al. | Sep 2017 | A1 |
20170346588 | Prins et al. | Nov 2017 | A1 |
20180013508 | Rabinovich et al. | Jan 2018 | A1 |
20210167941 | Yamaguchi et al. | Jun 2021 | A1 |
Number | Date | Country |
---|---|---|
10-2015-0038127 | Apr 2015 | KR |
10-2018-0122688 | Nov 2018 | KR |
Entry |
---|
International Search Report, dated Feb. 21, 2020, and Written Opinion, dated Feb. 20, 2020, issued in PCT/US2019/033455, 8 pgs. |
Number | Date | Country | |
---|---|---|---|
20240160242 A1 | May 2024 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 17189826 | Mar 2021 | US |
Child | 18304404 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 18304404 | Apr 2023 | US |
Child | 18419710 | US | |
Parent | PCT/US2019/033455 | May 2019 | WO |
Child | 17189826 | US |