Patent Grant
7426578
1. Field of the Invention
The field of the invention relates generally to network communications and more particularly to synchronizing data shared between a plurality of networked communication devices.
2. Background Information
Networked communication devices often need to share information. For example, the need often arises in networked environments where data on one device needs to be replicated on one or more other devices. It will be understood that even the simple task of sharing data can be problematic in a networked environment; however, problems are further complicated when the data to be shared is dynamic or changing over time in such a way that only the latest set of data is of value. Various approaches exist for accomplishing the replication of dynamic data in a networked environment; however, conventional approaches are limited and often ineffective.
One conventional approach to sharing data in a networked environment uses repositories of data known as buffers on each of the devices. The buffers can then be synchronized by sending messages over a network connection. In order to reduce the amount of data sent and thus the usage of network connection bandwidth, an entire copy of the original, or source buffer, can be sent to one or more destination buffers. Thereafter, only changes in the source buffer are sent at various intervals. The destination device can then apply the changes to its local buffer to maintain synchronization with the source buffer. This approach allows the update interval to be selected to match the desired, or available network bandwidth between the source and receiving device. Selecting the update interval, however, can be problematic.
For example, updates can be sent to a destination device anytime a change is made to the buffer on the source device. But this can be ineffective because the destination device or interconnecting network link may be incapable of accepting and processing the updates at the rate changes occur at the source device. Accordingly, updates must either be discarded, resulting in loss of synchronization, or queued-up, in which case a lag develops between source and destination devices corresponding to the length of the queue. Such loss of synchronization or lag between destination and source devices can lead to problems. Moreover, queues may also consume significant and potentially unbounded resources, leading to further problems.
Alternatively, updates can be sent when requested by the destination device. This allows updates to be sent at a rate that they can be processed, but the receiving buffer is only synchronized with the source buffer at times when an update is sent. Thus, the source buffer may go through several intermediate states in the interval between updates. These intermediate states will not be reflected in the destination buffer.
A further drawback to conventional approaches can occur when a plurality of destination buffers must be synchronized with a source buffer. Often the data handling capability of each destination differs. Further, the network connections between source and each of the destination devices are not necessarily identical in terms of bandwidth, speed, latency, or reliability. As a result, changes sent to the destination devices can be sent only as frequently as can be handled by the slowest connected device or network connection. Accordingly, devices capable of receiving more information or more intermediate states are not able to operate using their full capability.
For example, a multimedia collaboration session, where a user's computer display, or region of the display, is shared with remote viewers, can be used to illustrate the problems with conventional data sharing approaches. The portion of the display to be shared is often captured in a buffer and transmitted to other viewers' computers. As the display changes, the source buffer is updated and updates are sent to destination buffers and displayed on viewers' displays; however, these updates are sent only at the rate corresponding to the slowest of all the connected networks and devices. Accordingly, even users with fast computers will experience less frequent updates and unpleasant artifacts such as jerkiness, low frame rate, and poor quality in displays involving changes or motion.
Alternatively, a separate instance of the source buffer can be maintained for each destination device and separate computation of changes. And update message transmission can be performed for each connected destination device. This technique allows each device to receive updates at a rate that best uses available network and device capabilities; however, this approach suffers from a limitation in that maintaining buffers and computing changes requires large amounts of memory and processing power. Thus, maintaining a separate buffer instance for each connected destination limits the number of endpoints that can be simultaneously connected. This is a serious limitation in a system such as a multimedia collaboration system, which may be of use only if a certain number of parties are able to connect.
Thus, a significant implementation challenge exists in synchronizing multiple destination buffers and devices to a source buffer containing data that changes over time. This is especially true when the data handling capacity of connected destination devices are not equal, as is typical in conventional networked environments. If all destinations devices are sent updates for every change in the source buffer, the volume of data may overwhelm the capacity of some devices and network links, resulting in loss of synchronization for those devices. If data is sent at a rate compatible with all devices, i.e. sent at the rate of the slowest receiving device and network link, devices with greater capability will receive poor quality data. If a separate data stream is created for each connected device, the resources of the sending device may become taxed and the result will be a limit to the number of destination devices that can connect simultaneously.
A source communication device comprises a cluster manager to group destination device connections into a plurality of performance clusters. The cluster manager can then assign a synchronization mechanism to each performance cluster. Each synchronization mechanism can then allow data to be optimally shared between the source device and destination devices within a given performance cluster.
These and other features, aspects, and embodiments of the invention are described below in the section entitled “Detailed Description of the Preferred Embodiments.”
Features, aspects, and embodiments of the inventions are described in conjunction with the attached drawings, in which:
The number of destination devices 111a-111e and the number of associated communication connections are shown for illustration only. Only a small number of connections and destination devices 111a-111e are shown for clarity and should in no way imply a limit or suggestion as to the number of communication devices 111a-111e that can be supported using the systems and methods described herein.
The number of destination devices 111 and the number of associated communication connections are shown for i11ustration only. Only a small number of connections and destination devices 111 are shown for clarity and should in no way imply a limit or suggestion as to the number of communication devices 111 that can be supported using the systems and methods described herein.
Often, communication connections 106-110 can have different performance capabilities. As described below, performance capability can be defined in a variety of ways, such as the bandwidth capability of each connection. Thus for example, connections 106 and 107 can have a relatively high bandwidth capability, while connection 108 can have a somewhat lower bandwidth capability and connections 109 and 110 can have a still lower bandwidth capability. Accordingly, cluster manager 118 can be configured to group destination devices 111a-111e, or communication links 106-110, into performance clusters, e.g., performance clusters 119, 120, and 121, based on their similar performance capabilities. In
Source device 101 can be configured to share data stored in the source data buffer 102 with the plurality of destination devices 111a-111e. Communication connections 106-110 provide mechanisms for transferring data, i.e., physical communications channels, while the synchronization mechanisms 103-105 and corresponding synchronization mechanisms 116a, 116b, 116c, 116d and 116e associated with destination devices 111a-111e can be configured to provide the computation and protocols needed to share data between source data buffer 102 and destination data buffers 117a-117e over communication connections 106-110.
In one embodiment, the similarity in performance capability of the various communication connections is determined, in step 304, by maintaining statistics for data rate capability of each communication connection. Such statistics can be derived, for example, from observing data transfers between source device 101 and destination devices 111a-111e. In another embodiment, connection security associated with each of the communication connections 106-110 can be used to determine similar performance capabilities in step 304. In still another embodiment, the error rate associated with data transfer of each communication connection can be used in step 304. In yet another embodiment, latency associated with data transfer of each communication connection can be used in step 304. In fact, it will be understood that a variety of performance parameters and related information can be used to determine the similarity in the performance capabilities of the various communication connections. Thus, nothing within the specification or the claims that follow should be seen as limiting the systems and methods described herein to the use of any particular parameters or set of parameters.
In one embodiment, the number of synchronization mechanisms can be determined dynamically and can change as needed to accommodate destination devices 111a-111e as they connect with source device 101. Several algorithms can be used in selecting the number of synchronization mechanisms, some of which are described below. This dynamic capability can allow for a trade off between improved client service, which results when there are fewer destination devices 111a-111e per cluster, and reduced server resource usage, which results from having a large amount of clusters. Thus, for example, if there are only a few destination devices 111a-111e, or if client service is important, then cluster manager can assign, for example, each destination device 111 to its own synchronization mechanism. On the other hand, if there are a lot of destination devices 111a-111e, or if client service is not as important, then cluster manager can assign fewer synchronization mechanisms. Moreover, depending on the embodiment, cluster manager 118 can be configured to dynamically update the destination device groupings and add or remove synchronization mechanisms as required.
Further, in one particular embodiment, the correspondence between a destination device 111a-111e and a particular synchronization mechanism 103, 104, or 105 can also be dynamic. In other words, the corresponding communication connection for a particular destination device 111a-111e can be moved to a different synchronization mechanism if the corresponding performance capabilities change such that a different performance cluster 119, 120, or 121 is more appropriate. Thus, for example, cluster manager 118 can be configured to monitor, in step 312, a set of statistics associated with the performance of each communication connection 106-110 and to detect any change therein. If a significant change is detected, then the statistics can be used to determine if another performance cluster 119, 120, or 121 is more appropriate for the particular destination device 111a-111e.
In one embodiment, all connections 106-110 or all destination devices 111a-111e begin a session as part of a primary performance cluster. After a small number of updates the average latency for each destination device 111a-111e or communication connection 106-110 is gathered. Cluster manager 118 can be configured to then perform a cluster division (step 306) to organize destination devices 111a-111e according to their performance levels, e.g., their average latencies. For example, an initial calculation of the average latencies for each of the plurality of connections can be performed and used to determine the mean latency for the primary performance cluster. A standard deviation relative to the mean can also be calculated. The number of performance clusters required can then be determined based on the percentage of communication connections 106-110 with latencies within a certain number of standard deviations from the mean.
In one particular embodiment, a minimum standard deviation threshold can be used in order to prevent the creation of extra clusters when the performance level is very similar.
Further, in one embodiment, communication connections 106-110 can be placed into an appropriate performance cluster (step 308) using an algorithm, such as the K-means algorithm. The K-means algorithm is a partitioning method based on minimizing the sum-of-squares distance between average latencies for each communication connection 106-110 and the mean latency for the primary performance cluster, allowing division of communication connections 106-110 into (K) performance clusters. This is an iterative approach that maximizes the ratio of variation between performance clusters relative to the variation within a performance cluster. This approach allows a quick calculation with a resulting distinct separation of performance levels. Depending on the embodiment, the K-means algorithm is executed periodically or as needed, e.g., if there is a change in performance capabilities (step 312) or a new destination device 111a-111e joins the session (step 314).
In step 314, the cluster manager 118 can monitor the communication connections 106-110 to detect new connection, i.e., a communication connection that is established after a session has begun. The cluster manager 118 can determine the performance capabilities of the new connection and add the new connection to a performance cluster based upon the performance capabilities of the new connection. In one embodiment, such communication connections will be inserted into a performance cluster (step 308) without moving other communication connections at least until some initial lag statistics are computed for the new communication connection. The new communication connection can, for example, then be moved into a performance cluster based on, e.g., a least squares analysis.
A new insertion into a performance cluster can require a resynchronization for destination devices 111a-111e within the performance cluster on the next cluster-wide update. In one embodiment, only the newly added destination device 111a-111e need be resynchronized while the other destination devices 111a-111e remain synchronized.
A destination device 111a-111e that does not respond within a timeout threshold, as determined in step 412, can be removed from its performance cluster, in step 414, so that other destination devices 111a-111e in the performance cluster can still receive updates. This can, for example, prevent a network interruption or an issue associated with a destination device 111a-111e from harming the other cluster participants' experiences. If a destination device 111 has reached this timeout, in step 412, but eventually responds, in step 416, then it can still be allowed to receive full-buffer updates, in step 418, e.g., until its Recent Average Latency performance merits insertion back into one of the performance clusters 119-121.
In one embodiment, the synchronization mechanisms 103-105 and the synchronization mechanisms 116a-116e can be configured to operate by dividing the data in source data buffer 102 into a number of blocks or sections. Initially, or whenever synchronization is lost, a complete set of all blocks can be sent from source device 101 to a destination device 111a-111e. The associated destination device buffer 117a-117e can then be updated using the complete set of blocks so that it is a replica of the source data buffer 102. Subsequently source device 101 can send only blocks that have changed subsequent to the last update sent to the destination data buffer 117a-117e. This approach can, for example, result in considerable savings in network bandwidth.
In one embodiment, updates are sent when all connected devices in a cluster have requested an update.
It should be noted that the original source data buffer can be located on a remote source device, e.g., a device that is not immediately connected with destination devices 111a-111e. For example,
In addition, intermediate source device 201 can comprise an intermediate source data buffer 202, which can be kept in synchronization with remote source data buffer 207 using synchronization mechanisms 203 and 205. Destination data buffers 117a-117e can then be kept in synchronization with intermediate source data buffer 202 as described above in relation to source data buffer 102.
In one example of
| Number | Name | Date | Kind |
|---|---|---|---|
| 5440699 | Farrand et al. | Aug 1995 | A |
| 5530795 | Wan | Jun 1996 | A |
| 5559875 | Bieselin et al. | Sep 1996 | A |
| 5583993 | Foster et al. | Dec 1996 | A |
| 5608426 | Hester | Mar 1997 | A |
| 5608872 | Schwartz et al. | Mar 1997 | A |
| 5623603 | Jiang et al. | Apr 1997 | A |
| 5634018 | Tanikoshi et al. | May 1997 | A |
| 5649104 | Carleton et al. | Jul 1997 | A |
| 5706502 | Foley et al. | Jan 1998 | A |
| 5742670 | Bennett | Apr 1998 | A |
| 5745711 | Kitahara et al. | Apr 1998 | A |
| 5748898 | Ueda | May 1998 | A |
| 5754830 | Butts et al. | May 1998 | A |
| 5761419 | Schwartz et al. | Jun 1998 | A |
| 5764901 | Skarbo et al. | Jun 1998 | A |
| 5778368 | Hogan et al. | Jul 1998 | A |
| 5781909 | Logan et al. | Jul 1998 | A |
| 5786814 | Moran et al. | Jul 1998 | A |
| 5809247 | Richardson et al. | Sep 1998 | A |
| 5809250 | Kisor | Sep 1998 | A |
| 5819243 | Rich et al. | Oct 1998 | A |
| 5844979 | Raniere et al. | Dec 1998 | A |
| 5859974 | McArdle et al. | Jan 1999 | A |
| 5894556 | Grimm et al. | Apr 1999 | A |
| 5907324 | Larson et al. | May 1999 | A |
| 5916302 | Dunn et al. | Jun 1999 | A |
| 5930238 | Nguyen | Jul 1999 | A |
| 5930473 | Teng et al. | Jul 1999 | A |
| 5933417 | Rottoo | Aug 1999 | A |
| 5940082 | Brinegar et al. | Aug 1999 | A |
| 5944791 | Scherpbier | Aug 1999 | A |
| 5951652 | Ingrassia, Jr. et al. | Sep 1999 | A |
| 5953506 | Kalra et al. | Sep 1999 | A |
| 5978835 | Ludwig et al. | Nov 1999 | A |
| 5991276 | Yamamoto | Nov 1999 | A |
| 5991796 | Anupam et al. | Nov 1999 | A |
| 5995606 | Civanlar et al. | Nov 1999 | A |
| 5995951 | Ferguson | Nov 1999 | A |
| 6018774 | Mayle et al. | Jan 2000 | A |
| 6038651 | VanHuben et al. | Mar 2000 | A |
| 6047323 | Krause | Apr 2000 | A |
| 6049602 | Foladare et al. | Apr 2000 | A |
| 6061440 | Delaney et al. | May 2000 | A |
| 6061717 | Carleton et al. | May 2000 | A |
| 6072463 | Glaser | Jun 2000 | A |
| 6104392 | Shaw et al. | Aug 2000 | A |
| 6105083 | Kurtze et al. | Aug 2000 | A |
| 6151621 | Colyer et al. | Nov 2000 | A |
| 6151688 | Wipfel et al. | Nov 2000 | A |
| 6157401 | Wiryaman | Dec 2000 | A |
| 6192394 | Gutfreund et al. | Feb 2001 | B1 |
| 6199169 | Voth | Mar 2001 | B1 |
| 6237025 | Ludwig et al. | May 2001 | B1 |
| 6240444 | Fin et al. | May 2001 | B1 |
| 6263361 | Hoyer et al. | Jul 2001 | B1 |
| 6263365 | Scherpbier | Jul 2001 | B1 |
| 6295551 | Roberts et al. | Sep 2001 | B1 |
| 6295585 | Gillett et al. | Sep 2001 | B1 |
| 6301258 | Katseff et al. | Oct 2001 | B1 |
| 6310941 | Crutcher et al. | Oct 2001 | B1 |
| 6321252 | Bhola et al. | Nov 2001 | B1 |
| 6335927 | Elliott et al. | Jan 2002 | B1 |
| 6343313 | Salesky et al. | Jan 2002 | B1 |
| 6411988 | Tafoya et al. | Jun 2002 | B1 |
| 6463585 | Hendricks et al. | Oct 2002 | B1 |
| 6519660 | Rooney | Feb 2003 | B1 |
| 6542936 | Mayle et al. | Apr 2003 | B1 |
| 6598074 | Moller et al. | Jul 2003 | B1 |
| 6728784 | Mattaway | Apr 2004 | B1 |
| 6748447 | Basani et al. | Jun 2004 | B1 |
| 6785244 | Roy | Aug 2004 | B2 |
| 20010034752 | Kremien | Oct 2001 | A1 |
| 20020031126 | Crichton et al. | Mar 2002 | A1 |
| 20020056006 | Vange et al. | May 2002 | A1 |
| 20020177910 | Quarterman et al. | Nov 2002 | A1 |
| 20030093597 | Marshak et al. | May 2003 | A1 |
| 20030229900 | Reisman | Dec 2003 | A1 |
| 20050015471 | Zhang et al. | Jan 2005 | A1 |
| 20050080850 | Salesky et al. | Apr 2005 | A1 |
| Number | Date | Country |
|---|---|---|
| 19717167 | Oct 1998 | DE |
| 0622931 | Nov 1994 | EP |
| 0874323 | Oct 1998 | EP |
| 0908824 | Apr 1999 | EP |
| 7049837 | Feb 1995 | JP |
| 11249995 | Sep 1999 | JP |
| WO 9926153 | May 1999 | WO |
| WO 0056050 | Sep 2000 | WO |
| WO 0169418 | Sep 2001 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20050132074 A1 | Jun 2005 | US |
