The present invention relates generally to a communications mechanism for use between computing devices. More specifically, the invention relates to a communication mechanism for use in a distributed computing system.
In a distributed computing environment, it is difficult to provide direct communications between large numbers of computing devices where each computing device communicates with each other computing device. For example, one solution is to use 1-to-N multicast channels. That is, each computing device (e.g., a participant to an on-line meeting or webinar) creates a multicast channel and every other computing device subscribes to every other computing device's channel. In such a solution, a system with N computing devices requires N channels and N2 channel subscriptions. As a result, a need arises to maintain O(N2) state. Such a solution scales poorly for a large number of computing devices.
As a high level overview, the present invention avoids the overhead required for the N2 solution. Generally, the invention features a single shared channel to which each computing device subscribes. As a result, only N subscriptions and O(N) state is required instead of N2 subscriptions and O(N2) state required when using N 1-to-N multicast channels. Each computing device publishes its current state on the shared channel in the form of a set of packets. Each computing device receives updates of the other computing devices' state.
Also, in some cases, the invention features the ability to propagate data in a bandwidth adaptive manner. This allows for computing devices with both slow (e.g., 56K) and fast (e.g., T1 and broadband) connection speeds to participate in the system without interfering with one another.
In one aspect, the invention features a method for bandwidth adaptive communication among a plurality of participants in a distributed system. The method includes receiving a communication from a first participant, storing payload data in a channel memory element, and transmitting the payload data to a second participant. The communication includes metadata and payload data. The channel memory element includes a plurality of memory locations. One of the plurality of memory locations is selected using the metadata and is associated with the first participant.
In one embodiment, the communication includes a participant identifier and payload data. The communication can represent the state of the participant and include metadata and payload data.
In another embodiment, storing includes storing the payload data in one of a plurality of array locations. The one of the plurality of array locations is selected using the metadata and associated with the first participant.
In other embodiments, transmitting includes transmitting the array to a second participant. Transmitting also can include transmitting the payload data to a second participant in response to information identifying the first participant payload data most-recently received from the first participant. Transmitting can also include transmitting the payload data to a second participant in response to a request from the second participant. Further, transmitting can include transmitting the payload data to a second participant in response to a request from the second participant. The request identifies the first participant payload data most-recently stored by the second participant.
In another embodiment, the method includes before transmitting, receiving a second communication from the first participant. The communication includes second payload data. Further, the method includes storing the second payload data in the one of a plurality of memory locations associated with the first participant.
In yet another embodiment, the method includes receiving a communication from a second participant, storing the payload data in one of a plurality of memory locations, and transmitting the payload data to the first participant. The communication includes metadata and payload data. The one of the plurality of memory locations is selected using the metadata and associated with the second participant.
In another embodiment, the method includes receiving a second communication from the first participant. Further, the method includes storing the second communication in a second memory location associate with the first participant.
In another aspect, the invention features a distributed system for bandwidth adaptive communication among a plurality of participants. The system includes a plurality of participants transmitting state data and a server. The server receives the state data transmitted by the plurality of clients, stores the received state data in a memory element having a plurality of memory location, and transmits the state data received from one of the plurality of participants to each of the other ones of the plurality of participants. Each of the plurality of memory locations is associated with a respective one of the plurality of participants.
The foregoing discussion will be understood more readily from the following detailed description of the invention, when taken in conjunction with the accompanying drawings, in which:
With reference to
In other embodiments, the participants 110 communicate with the server 150 through a second network 140′, through a communication link 180 that connects network 140 to the second network 140′. The protocols used to communicate through communications link 180 can include any variety of protocols used for long haul or short transmission. For example, TCP/IP, IPX, SPX, NetBIOS, NetBEUI, SONET and SDH protocols. The combination of the networks 140, 140′ can be conceptually thought of as the Internet. As used herein, Internet refers to the electronic communications network that connects computer networks and organizational computer facilities around the world.
The participant 110 can be any personal computer, server, Windows-based terminal, network computer, wireless device, information appliance, RISC Power PC, X-device, workstation, minicomputer, personal digital assistant (PDA), main frame computer, cellular telephone or other computing device that provides sufficient faculties to execute participant software and an operating system. Participant software executing on the participant 110 provides the ability to read and write to a shared communication channel established among the participants 110 and the server 150.
The server 150 can be any type of computing device that is capable of communication with one or more participants 110. For example, the server 150 can be a traditional server computing device, a web server, an application server, a DNS server, or other type of server. In addition, the server 150 can be any of the computing devices that are listed as participant devices. In addition, the server 150 can be any other computing device that provides sufficient faculties to execute server software and an operating system. Server software executing on the server 150 provides the functionality to create a shared communication channel among the participants 110 and the server 150. Additional functionality provided by the server software includes, but is not limited to, reading and writing to the shared communication channel.
In one embodiment, different channels carry different types of communications among the participants 110 and the server 150. For example in an on-line meeting environment, a first communication channel carries screen data from a presenting participant 110 to the server 150, which, in turn, distributes the screen data to the other participants 110. A second communications channel is shared, as described in more detail below, to provide real-time, communications (e.g., chat information, business card information, question and answer information, polling information, session feedback information, voice data, video data, distributed control information, state information, and the like) among the participants.
The participant software 212 is in communication with various components (e.g., the operating system 208) of the participant 110 to provide features of the invention. As a general overview, the participant software 212 creates and maintains a local copy of the shared channel. The participant software 212 provides a means to read and write data to the shared channel and transmit the changes to the other participants 110 and the servers 150.
With reference to
The server software 312 is in communication with various components (e.g., the operating system 308) of the server 150 to provide features of the invention. As a general overview, the server software 312 creates the shared channel and maintains a local copy of the shared channel state. The server software 312 provides a means to transmit the changes to the other participants 110 or other servers 150.
With reference to
Each field 410 stores a respective data packet 414 (i.e., 414A, 414B, 414C, 414D, 414E, 414E′, 414E″, 414F, 414F′, 414G, 414G′, 414G″). Each participant can read each of the memory locations 410 of the shared channel, but can only write to those memory locations 410 associated with the respective participant 110. Each data packet 414 includes metadata and payload data. The metadata can include, but is not limited too, a participant ID, a memory location ID, and the like. The payload data can include, but is not limited to, communication data, chat data, state data, video data, distributed control data, and the like. In one embodiment, each field 410 includes one or more array locations. Each array location stores one of the metadata and the payload data. The array locations can be conceptually thought of as sub-slots within each slot of the shared channel 400.
With reference to
In one embodiment, the participant software 212 assigns (STEP 520) a monotonically increasing packet identifier. As used herein, monotonic refers to changing in one direction only; thus either strictly rising or strictly falling, but not reversing direction. In other embodiments, it should be understood that, other packet identifiers are used. Using monotonically increasing packet identifiers provides a means to allow synchronization among the local copies 236 of the shared communications channel 400 and the lateset state data of shared communications channel 400 stored by the server 150.
The transmitting (STEP 530) of the packet 414 to the shared communication channel 400 occurs after completion of the write operation of the participant. In one embodiment, the transmission occurs when the bandwidth between the participant 110 and the server 150 permits. This feature provides a bandwidth adaptive transmission module. For example, assume the participant 110A is connected to the server 150 using a 56 k modem the speed at which changes between the participant 110A and the server 150 are communicated differs greatly than if the participant 110A and the server 150 are connected by a T1 line. As such, the participant 110A waits until the bandwidth permits the transmission. These same bandwidth adaptive principles also apply when the participant 100 receives packets 414 from the server 150.
With reference to
In one embodiment, storing (STEP 620) includes overwriting the contents of the associated memory location 410 by the server software 312 with the new packet 414. Prior to overwriting the contents of the associated memory location 410, the server software compares the packet identifier to determine if the received packet is actually newer than the current packet. A situation can arise, due to the bandwidth limitations connecting some of the participants 110 to the server 150 and when the system includes multiple servers 150 or peer-to-peer connection, where some of the packets 410 by the server software may actually be older than the current packet stored in the memory location. These situation are described below in more detail.
In one embodiment, transmitting (STEP 630) includes transmitting the entire contents of the shared communications channel 400 to each of the other participants 110. The transmission can occur in a multicast manner or some other manner. In another embodiment, only the updated memory locations 410 are transmitted to the participants. Said another way, the server software 312 synchronizes the contents of the shared channel with each of the participants. The synchronization occurs using either a “push” from the server 150 to the participants 110 or a “pull” from the participants. Each of which is described below.
In the case of a push from the server 150, a running history of the state of the shared channel 400 is maintained by the server 150 for each participant 110. When the server 150 receives a new packet 410, the server software 312 compares the present state of the shared communication channel 400 to the record of the running history to determine which packets 414 to transmit to the participant 110. Prior to transmitting the new packets 414 to the participant, the server software receives an indication or determines whether the bandwidth connecting the participant 110 to the server 150 is available to complete the communication. If the bandwidth is available, the new packets are transmitted. If the bandwidth is not available, the new packets are not transmitted. Instead, when the bandwidth is available a new comparison against the running history is performed and all the new packets are transmitted. As such, intermediate changes to the shared communications channels 400 are captured and sent to the participant.
In a pull model, each participant 100 periodically queries the server 150 for changes to the state of the shared communications channel 400. In one embodiment, each time a new packet is sent from the participant 110 to the server 150, a request for any new packets of the other participants 110 is included. In another embodiment, each participant 110 queries the server 150 every number of predetermined milliseconds or other time period. In still another embodiment, each participant 110 queries the server 150 when the bandwidth connecting the server 150 and the participant 110 is available to receive updates. As part of the query, the present state of the local copy 236 of the shared channel is transmitted to the server 150. The server software 312 compares the local copy 236 with the present state of the shared communication channel 400 to determine which packets to transmit to the requesting participant 110.
The described synchronization methods, provide a “loose” synchronization of the state of the shared communications channel 400 among the participants 110 and the server. The local copies 236 of the participants 110 differ during transitional periods while the synchronization mechanism propagates the changes. Given sufficient time and no new changes, each of the local copies 236 will converge to the same state of the shared communications channel 400. It is important to note that if the communication link between the server 150 and the participant 110 is bandwidth limited, the associated memory location 410 for the participant may be updated one or more times by the participant 110 before the new packet 410 is transmitted to the server 150. It should be understood that the same principles apply to receiving packets 414 from the server. This described synchronization does not guarantee delivery of all packets from a participant 110 to the server 150 or conversely from the server 150 to the participant 110. It only guarantees the deliver of the most recent packet when the bandwidth is available. This relatively weak delivery guarantee describes that bandwidth adaptive nature of the distributed computing environment, because it allows participants 110 and servers 150 with slower communication links 120 to coexist with participants 110 and servers 150 with faster communication links 120 without affecting one another. The participants 110 and servers 150 connected by slower (i.e., bandwidth limited) links receive fewer updates while participants 110 and servers 150 connected by faster communication links 120 (i.e., higher bandwidth links) may receive all updates.
With reference to
In
In
In
With reference to
In such an embodiment, the servers 150 may be communicating with each other using different bandwidth allocations. As such, the bandwidth adaptive features described above apply equally to communications among the servers 150 as well as among the servers 150 and the participants 110.
Also, a single participant can communicate with multiple servers 150. For example, participant 110F communicates with a first server 150B and a second server 150C. The participant 110F maintains a local copy of the state of the shared channel 400 as known by each of the first server 150B and the second server 150C. Such an embodiment provides redundancy and fault tolerance with respect to the state of the shared channel 400.
One exemplary implementation of the described invention is use in an on-line collaboration product to perform on-line meetings or webinars. An on-line meeting consists of one or more participants 110 that communicate through a communication server 150. It should be understood that multiple communications server 150 can used if the number of participants require more than a single communication server 150. In an on-line meeting, one of the participants is a presenter and controls the flow of the meeting. A presenter transmits a series of images that may represent a slide presentation. As the presenter displays a slide, the page image representing that slide is transmitted to all viewers. In many embodiments, each slide is represented by multiple data packets and transmitted over specific channel designated for screen sharing data. Each of the participants 110 are labeled “viewers” and view the shared screen data. Synchronization of the shared screen data among the viewers and presenter occurs using a reliable multicast protocol. This type of “sharing” represents a 1-to-N type of communication.
In addition to sharing screen data it is desirable to enable “chat” communication among the viewers and the presenter (e.g., viewer to viewer communication as well as viewer to presenter communications). As such, a shared channel 400 is established among the presenter and viewers to carry the chat data. Each viewer need to only subscribe to the shared channel to enable the chat feature of the on-line meeting. Each time the viewer types a chat message, the viewer's respective field 410 of the shared channel is updated with the new message payload. In response, the updated chat data is transmitted to the other viewers and presenter as described above. This type of “sharing” represents an N-to-N type of communication.
There are numerous on-line collaboration products that can operate in the distributed computing environment 100. Exemplary products include, but are not limited to GOTOMEETING and GOTOWEBINAR offered by Citrix Online, LLC of Santa Barbara Calif. Certain aspects and features described below can be embodied in such a product. Other products include WEBEX EMX, WEBEX ENTERPRISE EDITION, WEBEX EVENT CENTER, WEBEX GLOBALWATCH, WEBEX MEETING CENTER, WEBEX MEETMENOW, WEBEX PRESENTATION STUDIO, WEBEX SALES CENTER, WEBEX TRAINING CENTER, WEBEX WEBOFFICE, AND WEBEX WORKSPACE offered by WebEx Communications, Inc. of Santa Clara Calif. Another example is the LIVEMEETING product offered by of Microsoft Corporation of Redmond, Wash.
The previously described embodiments may be implemented as a method, apparatus or article of manufacture using programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof. The term “article of manufacture” as used herein is intended to encompass code or logic accessible from and embedded in one or more computer-readable devices, firmware, programmable logic, memory devices (e.g., EEPROMs, ROMs, PROMs, RAMs, SRAMs, etc.), hardware (e.g., integrated circuit chip, Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), etc.), electronic devices, a computer readable non-volatile storage unit (e.g., CD-ROM, floppy disk, hard disk drive, etc.), a file server providing access to the programs via a network transmission line, wireless transmission media, signals propagating through space, radio waves, infrared signals, etc. The article of manufacture includes hardware logic as well as software or programmable code embedded in a computer readable medium that is executed by a processor. Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope of the present invention.
Although the present invention has been described with reference to specific details, it is not intended that such details should be regarded as limitations upon the scope of the invention, except as and to the extent that they are included in the accompanying claims.
Number | Name | Date | Kind |
---|---|---|---|
4013828 | Judice | Mar 1977 | A |
4736369 | Barzilai et al. | Apr 1988 | A |
5031089 | Liu et al. | Jul 1991 | A |
5210753 | Natarajan | May 1993 | A |
5541927 | Kristol et al. | Jul 1996 | A |
5553083 | Miller | Sep 1996 | A |
5564016 | Korenshtein | Oct 1996 | A |
5727002 | Miller | Mar 1998 | A |
5754774 | Bittinger et al. | May 1998 | A |
5764235 | Hunt et al. | Jun 1998 | A |
5771383 | Magee et al. | Jun 1998 | A |
5812792 | Haddock et al. | Sep 1998 | A |
5826025 | Gramlich | Oct 1998 | A |
5845265 | Woolston | Dec 1998 | A |
5892905 | Brandt et al. | Apr 1999 | A |
5900018 | Lynch | May 1999 | A |
5918248 | Newell et al. | Jun 1999 | A |
5956027 | Krishnamurthy | Sep 1999 | A |
6018766 | Samuel et al. | Jan 2000 | A |
6081829 | Sidana | Jun 2000 | A |
6167432 | Jiang | Dec 2000 | A |
6175571 | Haddock et al. | Jan 2001 | B1 |
6226618 | Downs et al. | May 2001 | B1 |
6246758 | Low | Jun 2001 | B1 |
6249291 | Popp et al. | Jun 2001 | B1 |
6289382 | Bowman-Amuah | Sep 2001 | B1 |
6327276 | Robert et al. | Dec 2001 | B1 |
6332163 | Bowman-Amuah | Dec 2001 | B1 |
6339832 | Bowman-Amuah | Jan 2002 | B1 |
6343313 | Salesky et al. | Jan 2002 | B1 |
6400996 | Hoffberg et al. | Jun 2002 | B1 |
6415329 | Gelman et al. | Jul 2002 | B1 |
6434568 | Bowman-Amuah | Aug 2002 | B1 |
6434628 | Bowman-Amuah | Aug 2002 | B1 |
6438594 | Bowman-Amuah | Aug 2002 | B1 |
6442748 | Bowman-Amuah | Aug 2002 | B1 |
6477580 | Bowman-Amuah | Nov 2002 | B1 |
6477665 | Bowman-Amuah | Nov 2002 | B1 |
6496850 | Bowman-Amuah | Dec 2002 | B1 |
6502213 | Bowman-Amuah | Dec 2002 | B1 |
6529909 | Bowman-Amuah | Mar 2003 | B1 |
6529948 | Bowman-Amuah | Mar 2003 | B1 |
6539396 | Bowman-Amuah | Mar 2003 | B1 |
6549949 | Bowman-Amuah | Apr 2003 | B1 |
6550057 | Bowman-Amuah | Apr 2003 | B1 |
6571282 | Bowman-Amuah | May 2003 | B1 |
6578068 | Bowman-Amuah | Jun 2003 | B1 |
6584569 | Reshef et al. | Jun 2003 | B2 |
6601192 | Bowman-Amuah | Jul 2003 | B1 |
6601234 | Bowman-Amuah | Jul 2003 | B1 |
6606660 | Bowman-Amuah | Aug 2003 | B1 |
6615199 | Bowman-Amuah | Sep 2003 | B1 |
6615253 | Bowman-Amuah | Sep 2003 | B1 |
6636242 | Bowman-Amuah | Oct 2003 | B2 |
6640145 | Hoffberg et al. | Oct 2003 | B2 |
6640238 | Bowman-Amuah | Oct 2003 | B1 |
6640244 | Bowman-Amuah | Oct 2003 | B1 |
6640249 | Bowman-Amuah | Oct 2003 | B1 |
6658464 | Reisman | Dec 2003 | B2 |
6715145 | Bowman-Amuah | Mar 2004 | B1 |
6742015 | Bowman-Amuah | May 2004 | B1 |
6842906 | Bowman-Amuah | Jan 2005 | B1 |
6850252 | Hoffberg | Feb 2005 | B1 |
6918113 | Patel et al. | Jul 2005 | B2 |
6959320 | Shah et al. | Oct 2005 | B2 |
7006881 | Hoffberg et al. | Feb 2006 | B1 |
7013327 | Hickman et al. | Mar 2006 | B1 |
7023979 | Wu et al. | Apr 2006 | B1 |
7111298 | Michael et al. | Sep 2006 | B1 |
7113934 | Levesque et al. | Sep 2006 | B2 |
7197535 | Salesky et al. | Mar 2007 | B2 |
7200804 | Khavari et al. | Apr 2007 | B1 |
7310675 | Salesky et al. | Dec 2007 | B2 |
7369515 | Salesky et al. | May 2008 | B2 |
7418476 | Salesky et al. | Aug 2008 | B2 |
7426191 | Salesky et al. | Sep 2008 | B2 |
7484208 | Nelson | Jan 2009 | B1 |
7526469 | Narita et al. | Apr 2009 | B2 |
7593987 | Salesky et al. | Sep 2009 | B2 |
7627663 | Salesky et al. | Dec 2009 | B2 |
7715331 | Salesky et al. | May 2010 | B2 |
7716344 | Salesky et al. | May 2010 | B2 |
7769961 | Kottomtharayil et al. | Aug 2010 | B2 |
7813304 | Salesky et al. | Oct 2010 | B2 |
7822859 | Salesky et al. | Oct 2010 | B2 |
7836163 | Salesky et al. | Nov 2010 | B2 |
7877489 | Salesky et al. | Jan 2011 | B2 |
7934002 | Salesky et al. | Apr 2011 | B2 |
20010003193 | Woodring et al. | Jun 2001 | A1 |
20020049608 | Hartsell et al. | Apr 2002 | A1 |
20020049841 | Johnson et al. | Apr 2002 | A1 |
20020059274 | Hartsell et al. | May 2002 | A1 |
20020065864 | Hartsell et al. | May 2002 | A1 |
20020073059 | Foster et al. | Jun 2002 | A1 |
20020073238 | Doron | Jun 2002 | A1 |
20020078142 | Moore et al. | Jun 2002 | A1 |
20020083183 | Pujare et al. | Jun 2002 | A1 |
20020120607 | Price et al. | Aug 2002 | A1 |
20020133675 | Hirayama | Sep 2002 | A1 |
20020174227 | Hartsell et al. | Nov 2002 | A1 |
20030009538 | Shah et al. | Jan 2003 | A1 |
20030140159 | Campbell | Jul 2003 | A1 |
20030172372 | Crisan et al. | Sep 2003 | A1 |
20040031058 | Reisman | Feb 2004 | A1 |
20040098748 | Bo et al. | May 2004 | A1 |
20040103147 | Flesher et al. | May 2004 | A1 |
20040111639 | Schwartz et al. | Jun 2004 | A1 |
20040151177 | Burton et al. | Aug 2004 | A1 |
20050033824 | Takahashi et al. | Feb 2005 | A1 |
20050182909 | Volp et al. | Aug 2005 | A1 |
20060004765 | Anderson et al. | Jan 2006 | A1 |
20060064716 | Sull et al. | Mar 2006 | A1 |
20060136677 | Fuhs et al. | Jun 2006 | A1 |
20060161671 | Ryman et al. | Jul 2006 | A1 |
20070112880 | Yang et al. | May 2007 | A1 |
20070233691 | Chaushev | Oct 2007 | A1 |
20070233891 | Luby et al. | Oct 2007 | A1 |
20090019248 | Jeong et al. | Jan 2009 | A1 |
20090125904 | Nelson | May 2009 | A1 |
20100306674 | Salesky et al. | Dec 2010 | A1 |
Number | Date | Country |
---|---|---|
WO-03036499 | May 2003 | WO |
WO-2005107220 | Nov 2005 | WO |
Number | Date | Country | |
---|---|---|---|
20070260715 A1 | Nov 2007 | US |