Patent Application
20070285501
The present invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, and like reference numerals designate like structural elements.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. In other instances, well known process operations and implementation details have not been described in detail in order to avoid unnecessarily obscuring the invention.
A computer cluster is a group of loosely coupled computers that work together to solve problems, thereby multiplying their individual processing power. There are many different types of computer clusters. For instance, some clusters provide hardware redundancy for improved availability; some provide load-balancing functions whereby network traffic is routed through a front end for distribution to a back end cluster of servers. Other clusters split computational tasks into smaller elements and distributes them throughout a cluster for parallel processing. Because of the conceptual nature of computer clusters, the term, “cluster” in the present context should be understood to have a broad general meaning.
Network connections 110 may, for example, be Internet connections. Each external client 104 is operated by a meeting attendee. The master server 102 and the external clients 104 allow each meeting attendee to see and hear each other in real time using videoconferencing technology as described in related U.S. patent applications Ser. Nos. 10/192,130 and 10/192,080, which are incorporated by reference herein. By “real time” it is meant that a transmission sent from one user is received by other users in an amount of time that depends primarily on network latency.
In addition to external clients 104, cluster node servers 122, 122′ are in communication with master server 102. With respect to master server 102, each cluster node server 122, 122′ behaves similarly to an individual external client 104 as will be described in more detail herein below.
Each cluster node server 122, 122′ lives in a corresponding local area network 120, 120′ and connects with master server 102 over one or more network connections 112, 112′, which may, for example, be over a wide area network such as the Internet. Each cluster node server 122, 122′ also connects to a plurality of cluster node clients 130, 130′ over local area connections. That is, the cluster node server 122, 122′ are each on the same local area network (LAN) 120, 120′ as the respective cluster node clients 130, 130′. In addition, each cluster node server 122, 122′ includes a respective local client 128, 128′. Users of cluster node clients 130 and local clients 128 can see and hear each other by passing audio and video information through cluster node server 122. In addition, users of cluster node clients 130′ and local client 128′ can see and hear each other by passing audio and video information through cluster node server 122′. Finally, users of cluster node clients 130, 130′ and local clients 128, 128′ and external clients 104, can all see and hear each other by passing audio and video information through master server 102. In addition to seeing and hearing each other, user data such as documents, annotations, and live desktop views can be transmitted to other users from any user's location.
Master server 102, cluster node server 122, outside clients 104, and cluster node clients 130 each comprise a multipurpose computer system.
MM switch 204 directs data traffic flowing between the server processes, MCU server 200 and MM server 202 and the connected clients, which include local client 128 and cluster node clients 130. MM switch 204 maintains connections to the external cluster node clients 130 over a local area network connection. In one embodiment, local client 128 communicates to MM switch 204 over a local “virtual” connection within cluster node server 122. In another embodiment, local client 128 communicates by other known means, such as by an inter-process communication mechanism provided by the platform. MM switch 204 connects the connected clients to MCU server 200 and MM server 202 through inter-process communication or other means. With regard to audio-video streams, MM switch pass audio-video information from each connected client to MCU server 200, and a composite audio-video received from master server 102 to each connected client.
MCU server 200 receives audio and video streams from MM switch 204, which receive them from local client 128 and cluster node clients 130. At least a portion of the audio and video streams received by MCU server 200 is transmitted to master server 102. In one embodiment, MCU server 200 combines the audio feeds and downsamples the video and generates a composite video which is than transmitted with the combined audio to master server 102 via cluster agent 180. In another embodiment, MM server 202 selects one of the videos for transmission to master server 102. The selection may be made automatically or manually. For instance, the cluster client having the loudest audio signal can be selected to be transmitted, whereas a manual selection can be made by local client 128 via a special interface as described in more detail below with reference to
MM server 202 receives various data from local client 128 and cluster node clients 130 via MM switch 204. The data includes application sharing data, file and document sharing, and control signals. Application sharing data can include a live view of the user's desktop and/or data representing keyboard and/or mouse input. File and document sharing provides a mechanism to share documents or other fixed data for review by other attendees. The control signals relate to commands or status data sent and received from each client to the servers.
Cluster agent 180 includes H.323 connections 182 connecting MCU server 200 with master server 102 to transmit audio/video data to and from master server 102. H.323 is a codec and specification from the ITU Telecommunication Standardization Sector (ITU-T). H.323 is an industry standard codecs and protocol to provide audio-visual communication sessions on any packet network. In addition to video data, cluster agent 180 maintains an application sharing channel 184 for sending and receiving application sharing data to and from local client 128 and cluster node clients 130. Typically, only one conference attendee will be sharing his or her desktop at a time, so all application sharing data of connected clients can be handled with a single connection to master server 102. Media file uploader channel 186 for sharing document data is also shared by all users connecting to the conference via cluster node server 122. Control data is passed to and from master server 102 through individual control channels 190, one channel for each connected client.
Cluster agent control module 192 communicates with MM server 202 to provide a channel for receiving commands such as startup, shut down, configuration, etc., as would be apparent to those skilled in the art. Each data channel within cluster agent 180 includes a connection to MCU server 200 or MM server 202 and a connection to master server 102. The connections to MCU server and MMserver 202 may be provided by any available inter-process communications means provided by the platform. Connections to master server 102 are provided through an outside network, and may entail traversing a firewall as mentioned in related U.S. patent application Ser. No. 10/681,732, which is incorporated herein by reference.
Video window 308 provides a composite video display of each participant identified by their name. For purposes of illustration, the terms, “Participant 1,” “Participant 2,” etc., are used in place of names. In this example, Participant 3 is a group which is connected by a cluster server. The cluster server has composited all the members of the group into a single video feed and presents the group as “Participant 3” to master server 102, which then composites that video with other videos received by other users to generate the display seen in video window 308. As mentioned previously, it is also possible to select an individual attendee or have the system automatically select one based on such factors as audio volume level.
In one embodiment, the configuration of video display 308 is user configurable by sending instructions to master server 102 from local client 128, or other connected client as assigned or selected by the cluster node server. As described in more detail in related U.S. patent application Ser. No. 10/192,080 which is incorporated herein, the videoconferencing system includes a client component and a server component. The client component includes a client monitor and a conference client. The client monitor captures input from the conference client. In one embodiment, the conference client is a peer-to-peer videoconferencing application. One example of a peer-to-peer videoconferencing application is MICROSOFT'S NETMEETING™ application. However, one skilled in the art will appreciate that any peer-to-peer videoconferencing application is suitable for the embodiments described herein. Thus, the system enhances pre-existing applications, which may already be installed on a personal computer, with increased functionality enabled through data provided by the client monitor. In addition, the client monitor can incorporate a graphical user interface (GUI) in which the video window of the peer-to-peer application is a component. The client monitor provides the captured input from the conference client to a server component. The captured input is transmitted to the server component which can dynamically modify the GUI being presented to a participant based on the captured input provided to the server component. For example, the client monitor can capture events, such as mouse clicks or mouse clicks in combination with keyboard strokes, executed by a user when his mouse pointer is within a region of the conference client that displays the video signal. In one embodiment, the events are transmitted through a back-channel connection to the server component for interpretation. Thus, the back-channel connection allows for active regions and user interface objects within the video stream to be used to control functionality and content. Consequently, users indirectly control how video is presented in different regions or can manipulate the layout based upon server processing of client events. As will be described below, additional communication exchange is available between participants using this system's back-channel connection.
Broadcast select window 312 therefore provides an interface for a selected user, e.g., the user of local client 128, to select how cluster node server 122 manages the multiple connected client video streams. It is also possible to allow or assign responsibility to one of cluster clients 130 to manage the broadcast select option. In the example provided, broadcast select window 312 includes an “Auto Select” option, and “Composite” option” and options for each connected client, which are each identified by their name. For purposes of illustration, the terms “Participant 5” “Participant 6” etc., are used in place of names and the cluster node is connected to master server 102 as “Participant 3” as shown in video window 308. When “Composite” is selected, the cluster node server generates a composite image of all users connected to the cluster node server; when “Auto Select” is selected, the cluster node server automatically selects which user's video feed to transmit to the master server. Such a selection may be based on a number of criteria, e.g., last person to speak based on audio signal loudness, length of time being broadcast, and random, etc. Other options are provided for manually selecting the video feed. Additional details relating to graphical user interface 300 are described in related U.S. patent application Ser. No. 10/192,080 which is incorporated herein by reference.
In operation 340, each cluster node server transmits at least a portion of the audio/video streams received by its connected cluster node clients and local client to the master server. The portion could be just a selected one of the video streams received from the clients or it could be a downsampled and composited version of video streams from all of the connected cluster node clients and local client. The composite audio/video stream that may be generated by a cluster node server is referred to herein as a “local composite stream.” With regard to the audio signals, all the audio signals from connected cluster node clients and the local client are combined into a single audio stream which may be packaged with the selected video stream or composited video stream.
In operation 342, the master server 102 combines the audio/video streams received from outside clients 104 and cluster node servers 122 into a composite audio/video stream. In one embodiment, the videos are downsampled and combined to form a composite image and the audio signals are added together. In one embodiment, the composite audio/video stream is customized for each outside client and cluster node server in accordance with their preferences and commands. Thus, each outside client and a selected user connected to each cluster node server can select from multiple layout options for the composite video.
In operation 344, each composite audio/video stream generated by the master server is transmitted to a respective one of the outside clients and cluster node servers. Thus, each outside client and each cluster node server receives the composite audio/video stream generated for their use. The cluster node servers, in operation 346, forward the composite audio/video stream received from the master server to the cluster node clients. In addition, if a local client is present on the cluster node server, the local client receives the composite audio/video stream as well. The procedure then ends as indicated by finish block 348.
In operation 358, a meeting owner creates a meeting and selects outside attendees and groups allowed to join a meeting. If a particular attendee is a member of a group, they need not separately register with the master server. Outside attendees are individual users that access the master server directly, i.e., not through a cluster node server. Each outside attendee and group is given a unique username and password, which allows them to log in to the master server at the time of the meeting.
In operation 360, the cluster node server connects to the master server using its unique group name and password, and creates temporary accounts on the master server for each of its group members which are users connected to the cluster node server. This allows each of the users connecting through the cluster node server to be identified by others outside the cluster node. A control channel is established for each client connected to the cluster node server.
In operation 362, the cluster node server creates temporary local accounts for all users external to the node server. This allows each user connected through the cluster node server to identify and interact with individual users outside the cluster node. Individual cluster node server is fully responsible for whom to accept to participate in the videoconference call. The user's login information is kept on the individual cluster node server. As a result, the participant user authentication is distributed around the cluster.
Once the temporary accounts are created, the attendees are able to see, hear, and otherwise interact with each other through the conferencing system, and the procedure is completed as indicated by finish block 364.
With the above embodiments in mind, it should be understood that the invention can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared and otherwise manipulated. Further, the manipulations performed are often referred to in terms such as producing, identifying, determining, or comparing.
Any of the operations described herein that form part of the invention are useful machine operations. The invention also relates to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
The invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. The computer readable medium also includes an electromagnetic carrier wave in which the computer code is embodied. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
Embodiments of the present invention can be processed on a single computer, or using multiple computers or computer components which are interconnected. A computer, as used herein, shall include a standalone computer system having its own processor(s), its own memory, and its own storage, or a distributed computing system, which provides computer resources to a networked terminal. In some distributed computing systems, users of a computer system may actually be accessing component parts that are shared among a number of users. The users can therefore access a virtual computer over a network, which will appear to the user as a single computer customized and dedicated for a single user.
Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
This application is related to U.S. patent application Ser. No. 10/192,130 filed on Jul. 10, 2002 and entitled “Method and Apparatus for Controllable Conference Content Via Back-Channel Video Interface;” U.S. patent application Ser. No. 10/192,080 filed on Jul. 10, 2002 and entitled “Multi-Participant Conference System with Controllable Content Delivery Using a client monitor Back-channel;” U.S. patent application Ser. No. 10/681,732 filed on Oct. 8, 2003 and entitled, “Connectionless TCP/IP Data Exchange;” U.S. patent application Ser. No. 11/051,674 filed on Feb. 4, 2004 and entitled “Adaptive Bitrate Adjustment of Multimedia Communications Channel Using Transport Control Protocol;” U.S. patent application Ser. No. 11/340,062 filed on Jan. 25, 2006 and entitled, “IMX Session Control and Authentication;” and U.S. patent application Ser. No. 11/199,600 filed on Aug. 9, 2005 and entitled “Client-Server Interface to Push Messages to the Client Browser” all of which are incorporated herein by reference.
