Patent Application
20140114664
Video conferencing is a technology that has been instrumental in the advancement and development of global commerce. Video conferencing can facilitate meetings and collaborations between parties in different geographic locations, including different cities, states or provinces, and even different continents. Video conferencing can be conducted using dedicated video conferencing applications, or integrated into applications or websites for collaboration, social networking, public forums, and the like. In addition, dedicated and secured video conferencing systems can be used in business environments.
A video conferencing system typically includes one or more video capture devices in communication with a video conferencing server over a network. Some video conferencing systems even allow users to view multiple video streams at the same time. Video conferencing systems also typically include an audio capture device in communication with the video conferencing server.
The video and audio streams are generally communicated to the video conferencing server as digital data streams, such as Internet Protocol (IP) data streams. In other systems, the video is communicated independently of the audio. For example, some systems only communicate the video to the video conferencing server, while the audio is communicated via a telephone bridge.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Embodiments of systems and methods are provided to link the audio and video signals in a video conferencing system in order to identify a Dominant Speaker (DS). Multiview systems allow video streams from several video conferencing participants to be displayed simultaneously to users. In such systems, it is helpful to identify a dominant speaker so that the video associated with the dominant speaker may be displayed to users. A variety of criteria may be used to identify the dominant speaker, for example the speaker with the highest volume audio signal may be selected as a dominant speaker. Alternatively, a speaker with a most active audio signal may be selected as a dominant speaker.
Once the dominant speaker is identified by the video conferencing server, the video conferencing server may then push the video stream associated with the dominant speaker to the participants, and each participant's video conferencing application may indicate the dominant speaker in some way. For example, the video of the dominant speaker may be rendered in a larger window. Alternatively, the video of the dominant speaker may be highlighted or tagged in some manner.
Embodiments of methods and systems for dominant speaker identification in video conferencing are described. Such methods and systems identify a list of dominant speakers. In one embodiment, the list includes identification of a group of most recent dominant speakers. In another embodiment, the dominant speaker list includes identification of a group of most active dominant speakers. In one embodiment, the dominant speaker list may include a Media Source Identifier (MSID) associated with each of the dominant speakers on the list. The dominant speaker list may be distributed to each client in the video conference.
In one embodiment, each of the clients may request a set of video streams for rendering by the client. In one embodiment, the client may request the set of video streams associated with the dominant speakers identified on the dominant speaker list. Alternatively, each participant may request a set of video streams associated with a user selection. In still another embodiment, a combination of the dominant speakers on the dominant speaker list and the user selection may be requested by the client.
Embodiments of a computer-implemented method are described. In one embodiment, the computer-implemented method includes identifying one or more dominant speakers in a video conference. The method may also include generating a list of the one or more dominant speakers. Additionally, the method may include communicating the list of one or more dominant speakers to clients in a video conferencing system. In a further embodiment, the method includes communicating the list of the one or more dominant speakers to a client in response to the client joining the video conference.
In one embodiment, the method also includes identifying a change in the one or more dominant speakers in the video conference, generating an update to the list of one or more dominant speakers, and communicating the update to clients in a video conferencing system. The method may also include resending the list of dominant speakers to the clients periodically.
In one embodiment, the list of dominant speakers comprises a media source identifier associated with each of the identified dominant speakers. Additionally, the method may include automatically providing one or more data streams associated with each of the media source identifiers on the dominant speaker list.
In certain embodiments, computer-readable storage mediums may be configured with computer readable instructions that, when executed by a processor, cause the processor to perform operations for carrying out embodiments of the method described above. Additionally, embodiments of computer systems for carrying out such methods are also described. For example, a computer system is described having one or more processors, and one or more computer-readable storage media having stored thereon computer-executable instructions that, when executed by the one or more processors, causes the processors to perform operations including identifying one or more dominant speakers in a video conference, generating a list of the one or more dominant speakers, and communicating the list of one or more dominant speakers to clients in a video conferencing system.
In one embodiment, the computer system comprises a Multipoint Control Unit (MCU). The MCU may be configured to be in communication with a client in a video conferencing system.
To further clarify the above and other advantages and features of embodiments of the present invention, a more particular description of embodiments of the present invention will be rendered by reference to the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
Embodiments disclosed herein are directed to methods, systems, and software for dominant speaker identification in video conferencing. These systems and methods may be incorporated into a wide range of systems and solutions for video conferencing, including for example, dedicated video conferencing systems, web-based video conferencing systems, and application-driven video conferencing systems. Certain embodiments may be incorporated into applications with additional content, such as collaboration software, and may provide a significant commercial benefit over previous versions of such software because of an enhanced overall user experience.
In one embodiment, the media sources 102a-n may include video capture devices. The video capture devices may include, for example, a video camera, webcam, or other specialized video conferencing capture device. In certain embodiments, the video capture device may be coupled to a computer or other hardware suitable for running a video codec which may generate one or more data streams from the video captured by the video capture device. The media sources 102a-n may then each transmit the data streams through the network 104. In one embodiment, the media sources 102a-n may transmit the data streams to the MCU 106. Alternatively, the media sources 102a-n may transmit the data streams to the media requestors 108a-n upon receiving instructions from the MCU or a direct request from one of the media requestors 108a-n.
In one embodiment, the network 104 may include one or more network routing devices. For example, the network 104 may include one or more Internet routing devices configured to route traffic from the media sources 102a-n to, for example, the MCU 106.
The MCU 106 may be configured to route audio and/or video from the media sources 102a-n to the media requestors 108a-n. Because the MCU 106 handles routing of audio and video data streams, the MCU 106 is sometimes referred to as an Audio Video MCU (AVMCU). In one embodiment, the MCU 106 may be configured as a bridge to connect calls from multiple sources. Participants in a video conference may call the MCU 106, or alternatively the MCU 106 may call the participants once video conference configuration information has been set. The MCU 106 may use various communication protocols, such as IP, Voice Over IP (VOIP), or Plain Old Telephone Service (POTS) networks for communication of video and/or audio data streams. In one embodiment, the MCU 106 may be configured as a web-based server of a web-based video conferencing application. In another embodiment, the MCU 106 may operate in the background, participating only in the routing of data streams between the media sources 102a-n and media requestors 108a-n. The MCU may be configured in software, hardware, or a combination of the two.
A media requestor 108a-n may be a computing device configured to receive media data streams originating from the media sources 102a-n and render the data streams into displayed video. In one embodiment, a media requestor 108a-n may be a desktop computer. In another embodiment, the media requestor 108a-n may be a laptop, tablet, or mobile Personal Computer (PC). In still a further embodiment, the media requestor 108a-n may be a smartphone, Personal Data Assistant (PDA), mobile phone, or the like. One of ordinary skill in the art will recognize various embodiments of a media requestor 108a-n that may be adapted for use with the present embodiments.
The first media source 102a may include a codec configured to generate multiple layers of video including, but not limited to various Common Intermediate Format (CIF) layers, or high Definition (HD) layers. For example, a common codec may be configured to generate fifty (50) or more video layers, including but not limited to, SQCIF, QCIF, 4CIF, 16CIF, DCIF, HD 720p, HD 1080i, HD 1080p, and the like. One of ordinary skill in the art will recognize a variety of video layers that may be included in separate data streams. The video layers may include video of different frame rates, different resolution, and/or different color schemes. In addition, the data streams may include audio.
In the depicted embodiment, the first media source 102a sends four different media data streams to the MCU 106. Each media data stream includes a data stream identifier. For example, in the depicted embodiment the data stream identifier is a Synchronization Source (SSRC) identifier. Each data stream may include data associated with a different layer of media streaming from the first media source 102a.
In one embodiment, the MCU 106 may also receive requests for data streams from a first media requestor 108a and a second media requestor 108b. Due to hardware, or codec limitations, the first media requestor 108a and the second media requestor 108b may not be able to render the same quality of video. Thus, the first media requestor 108a and the second media requestor 108b may request different data streams from the MCU 106. In response to the request, the MCU 106 may send the data streams associated with SSRC1 and SSRC2 to the second media requestor 108b and the data stream associated with SSRC3 to the first media requestor. If neither the first media requestor 108a, nor the second media requestor 108b requests the fourth data stream, the MCU 106 may not pass the data stream associated with SSRC4 to either media requestor 108a-b. As described below, communications between the media requestors 108a-n and the MCU 106 may include information, such as MSIDs that may simplify routing of the data streams.
By way of example, Alice may be operating client 402a, generates video data streams of her in both CIF (SSRC1) and HD (SSRC2) video formats. Similarly, Bob may be associated with client 402b, which generates data streams for HD (SSRC3) video format. In such an embodiment, an MSID may be assigned to each client 402a-d. For example, the user name (Alice, Bob, Charles, and Dave) may be assigned as the MSID for each respective client 402a-d. Thus, MCU 106 may map data stream identifiers (SSRC1, SSRC2, and SSRC3) to the respective MSIDs.
In such an example, Charles may request video from Alice. If Charles' client 402c is only capable of rendering the CIF video layer, then Charles may only receive data steam SSRC 1. Thus, Charles may make a request to the MCU 106 for CIF layer video from MSID ‘Alice.’ In such an embodiment, MCU 106 may look up the SSRC associated with CIF video originating from Alice (SSRC1), and provide that data stream to Charles' client 402c.
Similarly, Dave's client 402d may request HD video from both Bob and Alice. In such an embodiment, Dave's client 402d may request MSID ‘Alice’ and MSID ‘Bob’ and specify the HD layer. In response, the MCU 106 may look up the SSRC associated with the HD video stream from both Alice and Bob and send both data streams (SSRC2 and SSRC3) to Dave's client 402d. Beneficially, in such an example each client only uses an MSID and its own capabilities to generate the request to the MCU. Thus, each client need not store and keep updated a list of each SSRC in use on the system 400.
The computer 500 typically includes a variety of computer-readable media 508. Computer-readable media 508 may be any available media that can be accessed by the computer 500 and includes both volatile and nonvolatile media, and removable and non-removable media, but excludes propagated signals. By way of example, and not limitation, computer-readable media 508 may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer 500. Communication media may include computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above may also be included within the scope of computer-readable media. Computer-readable media may be embodied as a computer program product, such as software stored on computer storage media.
The data storage or system memory 504 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) and random access memory (RAM). A basic input/output system (BIOS), containing the basic routines that help to transfer information between elements within computer 500, such as during start-up, is typically stored in ROM. RAM typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 502. By way of example, and not limitation, data storage 504 holds an operating system, application programs, and other program modules and program data.
Data storage 504 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, data storage 504 may be a hard disk drive that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive that reads from or writes to a removable, nonvolatile magnetic disk, and an optical disk drive that reads from or writes to a removable, nonvolatile optical disk such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The drives and their associated computer storage media, described above and illustrated in
A user may enter commands and information through a user interface 510 or other input devices such as a tablet, electronic digitizer, a microphone, keyboard, and/or pointing device, commonly referred to as mouse, trackball or touch pad. Other input devices may include a joystick, game pad, satellite dish, scanner, or the like. Additionally, voice inputs, gesture inputs using hands or fingers, or other natural user interface (NUI) may also be used with the appropriate input devices, such as a microphone, camera, tablet, touch pad, glove, or other sensor. These and other input devices are often connected to the processing unit 502 through a user input interface 510 that is coupled to the system bus 506, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 512 or other type of display device is also connected to the system bus 506 via an interface, such as a video interface. The monitor 512 may also be integrated with a touch-screen panel or the like. Note that the monitor and/or touch screen panel can be physically coupled to a housing in which the computing device 500 is incorporated, such as in a tablet-type personal computer. In addition, computers such as the computing device 500 may also include other peripheral output devices such as speakers and printer, which may be connected through an output peripheral interface or the like.
The computer 500 may operate in a networked or cloud-computing environment using logical connections 514 to one or more media devices, such as a media computer. The media computer may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 500. The logical connections depicted in
When used in a networked or cloud-computing environment, the computer 500 may be connected to a public or private network through a network interface or adapter 514. In some embodiments, a modem or other means for establishing communications over the network. The modem, which may be internal or external, may be connected to the system bus 506 via the network interface 514 or other appropriate mechanism. A wireless networking component such as comprising an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a network. In a networked environment, program modules depicted relative to the computer 500, or portions thereof, may be stored in the media memory storage device. It may be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
The described embodiments may be implemented in the context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by media processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or media computer storage media including memory storage devices.
In one embodiment, MCU 106 may be configured to include multiple modules or units. For example, MCU 106 may include a receiver unit 601, a MSID assignment unit 602, and a mapping unit 603. Further, MCU 106 may generate and store a map 604 for mapping MSIDs to SSRCs.
In one embodiment, the receiver 601 may utilize the network interface 514, to receive data streams from clients 302a-n. Additionally, the receiver 601 may receive requests for data streams from the clients 302a-n. The receiver may be in communication with the MSID assignment unit 602.
The MSID assignment unit 602 may be configured to assign an MSID to each client 302a-n. In one embodiment, the MSID assignment unit 602 may assign an MSID that is unique among the group of clients 302a-n such that request collisions are avoided. The MSID assignment unit 602 may assign one of a predetermined set of MSIDs. Alternatively, the MSID assignment unit 602 may assign a randomly generated MSID. In another embodiment, the MSID assignment unit 602 may assign an MSID in response to a user input, such as a name, telephone number, email address, user ID, or the like.
The mapping unit 603 may then generate map 604 to associate each data stream received from the clients 302a-n with a respective MSID associated with the client 302a-n. For example, if MSID assignment unit 602 assigns the MSID ‘Alice’ to client 302a, them mapping unit 603 may tag, group, or otherwise arrange the SSRCs associated with the data streams received from client 302a in association with the MSID ‘Alice.’ Mapping unit 603 may further update the map 604 with additional SSRCs as additional data streams become available from existing clients 302a-n, or as new clients 302a-n join the video conference.
In one embodiment, the map 604 may be stored in memory on the MCU 106. Alternatively, the map 604 may be stored on a data storage disk, such as a hard disk drive. The map 604 may be stored in database format. Alternatively, the map 604 may be stored as a hash table, an array of strings, an array of arrays, an array of pointers to MSIDs and/or SSRCs, or the like. One of ordinary skill in the art will recognize a variety of arrangements that may be suitable for mapping the MSIDs to SSRCs.
In one embodiment, the dominant speaker list generator 701 may identify a dominant speaker through one or more of multiple methods. For example, the dominant speaker list generator 701 may identify a dominant speaker by measuring the volume of audio signals received from each client 302a-n and identifying an audio signal with the highest volume. Alternatively, the dominant speaker list generator 701 may identify an audio signal with a greatest amount of activity within a period of time. In still a further embodiment, the dominant speaker list generator 701 may analyze video to determine a dominant speaker based upon body or lip motion. One of ordinary skill in the art may recognize alternative methods for identifying the dominant speaker.
Once the dominant speaker list generator 701 identifies the dominant speaker, it may add the current dominant speaker to a dominant speaker list. A dominant speaker list may include a file or table of dominant speakers. The dominant speaker list may include the MSID of the dominant speakers. In one embodiment, the dominant speaker list may include a history of the most recent dominant speakers. Alternatively the dominant speaker list may include a history of the most active dominant speakers. In still another embodiment, the dominant speaker list may include a list of the MSIDs that are most requested by the clients 302a-n. One of ordinary skill in the art may recognize additional criteria that may be used to generate the dominant speaker list.
The dominant speaker list may then be distributed to the clients 302a-n by sender 702. For example, the sender 702 may send an initial dominant speaker list to a new client 302 in the video conference upon identifying that the new client 302 has joined. In a further embodiment, the sender 702 may distribute updated dominant speaker lists to the clients 302a-n in response to an update to the list, or on a periodic basis.
It will be understood that in other embodiments, dominant or active participants may be selected using criteria other than audio participation. An active participant may be determined not only by the participant's input audio level or the frequency of input audio (i.e., a dominant speaker) but also by other non-audio inputs, such as changes in a video input, which may reflect movement of the participant, changes in the participant's location, or gestures by the participant. For example, a participant who does not speak or speaks infrequently may be designated as an active participant if he or she makes certain gestures, such as sign language or other signals, or moves by more than a threshold frequency or amount. Alternatively, an active participant on a video conference may be determined by activity on a non-audio and non-video channel, such as by identifying recent or concurrent email, text, messaging, document editing, document sharing, or other activity by the participant. For example, a participant who does not speak or speaks infrequently may be designated as an active participant if he or she sends email, texts, or other messages to other participants or shares or edits documents with other participants.
In still further embodiments, the receiver 601 may receive requests for data streams from the clients 302a-n. Each request may include, for example, a list of MSIDs, and identification of a video layer supported by the clients 302a-n. For example, referring back to the example in
The MBR encoder 902 may generate multiple data streams from the video captured by video capture device 901. For example, the MBR encoder may generate both an HD 720p data stream 903 and a CIF data stream 904. In addition, each of the HD data stream 903 and the CIF data stream 904 may further include one or more layers. In one embodiment, a distinct SSRC may be assigned to each layer in the data streams 903, 904.
Client 402a may send the data streams 903, 904 to the AVMCU 106. In response to requests from Bob and Charles, AVMCU 106 may pass the HD data stream 903 to Bob 402b and the CIF data stream 904 to Charles 402c. One of ordinary skill in the art will recognize that the present embodiment is merely for illustrative purposes, and that a wide variety of system configuration may be employed for video conferencing in accordance with the present embodiments.
Additionally, a side panel 1002 may be used for viewing a list of participants in the video conference. These participants generally would not be dominant speakers. In one embodiment, only a still-frame image of the participant is illustrated in panel 1002. Alternatively, only a name or MSID of the participant is displayed in the side panel 1002.
In addition, multiple video panels may be used for rendering a group of dominant speakers. For example, the dominant speaker list may specify that the most recent dominant speakers are Bob, Charles, Dave, Elliot, and Fabian. These dominant speakers may be displayed in windows 1003-1007 respectively. Since this is Alice's view, her video may also be displayed in window 1008. In a further embodiment, the view may include a method for identifying the currently active dominant speaker in the dominant speaker list. For example, if Elliot is currently speaking, Elliot's video window may be highlighted, enlarged, framed, or otherwise indicated.
In one embodiment, the method starts when Alice 402a (
Alice 402a may receive 1104 the dominant speaker list in turn, and request 1105 the data streams associated with the dominant speakers on the dominant speaker list. For example, the request may include the MSIDs of the dominant speakers as well as an indicator of the supported or requested media layers. In an alternative embodiment, Alice may automatically request the data streams associated with the clients on the dominant speaker list. In still a further embodiment, the MCU may automatically push the video streams associated with the clients on the dominant speaker list to Alice.
The MCU 106 may then receive 1106 the request from Alice 402a. In response, the mapping unit 603 may identify 1107 the SSRCs to send to Alice 402a in response to the request. Sender 702 may then send 1108 data stream(s) associated with the identified SSRCs to Alice 402a. In turn, Alice 402a may then receive 1109 the data stream(s) and render 1110 the data streams. For example, Alice 402a may render 1110 the data stream associated with video from Bob, Charles, Dave, Elliot, and Fabian in dominant speaker video windows (1003-1007 respectively).
In one embodiment, the dominant speaker list generator 701 (
The receiver 601 on the MCU 106 may then receive 1114 the request from Alice 402a and the identification unit 703 may identify 1115 the SSRCs associated with the requested MSIDs and layers. The sender 702 may then send 1116 the data streams associated with the identified SSRCs to Alice 402a.
Alice 402a may then receive 1117 the new data streams and render video associated with the new data streams in an updated view as illustrated in
Beneficially, such embodiments may provide greater user flexibility with regard to selection of videos for viewing, allow for frequent updating of dominant speaker videos, and avoid communication errors such as SSRC collisions and the like. In general, the present embodiments may provide a user of a video conferencing system a more robust and flexible participation experience.
In a computer-implemented method, such as a processor running instructions stored on a memory, a device identifies one or more active participants in a video conference. A list of the one or more active participants is generated. The list of one or more active participants is communicated to clients in a video conferencing system. The one or more active participants may comprise one or more dominant speakers in the video conference. The one or more dominant speakers in the video conference are identified during a selected interval or include a selected number of most-recent dominant speakers in the video conference.
The one or more active participants may comprise one or more dominant speakers in the video conference. The list may show an order of dominant speakers in descending order wherein a first participant in the list is a current dominant speaker, and a second participant in the list was a next previous dominant speaker.
The active participants in the video conference may be identified based upon one or more of an amount of audio interaction by participants, a frequency of audio interaction by participants, selected gestures by participants, an amount of movement by participants, a frequency of movement by participants, and/or activity by participants on a channel external to the video conference.
The list of one or more active participants that is communicated to a selected client in the video conference may include the selected client. The list of the one or more active participants may be communicated to a client in response to the client joining the video conference.
Changes in the one or more active participants may be identified in the video conference, and an update to the list of one or more active participants generated. The update to clients in the video conferencing system may be communicated to clients of the video conference system. Changes in one or more active participants occur when a participant on the list leaves the video conference. The list of dominant speakers may be resent to the clients periodically or upon receiving a request from a client.
In some embodiments, the list of active participants may include a media source identifier associated with each of the identified dominant speakers. One or more data streams associated with each of the media source identifiers on the active participants list may be automatically provided to clients.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
