Method and apparatus for video conferencing with audio redirection within a 360 degree view

FIELD OF THE INVENTION

[0002] This invention relates, generally, to video conference systems and, more specifically, to a technique for using a 360 degree cameras in video conferencing applications and sound localization techniques so that the remote video conference attendee can selectively see all or part of a conference room, including the active speaker.

BACKGROUND OF THE INVENTION

[0003] Recently, systems for enabling audio and/or video conferencing of multiple parties over packet-switched networks, such as the Internet, have become commercially available. Such systems typically allow participants to simultaneously receive and transmit audio and/or video data streams depending on the sophistication of the system. Conferencing systems used over packet-switched networks have the advantage of not generating long-distance telephone fees and enable varying levels of audio, video, and data integration into the conference forum. In a typical system, a conference server receives audio and/or video streams from the participating client processes to the conference, mixes the streams and retransmits the mixed stream to the participating client processes. Except for cameras, displays and video capture cards most video conferencing systems are implemented in software.

[0004] Existing video conferencing applications use standard video cameras that give a very narrow field of view to the remote people that are viewing the video conference. Typically, video conferencing vendors simply leave it up to the user to place the camera so that the remote video conference attendees can see as much of the action. This solution works fine for video conferences that are between individuals. If the video conferencing system is moved to a conference room, board room or class room, it becomes a problem to find a location in the room to place a standard video camera with only a single field of view so that the remote viewers can see anywhere in the room. A prior solution to this problem is to place the camera at one end of the room or in the corner of the room. With such approach, however, it is likely that images of the back of someone's head will be transmitted. Further, action at the end of the room opposite the camera is typically too small for remote viewers to discern.

[0005] Attempts have been made to provide a broader range of camera angles to a video teleconference. For example, U.S. Pat. No. 5,686,957, assigned to International Business Machines Corporation, discloses an automatic, voice-directional video camera image steering system that selects segmented images from a selected panoramic video scene, typically around a conference table, so that the active speaker will be the selected segmented image in the proper viewing aspect ratio, eliminating the need for manual camera movement or automated mechanical camera movement. The system includes an audio detection circuit from an array of microphones that can determine the direction of a particular speaker and provide directional signals to a video camera and lens system that electronically selects portions of that image so that each conference participant sees the same image of the active speaker.

[0006] However, in normal conversational style the image is likely to change at a rate which the viewer may find annoying. In addition, the system disclosed in U.S. Pat. No. 5,686,957 forces the viewer to always see the current speaker, without the ability to selectively view the rest of the conference environment.

[0007] In addition, with the advent of the Internet, and widespread use of protocols for real-time transmission of packetized video data, “virtual” video conferences are possible in which the participants exist at disparate locations during the conference.

[0008] Accordingly, a need exists for a video conferencing system that enables remote viewers to see all of the participants to a video conference and all the action in a video conferencing environment.

[0009] A further need exists for video conferencing system that enables a remote viewer to select a portion of the video conferencing environment as desired.

[0010] Another need exists for video conferencing system that enables each participant to independently select the entire field of view or a portion thereof, independent of the which speaker is talking.

[0011] Yet another need exists for video conferencing system that optionally uses sound localization to redirect the view of a video image during a “virtual” video conference.

SUMMARY OF THE INVENTION

[0012] The present invention automates the process of determining the current speaker in a virtual video teleconference by sending along an entire 360 degree view, data identifying a “suggested” portion of the 360 degree field of the current speaker. The present invention sends, to each conference participant, the azimuth direction in coordinates of the active speaker as determined by the sound detection technology at the source. Each participant can then independently choose to view: 1) the entire 360 degree video image; 2) the active speaker, as automatically suggested by the azimuth direction, or 3) a user selected portion of 360 degree video image. The invention permits true virtual conferences since the participants can decide for themselves what they want to see and not have it dictated by the technology or a camera operator, as in the prior art. Accordingly, the virtual video conferences are more like a real life meeting in which a participant gets audio clues the speaker, but can ignore such clues and focuses on something or someone else.

[0013] The video conference application of the present invention supports the use of both conventional and 360 degree cameras in virtual video conferences so that a complete 360 degree image may be transmitted to some or all of the conference participants, with the ability to view all or a part of the 360 degree image and to scroll through the image, as desired. At the recipient system, the video conference application senses whether an image is from a conventional or a 360 degree camera and adjusts the size of the viewing portal on the user interface accordingly. Viewers of 360 degree images are further provided with the option of viewing and scrolling the entire 360 degree image or only a portion thereof.

[0014] This invention enables merging of a video conferencing application with camera technology that is capable of capturing a 360 degree view around the camera, allowing a single camera to be placed in the middle of the room. Because the camera captures a full 360 degree field of view around the camera, everything in the room is visible to the remote video conference attendees. The video conferencing application of the present invention offers a remote video conference attendee various viewing techniques to see the room including a full room view displayed in a single window, thus allowing the user to see anything in the room at one time, and a smaller more traditional video window which appears to offer a standard camera narrow field of view but which is actually a view portal into the larger full room image. With such option, the viewer can scroll the view portal over the full room image simulating moving the camera around the room to view any desired location in the room. In addition, when the source of the image changes, i.e., the speaker changes for a 360 degree image to a conventional image, the user interface automatically adjusts the window size accordingly.

[0015] According to a first aspect of the invention, in a computer system capable of executing a video conferencing application having a user interface, a method comprises: (A) receiving a sequence of video data packets representing an entire 360 degree image; (B) receiving data identifying a portion of the 360 degree image associated with an active speaker; and (C) displaying a portion of the 360 degree image through the user interface. In one embodiment, (C) comprises displaying a portion of the 360 degree image identified as associated with the active speaker. In another embodiment, the method further comprises (D) receiving user defined selection indicia through the user interface indicating a portion of the 360 degree image to be viewed; and (C) further comprises displaying a portion of the 360 degree image identified by the user defined selection indicia.

[0016] According to a second aspect of the invention, a computer program product for use with a computer system capable of executing a video conferencing application with a user interface, the computer program product comprising a computer useable medium having embodied therein program code comprising (A) program code for receiving a sequence of video data packets representing an entire 360 degree image; (B) program code for receiving data identifying a portion of the 360 degree image associated with an active speaker; and (C) program code for displaying a portion of the 360 degree image through the user interface.

[0017] According to a third aspect of the invention, in a computer system capable of executing a video conferencing application with a user interface, a method comprises: (A) receiving a sequence of video data packets representing an entire 360 degree image; (B) receiving data identifying a portion of the 360 degree image recommended for display; and (C) displaying through the user interface the portion of the 360 degree image recommended for display.

[0018] According to a fourth aspect of the invention, a computer program product for use with a computer system capable of executing a video conferencing application with a user interface, the computer program product comprising a computer useable medium having embodied therein program code comprising (A) program code for receiving a sequence of video data packets representing an entire 360 degree image; (B) program code for receiving data identifying a portion of the 360 degree image recommended for display; and (C) program code for displaying through the user interface the portion of the 360 degree image recommended for display.

[0019] According to a fifth aspect of the invention, an apparatus for use with a computer system capable of executing a video conferencing application with a user interface, the apparatus comprising: (A) program logic for receiving a sequence of video data packets representing an entire 360 degree image; (B) program logic for receiving data identifying a portion of the 360 degree image recommended for display; and (C) program logic for displaying through the user interface the recommended portion of the 360 degree.

[0020] According to a sixth aspect of the invention, a system for displaying 360 degree images in a video conference comprises: (A) a source process executing on a computer system for generating sequence of video data packets representing an entire 360 degree image and data identifying a portion of the 360 degree image recommended for display; (B) a server process executing on a computer system for receiving the sequence of video data packets and recommendation data from the source process and for transmitting the sequence of video data packets and recommendation data to a plurality of receiving processes; and (C) a receiving process executing on a computer system and capable of displaying through a user interface the portion of the 360 degree image recommended for display.

[0021] According to a seventh aspect of the invention, in a computer system capable of executing a video conferencing application having a user interface, a method comprises: (A) receiving a sequence of video data packets representing an entire 360 degree image; (B) receiving data identifying a portion of the 360 degree image associated with an active speaker; (C) defining a viewing portal within the user interface for displaying a portion of the 360 degree image; and (D) displaying within the viewing portal the portion of the 360 degree image identified as associated with an active speaker. In one embodiment, the data identifying the portion of the 360 degree image associated with an active speaker comprises data coordinates defining a region within the 360 degree image and (D) comprises (D1) displaying within the viewing portal a portion of the region of the 360 degree image defined by the data coordinates. In another embodiment, the method further comprises:

[0022] (E) receiving user defined selection indicia through the user interface indicating the entire 360 degree image to be viewed; and (F) displaying the entire 360 degree image video through the user interface.

[0023] According to an eight aspect of the invention, a computer program product for use with a computer system capable of executing a video conferencing application with a user interface, the computer program product comprising a computer useable medium having embodied therein program code comprising: (A) program code for receiving a sequence of video data packets representing an entire 360 degree image; (B) program code for receiving data identifying a portion of the 360 degree image associated with an active speaker; (C) program code for defining a viewing portal within the user interface for displaying a portion of the 360 degree image; and (D) program code for displaying within the viewing portal the portion of the 360 degree image identified as associated with an active speaker.

[0024] According to an ninth aspect of the invention, in a computer system capable of executing a video conferencing application having a user interface, a method comprises: (A) receiving a sequence of video data packets representing an entire 360 degree image; (B) receiving data identifying a portion of the 360 degree image associated with an active speaker; and (C) displaying through the user interface one of: (i) the entire 360 degree image; (ii) the portion of the 360 degree image identified as associated with an active speaker; and (iii) a portion of the 360 degree image identified by user defined selection indicia received through the user interface.

[0025] According to an tenth aspect of the invention, a computer program product for use with a computer system capable of executing a video conferencing application with a user interface, the computer program product comprising a computer useable medium having embodied therein program code comprising: (A) program code for receiving a sequence of video data packets representing an entire 360 degree image; (B) program code for receiving data identifying a portion of the 360 degree image associated with an active speaker; and (C) program code for displaying through the user interface one of: (i) the entire 360 degree image; (ii) the portion of the 360 degree image identified as associated with an active speaker; and (iii) a portion of the 360 degree image identified by user defined selection indicia received through the user interface.

[0026] According to an eleventh aspect of the invention, an apparatus for use with a computer system capable of executing a video conferencing application with a user interface, the apparatus comprises: (A) program logic for receiving a sequence of video data packets representing an entire 360 degree image; (B) program logic for receiving data identifying a portion of the 360 degree image associated with an active speaker; and (C) program logic for displaying through the user interface one of: (i) the entire 360 degree image; (ii) the portion of the 360 degree image identified as associated with an active speaker; and (iii) a portion of the 360 degree image identified by user defined selection indicia received through the user interface.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and further advantages of the invention may be better understood by referring to the following description in conjunction with the accompanying drawings in which:

[0028]
FIG. 1 is a block diagram of a computer systems suitable for use with the present invention;

[0029]
FIG. 2 is a illustrates conceptually the relationship between the components of the system in which the present invention may be utilized;

[0030]
FIG. 3 is a block diagram conceptually illustrating the functional components of the multimedia conference server in accordance with the present invention;

[0031]
FIG. 4 is a illustrates conceptually a system for capturing and receiving video data;

[0032]
FIG. 5 is an illustration of a prior art RTP packet header;

[0033] FIGS. 6A-B form a flow chart illustrating the process steps performed during the present invention;

[0034]
FIG. 7 is screen capture of a user interface in which a complete 360 degree image is viewable in accordance with the present invention;

[0035]
FIG. 8 is screen capture of a user interface in which a portion of a 360 degree image is viewable in accordance with the present invention;

[0036]
FIG. 9 is a illustrates conceptually the placement of the microphone array in relation to a 360 degree camera; and

[0037]
FIG. 10 illustrates conceptually a microphone array and audio processing logic useful with the present invention.

DETAILED DESCRIPTION

[0038]
FIG. 1 illustrates the system architecture for a computer system 100, such as a Dell Dimension 8200, commercially available from Dell Computer, Dallas Tex., on which the invention can be implemented. The exemplary computer system of FIG. 1 is for descriptive purposes only. Although the description below may refer to terms commonly used in describing particular computer systems, the description and concepts equally apply to other systems, including systems having architectures dissimilar to FIG. 1.

[0039] The computer system 100 includes a central processing unit (CPU) 105, which may include a conventional microprocessor, a random access memory (RAM) 110 for temporary storage of information, and a read only memory (ROM) 115 for permanent storage of information. A memory controller 120 is provided for controlling system RAM 110. A bus controller 125 is provided for controlling bus 130, and an interrupt controller 135 is used for receiving and processing various interrupt signals from the other system components. Mass storage may be provided by diskette 142, CD ROM 147 or hard drive 152. Data and software may be exchanged with computer system 100 via removable media such as diskette 142 and CD ROM 147. Diskette 142 is insertable into diskette drive 141 which is, in turn, connected to bus 130 by a controller 140. Similarly, CD ROM 147 is insertable into CD ROM drive 146 which is connected to bus 130 by controller 145. Hard disk 152 is part of a fixed disk drive 151 which is connected to bus 130 by controller 150.

[0040] User input to computer system 100 may be provided by a number of devices. For example, a keyboard 156 and mouse 157 are connected to bus 130 by controller 155. An audio transducer 196, which may act as both a microphone and a speaker, is connected to bus 130 by audio/video controller 197, as illustrated. A camera or other video capture device 199 and microphone 192 are connected to bus 130 by audio/video controller 197, as illustrated. In the illustrative embodiment, video capture device 199 may be any conventional video camera or a 360 degree camera capable of capturing an entire 360 degree field of view.

[0041] It will be obvious to those reasonably skilled in the art that other input devices such as a pen and/or tablet and a microphone for voice input may be connected to computer system 100 through bus 130 and an appropriate controller/software. DMA controller 160 is provided for performing direct memory access to system RAM 110. A visual display is generated by video controller 165 which controls video display 170. In the illustrative embodiment, the user interface of a computer system may comprise a video display and any accompanying graphic use interface presented thereon by an application or the operating system, in addition to or in combination with any keyboard, pointing device, joystick, voice recognition system, speakers, microphone or any other mechanism through which the user may interact with the computer system. Computer system 100 also includes a communications adapter 190 which allows the system to be interconnected to a local area network (LAN) or a wide area network (WAN), schematically illustrated by bus 191 and network 195.

[0042] Computer system 100 is generally controlled and coordinated by operating system software, such as the WINDOWS NT, WINDOWS XP or WINDOWS 2000 operating system, available from Microsoft Corporation, Redmond Wash. The operating system controls allocation of system resources and performs tasks such as process scheduling, memory management, and networking and I/O services, among other things. In particular, an operating system resident in system memory and running on CPU 105 coordinates the operation of the other elements of computer system 100. The present invention may be implemented with any number of commercially available operating systems including OS/2, AIX, UNIX and LINUX, DOS, etc. One or more applications 220 such as Lotus Notes or Lotus Sametime, both commercially available from Lotus Development Corp., Cambridge, Mass. may execute under control of the operating system. If operating system 210 is a true multitasking operating system, multiple applications may execute simultaneously.

[0043] In the illustrative embodiment, the present invention may be implemented using object-oriented technology and an operating system which supports execution of object-oriented programs. For example, the inventive control program module may be implemented using the C++ language or as well as other object-oriented standards, including the COM specification and OLE 2.0 specification for MicroSoft Corporation, Redmond, Wash., or, the Java programming environment from Sun Microsystems, Redwood, Calif.

[0044] In the illustrative embodiment, the elements of the system are implemented in the C++ programming language using object-oriented programming techniques. C++ is a compiled language, that is, programs are written in a human-readable script and this script is then provided to another program called a compiler which generates a machine-readable numeric code that can be loaded into, and directly executed by, a computer. As described below, the C++ language has certain characteristics which allow a software developer to easily use programs written by others while still providing a great deal of control over the reuse of programs to prevent their destruction or improper use. The C++ language is well-known and many articles and texts are available which describe the language in detail. In addition, C++ compilers are commercially available from several vendors including Borland International, Inc. and Microsoft Corporation. Accordingly, for reasons of clarity, the details of the C++ language and the operation of the C++ compiler will not be discussed further in detail herein.

[0045] Video Compression Standards

[0046] When sound and video images are captured by computer peripherals and are encoded and transferred into computer memory, the size (in number of bytes) for one seconds worth of audio or a single video image can be quite large. Considering that a conference is much longer than 1 second and that video is really made up of multiple images per second, the amount of multimedia data that needs to be transmitted between conference participants is quite staggering. To reduce the amount of data that that needs to flow between participants over existing non-dedicated network connections, the multimedia data can be compressed before it is transmitted and then decompressed by the receiver before it is rendered for the user. To promote interoperability, several standards have been developed for encoding and compressing multimedia data.

[0047] H.263 is a video compression standard which is optimized for low bitrates (<64 k bits per second) and relatively low motion (someone talking). Although the H.263 standard supports several sizes of video images, the illustrative embodiment uses the size known as QCIF. This size is defined as 176 by 144 pixels per image. A QCIF-sized video image before it is processed by the H.263 compression standard is 38016 bytes in size. One seconds worth of full motion video, at thirty images per second, is 1,140,480 bytes of data. In order to compress this huge amount of data into a size of about 64 k bits, the compression algorithm utilizes the steps of: i) Differential Imaging; ii) Motion estimation/compensation; iii) Discrete Cosine Transform (DCT) Encoding; iv) Quantization and v) Entropy encoding.

[0048] The first step in reducing the amount of data that is needed to represent a video image is Differential Imaging, that is, to subtract the previously transmitted image from the current image so that only the difference between the images is encoded. This means that areas of the image that do not change, for example the background, are not encoded. This type of image is referred to as a “D” frame. Because each “D” frame depends on the previous frame, it is common practice to periodically encode complete images so that the decoder can recover from “D” frames that may have been lost in transmission or to provide a complete starting point when video is first transmitted. These much larger complete images are called “I” frames. Typically, human beings perceive 30 frames per second as real motion video, however, this can drop as low as 10-15 per second to still be perceptible as video images. The H.263 codec is a bitrate managed codec, meaning the number of bits that are utilized to compress a video frame into an I-frame is different than the number of bits that are used to compress each D-frame. Compressing only the visual changes between the delta frame and the previously compressed frame makes a delta frame. As the encoder compresses frames into either the I-frame or D-frame, the encoder may skip video frames as needed to maintain the video bitrate below the set bitrate target.

[0049] The next step in reducing the amount of data that is needed to represent a video image is Motion estimation/compensation. The amount of data that is needed to represent a video image is further reduced by attempting to locate where areas of the previous image have moved to in the current image. This process is called motion estimation/compensation and reduces the amount of data that is encoded for the current image by moving blocks (16×16 pixels) from the previously encoded image into the correct position in the current image.

[0050] The next step in reducing the amount of data that is needed to represent a video image is Discrete Cosine Transform (DCT) Encoding. Each block of the image that must be encoded because it was not eliminated by either the differential images or the motions estimation/compensation steps is encoded using Discrete Cosine Transforms (DCT). These DCT are very good at compressing the data in the block into a small number of coefficients. This means that only a few DCT coefficients are required to recreate a recognizable copy of the block.

[0051] The next step in reducing the amount of data that is needed to represent a video image is Quantization. For a typical block of pixels, most of the coefficients produced by DCT encoding are close to zero. The quantizer step reduces the precision of each coefficient so that the coefficients near zero are set to zero leaving only a few significant nonzero coefficients.

[0052] The next step in reducing the amount of data that is needed to represent a video image is Entropy encoding. The last step is to use an entropy encoder (such as a Huffman encoder) to replace frequently occurring values with short binary codes and replaces infrequently occurring values with longer binary codes. This entropy encoding scheme is used to compress the remaining DCT coefficients into the actual data that that represents the current image. Further details regarding the H.263 compression standard can be obtained from the ITU-T H.263 available from the International Telecommunications Union, Geneva, Switzerland.

[0053] The H.263 compression standard is typically used for video data images of standard size. The ITU-T H.263+ video compression standard is utilized to encode and decode nonstandard video image sizes such as those generated by 360 degree cameras.

[0054] Sametime Environment

[0055] The illustrative embodiment of the present invention is described in the context of the Sametime family of real-time collaboration software products, commercially available from Lotus Development Corporation, Cambridge, Mass. The Sametime family of products provide awareness, conversation, and data sharing capabilities, the three foundations of real-time collaboration. Awareness is the ability of a client process, e.g. a member of a team, to know when other client processes, e.g. other team members, are online. Conversations are networked between client processes and may occur using multiple formats including instant text messaging, audio and video involving multiple client processes. Data sharing is the ability of client processes to share documents or applications, typically in the form of objects. The Sametime environment is an architecture that consists of Java based clients that interact with a Sametime server. The Sametime clients are built to interface with the Sametime Client Application Programming Interface, published by International Business machines corporation, Lotus Division, which provides the services necessary to support these clients and any user developed clients with the ability to setup conferences, capture, transmit and render audio and video in addition to interfacing with the other technologies of Sametime.

[0056] The present invention may be implemented as an all software module in the Multimedia Service extensions to the existing family of Sametime 1.0 or 1.5 products and thereafter. Such Multimedia Service extensions are included in the Sametime Server 300, the Sametime Connect client 310 and Sametime Meeting Room Client (MRC) 312.

[0057]
FIG. 2 illustrates a network environment in which the invention may be practiced, such environment being for exemplary purposes only and not to be considered limiting. Specifically, a packet-switched data network 200 comprises a Sametime server 300, a plurality of Meeting Program Client (MRC) client processes 312A-B, a Broadcast Client (BC) client 314, an H.323 client process 316, a Sametime Connect client 310 and an Internet network topology 250, illustrated conceptually as a cloud. One or more of the elements coupled to network topology 250 may be connected directly or through Internet service providers, such as America On Line, Microsoft Network, Compuserve, etc.

[0058] The Sametime MRC 312, may be implemented as a thin mostly Java client that provides users with the ability to source/render real-time audio/video, share applications/whiteboards and send/receive instant messages in person to person conferences or multi-person conferences. The Sametime BC 314 is used as a “receive only” client for receiving audio/video and shared application/whiteboard data that is sourced from the MRC client 312. Unlike the MRC client, the BC client does not source audio/video or share applications. Both the MRC and BC clients run under a web browser and are downloaded and cached as need when the user enters a scheduled Sametime audio/video enabled meeting, as explained hereinafter in greater detail.

[0059] The client processes 310, 312, 314, and 316 may likewise be implemented as part of an all software application that run on a computer system similar to that described with reference to FIG. 1, or other architecture whether implemented as a personal computer or other data processing system. In the computer system on which a Sametime client process is executing, a sound/video card, such as card 197 accompanying the computer system 100 of FIG. 1, may be an MCI compliant sound card while a communication controller, such as controller 190 of FIG. 1, may be implemented through either an analog digital or cable modem or a LAN-based TCP/IP network connector to enable Internet/intranet connectivity.

[0060] Server 300 may be implemented as part of an all software application which executes on a computer architecture similar to that described with reference to FIG. 1. Server 300 may interface with Internet 250 over a dedicated connection, such as a T1, T2, or T3 connection. The Sametime server is responsible for providing interoperability between the Meeting Room Client and H.323 endpoints. Both Sametime and H.323 endpoints utilize the same media stream protocol and content differing in the way they handle the connection to server 300 and setup of the call. The Sametime Server 300 supports the T.120 conferencing protocol standard, published by the ITU, and is also compatible with third-party client H.323 compliant applications like Microsoft's NetMeeting and Intel's ProShare. The Sametime Server 300 and Sametime Clients work seamlessly with commercially available browsers, such as NetScape Navigator version 4.5 and above, commercially available from America On-line, Reston, Va.; Microsoft Internet Explorer version 4.01 service pack 2 and above, commercially available from Microsoft Corporation, Redmond, Wash. or with Lotus Notes, commercially available from Lotus Development Corporation, Cambridge, Mass.

[0061]
FIG. 3 illustrates conceptually a block diagram of a Sametime server 300 and MRC Client 312, BC Client 314 and an H.323 client 316. As illustrated, both MRC Client 312 and MMP 304 include audio and video engines, including the respective audio and video codecs. The present invention effects the video stream forwarded from a client to MMP 304 of server 300.

[0062] In the illustrative embodiment, the MRC and BC component of Sametime environment may be implemented using object-oriented technology. Specifically, the MRC and BC may be written to contain program code which creates the objects, including appropriate attributes and methods, which are necessary to perform the processes described herein and interact with the Sametime server 300 in the manner described herein. Specifically, the Sametime clients includes a video engine which is capable of capturing video data, compressing the video data, transmitting the packetized audio data to the server 300, receiving packetized video data, decompressing the video data, and playback of the video data. Further, the Sametime MRC client includes an audio engine which is capable of detecting silence, capturing audio data, compressing the audio data, transmitting the packetized audio data to the server 300, receiving and decompressing one or more streams of packetized audio data, mixing multiple streams of audio data, and playback of the audio data. Sametime clients which are capable of receiving multiple audio streams also perform mixing of the data payload locally within the client audio engine using any number of known algorithms for mixing of multiple audio streams prior to playback thereof. The codecs used within the Sametime clients for audio and video may be any of those described herein or other available codecs.

[0063] The Sametime MRC communicates with the MMCU 302 for data, audio control, and video control, the client has a single connection to the Sametime Server 300. During the initial connection, the MMCU 302 informs the Sametime MRC client of the various attributes associated with a meeting. The MMCU 302 informs the client process which codecs to use for a meeting as well as any parameters necessary to control the codecs, for example the associated frame and bit rate for video and the threshold for processor usage, as explained in detail hereinafter. Additional information regarding the construction and functionality of server 300 and the Sametime clients 312 and 314 can be found in the previously-referenced co-pending applications.

[0064] It is within this framework that an illustrative embodiment of the present invention is being described, it being understood, however, that such environment is not meant to limit the scope of the invention or its applicability to other environments. Any system in which video data is captured and presented by a video encoder can utilize the inventive concepts described herein.

[0065] 360 Degree Video Conferencing

[0066] Referring to FIG. 4, video images are captured with camera 350, which in the illustrative embodiment may include either a traditional video camera or a 360 degree camera at the video conference participant's location. A 360 degree camera suitable for use with the present invention may be the TotalView High Res package, commercially available from BeHere Corporation, Cupertino, Calif., 95014, which includes a DVC MegaPixel Video Camera, and a PCI Video Capture Board. The DVC MegaPixel Video Camera includes a conical lense which generates a spherical image. The spherical image is processed with the PCI Video Capture Board to dewarp the video data, allowing the three-dimensional image to be converted to a two-dimensional image and stored in a video buffer therein. The two-dimensional image supplied by the PCI Video Capture Board is approximately 768×192 pixels, e.g., a long, thin two-dimensional image.

[0067]
FIG. 4 illustrates conceptually the components of the inventive system utilized to generate and process a video data stream in accordance with the present invention. As described previously, the video conferencing application 357 may be implemented with the Sametime 2.0. The operating system 362 may be implemented with any of the Windows operating system products including WINDOWS 95, WINDOW 98, WINDOWS 2000, WINDOWS XP, etc. As such either a conventional camera or the 360 degree camera described above will be considered by the operating system as a Video for Windows device. Upon initial configuration of the video conferencing application 357 the user specifies whether the video capture device is a conventional camera of a 360 degree camera.

[0068] Camera 350 captures a continual stream of video data and stores the data in a video buffer in the accompanying video processing card where the three-dimensional image is processed to dewarp the image and convert the processed three-dimensional image into a two-dimensional image. The device driver 360 for camera 350 periodically transfers the image data from the camera/card to the frame buffer 352 associated with the device driver 360. An interrupt generated by the video conferencing application 357 requests a frame from the frame buffer 352. Prior to the providing the frame of captured video data to video encoder 356, control program 358 may optionally modify the size of the image prior to transmission of the frame 354 to video encoder 356. For example, in the illustrative embodiment, the viewing window or portal presented by the user interface 365 of video conferencing application 357 is capable of displaying an image that is approximately 144 pixels in height. Accordingly, the image in buffer 352 may be cropped to 768×144 pixels. To crop the buffered image, control program 358 allocates a second video buffer 353, that may be smaller e.g., 768×144, and extracts the image data of interest from buffer 352 and writes the image data into buffer 353. Control program 358 then specifies the size of the image to be compressed in pixels to video encoder 356 prior to compression thereof. Accordingly, the video image to be compressed may have some the top most and bottom most pixel lines eliminated.

[0069] Thereafter, the video image from buffer 353 is provided to video encoder 356 for compression of the video data in accordance with the published H.263+ specification. Control program 358 indicates to video encoder 356 when the video data supplied to the encoder 356 is of a custom picture format based on the value of the image size supplied to video encoder 356. When a video frame is compressed with video encoder 356 using the H.263+ standard, a header is associated with the compressed data, the header indicating the size of the compressed video image. Specifically, a fixed length code word of 23 bits, referred to as the Custom Picture Format (CPFMT) field, is present only in the header if the use of a custom picture format is signaled in the PLUSPTYPE field of the H.263 header and the UFEP field of the H.263 header has a value of ‘001’. When present, the CPFMT field has the following format:

[0070] Bits 1-4 Pixel Aspect Ratio Code: A 4-bit index to the PAR value in

[0071] Table 5 of the H.263+ Specification. For extended PAR, the exact pixel aspect ratio shall be specified in EPAR value in Table 5.16 of the H.263+ Specification;

[0072] Bits 5-13 Picture Width Indication (PWI): Range [0, . . . , 511];

[0073] Number of pixels per line=(PWI+1)*4;

[0074] Bit 14 Equal to “1” to prevent start code emulation;

[0075] Bits 15-23 Picture Height Indication (PHI): Range [1, . . . , 288]; Number of lines=PHI*4.

[0076] The compressed output from video encoder 356, including the video data and the header, are provided to RTP protocol module 367 which places a wrapper around the compressed video data in accordance with the Real Time Transport (RTP) protocol. Code within RTP protocol module 367 sets two fields in the RTP header when a single video image is broken up into multiple packets for transport over a network. Within the RTP header, as illustrated in prior art FIG. 5, the fields of interest are the Marker bit (M) and the Sequence Number. The Marker bit (M) of the RTP fixed header is set to 1 when the current packet carries the end of current frame, otherwise the Marker bit is set to 0. The Marker bit is intended to allow significant events such as frame boundaries to be marked in the packet stream. The value of the Sequence Number field (16 bits) increments by one for each RTP data packet sent, and may be used by the receiving video conferencing process to detect packet loss and to restore packet sequence. The initial value of the sequence number may be random, e.g. unpredictable, to make known-plain text attacks on encryption more difficult. Additional information regarding the RTP and H.263 protocols can be found in the ITU RFC 1889 Realtime Transport Protocol; ITU RFC 2190 RTP Payload Format for H.263 Video Streams; and ITU H.263 Video coding for low bit rate communication, all publicly available from the International Telecommunications Union, Geneva, Switzerland.

[0077] Following compression and packetizing of the image, the image is transmitted as a series of packets 390A-N to one or more recipient participants to the video conference. The packets 390A-N are transmitted from the source video conferencing system on which application 357 is executing through the network 250 to one or more receiving systems on which video conferencing application 357 is executing. In the illustrative embodiment, described with reference to the Sametime environment, the packetized data will be sent from the source video conferencing process, to a Sametime server, such as server 300 described previously but not shown in FIG. 4, and subsequently transmitted to the receiving video conferencing processes.

[0078] Referring to FIGS. 6A-B, the process performed by control program 358 during the reception decompression and presentation of video data is illustrated. Following receipt of the sequence of packets comprising the image, the previously described process is reversed. Using the Sequence Number field to put the packets back in order and to make a determination as to where a video frame or a single video image starts and ends by examining the marker bit, RTP protocol module 367 arranges the sequence of packets into order and supplies them to video decoder 366. Control program 358 places a procedure call to video decoder 366 which returns a pointer value, indicating the location of the decompressed data, and a size value, indicating the size of the decompresses data, as illustrated by step 600. Based on a size value, a buffer of the appropriate size is allocated by control program 358 and the decompressed video data output from decoder 366 is written into video buffer 375. If the size value supplied by video decoder 366 indicates a 360 degree image, a buffer of appropriate size will be allocated, as illustrated by steps 602 and 604, a scrolling function is enabled within control program 358, as illustrated by 606. If the size value supplied by video decoder 366 indicates a conventional video image, a buffer 385 of appropriate size will be allocated and the image will be provided to the user interface module 380 of application 357 for presentation to the viewer, as illustrated in steps 602, 603 and 605.

[0079] Thereafter, if the image is a 360 degree image, control program 358 determines the mode in which the viewer wishes to receive the 360 degree image, as illustrated by decisional step 608. Such determination may be made by default or through receipt of command indicia through user interface 380. The video conferencing application 357 of the present invention provides multiple options for viewing a 360 degree image. Since the extended video image resides in the local video buffer of a viewer participant's system, the user may select, through the user interface, to view the entire image or a portion thereof through a viewing portal. If the user desires to view the entire image, the complete contents of the video buffer will be displayed within the viewing portal on the graphic user interface, as illustrated in step 612. If the viewer indicates that less than all of the entire 360 degree image is to be viewed, an initial portion of the video buffer data, representing, for example, the center portion of the 360 degree image will be presented within a viewing portal, as illustrated in step 610.

[0080] In the illustrative embodiment, the entire 360 degree image, approximately 768×144 pixels, may be presented through the viewing portal 700 which may “float” anywhere on the user interface of the video conferencing application 357, as illustrated in FIG. 7, or alternatively may have a default or “docked” position on the user interface. Alternatively, the user may choose to view less than all of the 360 degree image at a single instance, in which case the user interface will display a conventional or reduced size viewing portal 800, such as approximately 176×144 pixels, as illustrated in FIG. 8. As with viewing portal 700, viewing portal 800 may float or be docked on the user interface.

[0081] Thereafter, if the image is a 360 degree image, the user may selectively control the portions of the extended image presented through the user interface. In the illustrative embodiment, movement of a pointing device cursor within the viewing portal 800 or 900, converts the cursor to directional cursor. Thereafter, movement of the cursor in one of the designated directions, e.g., left, right, up, or down, causes the viewing portal, whether 176×144 pixels or 768×144 pixels, will be detected by control program 358 an cause the next frame displayed to scroll in the designated direction to allow for selective viewing of different portions of the 360 degree image, as illustrated by steps 614 and 616. Continuous scrolling of the image may cause the image to “wrap around” to provide a continuously viewable 360 degree image. In this manner, as the viewing portal is moved in the direction of movement of the pointing device cursor, the portion of the 360 degree image is displayed within the viewing portal scrolls continuously. This process continues until the transmission from the source is terminated, as illustrated by steps 618 and 620, or until the next set of received data packets indicates a different source, as illustrated by steps 618 and 600.

[0082] In accordance with another aspect of the present invention, the video conferencing application 357 automatically adjusts the dimensions of the viewing portal on the user interface in accordance with the size of the currently received video data. As the source of the video data changes, i.e., the speaker changes to a different location/system, control program 358 detects the size of the video image and automatically adjusts the size of the viewing portal presented by the user interface. If in steps 600 and 602, the size of the image reported by the video decoder indicated that the image is of a conventional size, the dimensions of the viewing portal on the user interface will be resized for a conventional video image and the scrolling function of control program 358 will be disabled, if the image previously displayed was a 360 degree image. In this manner, in a video conference having multiple participants where one participant is utilizing a conventional video camera and another participant is utilizing a 360 degree camera, the video conferencing application 357 will automatically adjust the initial dimensions of the viewing portal on the user interface without further commands from the viewer. The reader will appreciate that the present invention provides a technique in which a complete 360 degree image is transmitted from a source to some or all of the participants to a virtual video conference, with the ability for the recipient participants to view all or a part of the 360 degree image and to scroll through the image, as desired.

[0083] Although the invention has been described with reference to the H.263 and H.263+ video codecs, it will be obvious to those skilled in the arts that other video encoding standards, such as H.261 may be equivalently substituted and still benefit from the invention described herein. In addition, the present invention may be used with a general purpose processor, such as a microprocessor based CPU in a personal computer, PDA or other device or with a system having a special purpose video or graphics processor which is dedicated to processing video and/or graphic data.

[0084] Audio Localization and Redirection

[0085] In the inventive video conferencing application described previously, the entire 360 degree image is sent to all participants, not just a portion of the entire 360 degree image. This feature allows each participant to decide independently of the other participants what portion of the entire field of image to view. For instance, a participant may scroll their view of the to the active speaker, or, alternatively, may choose to focus on the clock on the wall or perhaps the slides being presented within the image of the room. However, if they wish to scroll their view to the active speaker, the participant will need to determine who is the active speaker and where the active speaker is located in the room. This can be accomplished by either scrolling the field of view, e.g. the viewing portal on the user interface, until the active speaker is located, or, develop a mental image of the position and voice of each participant in the room, and, when a voice is recognized, scroll the view to the active speaker. Neither technique is completely practical, if the active speaker changes frequently.

[0086] The present invention provides a technique in which the process of detecting the active speaker is automated by sending along with the entire 360 degree view, a “suggested” portion of the 360 degree field of view in the form of azimuth direction coordinate information. Such azimuth direction coordinate information is determined by the sound detection technology on the sending end to each conference participant. This extra azimuth direction coordinate information is sent to each participant in the conference just like the entire 360 degree video image. Each participant then, can independently and automatically choose to view the active speaker as suggested by the azimuth direction, or, can ignore the suggested azimuth direction and choose a view of something else in the 360 degree video image. Each participant can independently choose to use or ignore the suggested field of view which shows the active speaker. Referring to FIGS. 9-10, in addition to the elements of the source system illustrated in

[0087]
FIG. 4, the present invention may further comprises a microphone array and audio processing logic and an audio processing application 398. The primary purpose of the microphone array 390 is to detect from which angular segment the audio signal is received. The audio signal from a particular participant will then be the basis for generating the coordinates within the 360 video image of camera 350, as described hereinafter. Microphone array 390 may comprise four or more directional microphones spaced apart and arranged to form an array concentrically about the camera 350 on a surface, typically a conference room table, so that all of the participants in the conference will have audio access to the microphones for transmission of sound. The microphones comprising the array 390 are positioned in fixed relations to each other, depending on the number of microphones. In configuring the source system, the array 390 and the camera 350 are synchronized to have corresponding directional orientation. For example, microphones 400, 402, 404 and 406 may be placed at 90, 180, 270 and 360 degrees within the 360 perspective of camera 350, i.e. every 90 degrees. If eight microphones are utilized within array 390, the microphones may be placed at 45, 90, 135, 180, 225, 270, 315 and 360 degrees within the 360 degree perspective of camera 350, i.e. every 45 degrees. The audio signals generated from microphones 400, 402, 404 and 406 are connected to stereo audio cards 410, 412, 414 and 416, respectively, in the source system. Each of the stereo audio cards may devote two channels to each microphone. Alternatively, a multiple channel audio card, such as the Santa Cruz 6 Channel DSP Audio Accelerator, commercially available from Voyetra Turtle Beach, Inc., Yonkers, N.Y. 10701, may be used instead of individual audio cards.

[0088] In the illustrative embodiment, each microphone input signal is sampled by and an analog to digital converter on its respective audio card. The audio processing application 398 executes within the source system and detects from the plurality of samples generated by audio cards 410, 412, 414 and 416 which microphone is receiving the strongest amplitude signal, the second strongest amplitude signal, the third strongest amplitude signal, etc. Using this information, application 398 uses a triangulation algorithm to determine at which of microphones 400, 402, 404 and 406 the speaker is located. In the illustrative embodiment, the greater the number of microphones within the microphone array, the more accurate the localization algorithm will become. Prior art microphone arrays and the theory of determining the direction of the source of acoustical waves from an array of microphones is known. U.S. Pat. No. 5,206,721 discloses audio source detection circuitry. Additional discussion on the these concepts can be found in Array Signal Processing: Concepts and Techniques, authored by Don H. Johnson and Dan E. Dudgeon, Chapter 4, Beamforming, published by PTR Prentice-Hall, 1993, and Multidimensional Digital Signal Processing, authored by Dan E. Dudgeon and Russell M. Mersereau, Chapter 6, Processing Signals Carried by Propagating Waves, published by Prentice-Hall, Inc., 1984.

[0089] The Windows operating system includes an audio API that views each microphone as a wave device. The wave audio device driver on each audio card utilizes WaveOpen commands to the operating system to capture and sample audio signals from each of the microphones in array 390. Each of audio cards 410, 412, 414 and 416 provides amplitude data to audio processing application 398, which then determines which of the microphones is receiving the strongest signal from the speaker. The audio processing application 398 the generates an identifier used to identify which microphone is active. Such identifier is supplied to the audio engine within the Sametime client executing on the source the system. The audio signal from the active microphone is then sampled, buffered and supplied to an audio compression algorithm within the Sametime client executing on the source system. The Sametime client may utilize either the G.723 or G.711 audio compression standard implemented within the audio engine to compress the audio data. Note that while the audio signal from the active microphone is being sampled and compressed, the audio processing application 398 continues to determine which microphone has received the greatest amplitude signal, so that when the current speaker is finished, the microphone closest to the next speaker may be identified with little delay.

[0090] Based on the position of the active microphone within the array 309, the audio processing application 398 determines approximately which angular segment within the 360 degree spectrum of the room the audio source is positioned. Audio processing application 398 then generates an x-y coordinate pair identifying where in the 360 degree image the current speaker is located. Data representing the x-y coordinate pair and the compressed output from the audio encoder, including the audio data and the header, are provided to RTP protocol module 367 which places a wrapper around the compressed audio data in accordance with the Real Time Transport (RTP) protocol. The x-y coordinate data may be imbedded in the header of an actual audio packet and transmitted to the Sametime client recipients in the teleconference. Alternatively, the x-y coordinate pair data may be transmitted as part of a user packet if the RTCP (Real Time Control Protocol) is utilized.

[0091] Each of the RTP and RTCP protocols include algorithms for mapping the time stamps included with packets of audio data and video data to ensure that playback of the audio is synchronized with playback of the corresponding video. At the Sametime client executing on the receiving system, control program 358 extracts the x-y coordinate data from either the audio packet header or the RTCP user packet and provides the coordinate data to the rendering engine within the Sametime client along with the corresponding audio and video data.

[0092] In order to utilize the transmitted coordinate data, the recipient user must enable the tracking function within the rendering engine of the Sametime client which utilizes the coordinate data. Such enablement may occur via a graphic control, menus command or dialog box on the user interface of the Sametime client, or through specification of the appropriate parameter during configuration of the Sametime client on the receiving system. If the user interface is currently presenting data within a defined view port, as described with reference to FIGS. 6-8, and the tracking functionality is selected via the user interface, the coordinates will be provided to the scrolling algorithm within the rendering engine which will then cause the appropriate portion of the buffered 360 degree image to be rendered within the viewing portal.

[0093] The video conferencing application 357 of the present invention provides multiple options for viewing a 360 degree image. Since the extended video image resides in the local video buffer of a viewer participant's system, the user may select, through the user interface, to view the entire image or a portion thereof through a viewing portal. Referring again to FIG. 6, if the user desires to view the entire image, the complete contents of the video buffer will be displayed within the viewing portal on the graphic user interface, as illustrated in step 612. If the viewer indicates that less than all of the entire 360 degree image is to be viewed, and the tracking function has been enabled, the portion of the video buffer identified by the x-y coordinate data will be presented within a viewing portal, as illustrated in step 610. Thereafter, as the x-y coordinate data changes the portion of the video buffer identified by the newer x-y coordinate data will be presented within a viewing portal, as illustrated in step 610. Accordingly, while the tracking function is enabled, the 360 degree image will automatically scroll to the portion of the 360 degree image containing the active speaker. For example, if a participant positioned at the approximately 90 degree location of the 360 degree image is speaking, the view port will scroll to the approximately 90 degree portion of the 360 degree image. Thereafter, if a participant positioned at the approximately 270 degree location of the 360 degree image is speaking, the view port will scroll to the approximately 270 degree portion of the 360 degree image, etc. Note that if the tracking functionality has not been selected by the user, the x-y coordinate data will be discarded or ignored.

[0094] Using the present invention, a viewer recipient may initially choose to view the entire 360 degree image of the speakers at the source system. Thereafter, viewer recipient may choose to view lees than the entire 360 degree image, and may manually redirect the viewing portal as desired. Thereafter, viewer recipient may choose to enable the tracking function associated with the viewing portal, allowing the viewing portal to be redirected automatically to track whoever is speaking at the source system. Thereafter, the source of the image data may change to a participant that does not have a 360 degree camera and the image will default back to a static viewing portal.

[0095] Accordingly, the reader will appreciate that the subject application discloses a novel system which transmits all of a 360 image a viewer/recipient to a virtual teleconference and allows the viewer/recipient to: i) view the entire 360 degree image simultaneously; ii) view a user selected portion of the 360 degree image via a manually scrollable viewing portal; or iii) view a portion of the 360 degree image via an automatically redirected viewing portal which always displays the current speaker; or to switch among any of the forgoing options as desired.

[0096] A software implementation of the above-described embodiments may comprise a series of computer instructions either fixed on a tangible medium, such as a computer readable media, e.g. diskette 142, CD-ROM 147, ROM 115, or fixed disk 152 of FIG. 1A, or transmittable to a computer system, via a modem or other interface device, such as communications adapter 190 connected to the network 195 over a medium 191. Medium 191 can be either a tangible medium, including but not limited to optical or analog communications lines, or may be implemented with wireless techniques, including but not limited to microwave, infrared or other transmission techniques. The series of computer instructions embodies all or part of the functionality previously described herein with respect to the invention. Those skilled in the art will appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Further, such instructions may be stored using any memory technology, present or future, including, but not limited to, semiconductor, magnetic, optical or other memory devices, or transmitted using any communications technology, present or future, including but not limited to optical, infrared, microwave, or other transmission technologies. It is contemplated that such a computer program product may be distributed as a removable media with accompanying printed or electronic documentation, e.g., shrink wrapped software, preloaded with a computer system, e.g., on system ROM or fixed disk, or distributed from a server or electronic bulletin board over a network, e.g., the Internet or World Wide Web.

[0097] Although various exemplary embodiments of the invention have been disclosed, it will be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the spirit and scope of the invention. Further, many of the system components described herein have been described using products from International Business Machines Corporation. It will be obvious to those reasonably skilled in the art that other components performing the same functions may be suitably substituted. Further, the methods of the invention may be achieved in either all software implementations, using the appropriate processor instructions, or in hybrid implementations which utilize a combination of hardware logic and software logic to achieve the same results. Although an all software embodiment of the invention was described, it will be obvious to those skilled in the art that the invention may be equally suited for use with video system the use firmware or hardware components to accelerate processing of video signals. Such modifications to the inventive concept are intended to be covered by the appended claims.

	Number	Date	Country
Parent	10154043	May 2002	US
Child	10223021	Aug 2002	US

Method and apparatus for video conferencing with audio redirection within a 360 degree view

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

US Classifications

International Classifications

Abstract

Description

Claims

RELATED APPLICATIONS

Continuation in Parts (1)