Patent Application
20070263072
The subject matter of the present disclosure relates to a system and method for dynamically adjusting bandwidth between multiple video streams of a videoconference.
A videoconferencing system may handle video from two sources at the same time. One example of a prior art system for handling video from two sources is the iPower system available from Polycom, Inc., the Assignee of the present disclosure. The iPower system allows user to include video of participants along with content in a videoconferencing meeting. In particular, the iPower system provides a dual images function. With the dual images function, the iPower system shares bandwidth between video of participants and video of content when content is selected to be shown. This allows conference participants to see other participants and visual content simultaneously.
In the iPower system, one video stream (e.g., video of participants) on one of the encoders is deemed more important than the other video stream (e.g., content). The iPower system receives feedback only from the one encoder associated with the video stream deemed more important. The feedback is in the form of differences between frames of the more important video stream. The iPower system has a table that describes what percentage of the bandwidth to allocate to the one encoder deemed more important. The feedback and total video rate are used to calculate an index that is used to retrieve a percentage value from the table. The bit rate for the one encoder deemed more important is then calculated as the total video rate multiplied by the percentage value retrieved from the table. The bit rate of the other encoder is set to the remainder of the bandwidth.
In a videoconference, however, a relatively fixed amount of bandwidth is typically available for video. When a new video stream is added or when multiple video streams are present, the prior art iPower system selects only one video source as important and statically divides the available bandwidth between the video sources. As a result, the video quality of one or both of the sources can suffer.
The subject matter of the present disclosure is directed to overcoming, or at least reducing the effects of, one or more of the problems set forth above.
A videoconferencing unit includes first and second encoder that each encodes a video stream of frames and generates variables indicating changes between frames of the video streams. The unit also includes a controller operatively coupled to the first and second encoders. The controller compares the variables from the encoders and determines first and second bit rates for the first and second encoders based on the comparison. Then, the controller sets the first and second encoders to the first and second bit rates, respectively. Preferably, comparing the variables, determining the bit rates, and setting the bit rates are dynamically repeated as the unit operates. In addition, the dynamic repetition is preferably allowed or limited to one or more predetermined intervals of time so that changes to the bit rates of the encoders occur in a relatively consistent manner.
The foregoing summary is not intended to summarize each potential embodiment or every aspect of the present disclosure.
The foregoing summary, preferred embodiments, and other aspects of subject matter of the present disclosure will be best understood with reference to a detailed description of specific embodiments, which follows, when read in conjunction with the accompanying drawings, in which:
While the subject matter of the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are herein described in detail. The figures and written description are not intended to limit the scope of the inventive concepts in any manner. Rather, the figures and written description are provided to illustrate the inventive concepts to a person skilled in the art by reference to particular embodiments, as required by 35 U.S.C. § 112.
Referring to
In one embodiment, the video provided to the first video input 110A can be of videoconferencing participants from a camera, while the video provided to the second video input 110B can be of content (e.g., textual annotations or whiteboard information) from a computer or other device. Although the unit 100 in the present embodiment has two encoders 120A-B, the unit 100 in other embodiments can have more than two encoders 120 and corresponding inputs 110 and channels 130. The communications transmit channels 130A-B connect to a network 140 for the videoconference, which is used to deliver video and audio to remote endpoints (not shown), such as a videoconferencing system at another location. The controller 150 configures the video inputs 110A-B, encoders 120A-B, and communication transmit channels 130A-B and monitors their states.
Referring to
Now that the general operation of the videoconferencing unit 100 has been discussed, we not return to
As noted above, the encoders 120A-B receive streams of frames from the inputs 110A-B. An example of a video stream 300 of incoming frames 310 is schematically shown in
The video encoders 120A-B of
To determine the Sum of the Absolute Differences between frames and process the video stream to determine which encoder 120A-B needs more bandwidth, the videoconferencing unit 100 performs a process 400 for processing video frames as shown in
The resulting normalized value is clipped or truncated to fit into a range of “one” to “ten” or into some other numerical range (Block 408). (As used below, the truncated Sum of the Absolute Difference divided by the number of macroblocks is referred to as the SAD value). Although the range from one to ten is used, it will be appreciated that other scales can be used. An SAD value of “one” occurs when there is relatively little difference between the current frame and the previous frame. On the other hand, an SAD value of “ten” occurs when there is a relatively large amount of change between the current frame and the previous frame. The encoder 120A-B passes this SAD value along with the encoded frame to its corresponding communications transmit channel 130A-B (Block 410). Every time a compressed frame is sent to the communications transmit channel 130A-B, the controller 150 is called with the SAD value (Block 412).
The controller 150 receives the SAD values from the communication transmit channels 130A-B associated with the encoders 120A-B (Block 414). Then, the controller 150 determines how to assign or divide the bit rates to the encoders 120A-B using the SAD values received (Block 416). As discussed previously and as shown in
Before discussing the process 500 of
In the process 500, it is desirable to limit the number of bandwidth changes so the encoders 120A-B does not jump back and forth among different bit rates because the videoconferencing unit 100 may not be able to react immediately to changes in bandwidth. The decision of how frequently to adjust the bandwidths of the encoders 120A-B is dependent on the feedback (SAD values) from the encoders 120A-B. Furthermore, the decision of how frequently to adjust the bandwidths is also preferably dependent on intervals of time.
In the process 500, the controller 150 first determines whether the SAD values from the encoders 120A-B are in the appropriate range, e.g., from “one” to “ten” in the present example (Block 502). The controller 150 then determines if one encoder-A (e.g., encoder 120A) needs more bandwidth (Block 504). The controller 150 calculates SAD_Diff-A as a sum of the SAD value for encoder-A (SAD-A) minus the SAD value for encoder-B (SAD-B) (Block 506).
Then, the controller 150 determines whether the difference (SAD_Diff-A) is greater than or equal to the value of “four” (Block 510). If not, then there is not a significant change in the frames of encoder-A relative to encoder-B to warrant increasing the bit rate of encoder-A. Accordingly, the controller 150 instead determines whether encoder-B needs more bandwidth (Block 516), which is discussed below with reference to
If the difference (SAD_Diff-A) is greater or equal to “four” at Block 510, the controller 150 determines whether the difference (SAD_Diff-A) is less than or equal to “seven” (Block 512). If so, the controller 150 allows the bit rate to be adjusted every first interval, e.g., two seconds (Block 520). This first interval can be more or less than two seconds based on the implementation. In addition, this first interval may be fixed or predetermined or may be varied based on monitored parameters, operational conditions, or user-selection, for example.
Before actually adjusting the bit rate, the controller 150 first determines if the time since the last bit rate adjustment has been made is less than the first interval of two seconds (Block 522). If so, then the process does nothing and returns to receiving the next SAD values for the next frame because the process 500 has not been in the current state of adjusting the bit rate every two seconds for a long enough period of time (Block 514). If the time since the last bit rate adjustment has been made is more than two seconds at Block 522, then the controller 150 proceeds with steps for updating the bit rate discussed later.
However, if the difference (SAD_Diff-A) is greater than “seven” at Block 512, then the controller 150 allows the bit rate to be adjusted every second interval, e.g., one second (Block 530). Again, this second interval can be more or less than one seconds based on the implementation. In addition, this second interval may be fixed or predetermined or may be varied based on monitored parameters, operational conditions, or user-selection, for example. Preferably, this second interval (e.g., one second) is less than the first interval (e.g., two seconds) because the difference (SAD_Diff-A) being greater than “eight” indicates a greater need for bandwidth by encoder-A.
Before actually adjusting the bit rate, the controller 150 first determines if the time since the last bit rate adjustment has been made is less than one second (Block 532). If so, then the process does nothing and returns to receiving the next SAD values for the next frame because the process has not been in the current state of adjusting the bit rate every one second long enough (Block 514). If the time since the last bit rate adjustment has been made is more than one second at Block 532, then the controller 150 proceeds with steps for updating the bit rate discussed below.
At Block 540, the controller 150 starts the process of adjusting the bit rate assigned to the encoders 120A-B. Here, the controller 150 calculates a new bit rate value (“newBitRate”) that is equal to the product of the total video rate (“TotalVideoRate”) times a percentage of bit rate split between the encoders (“BitRateSplitPercent”) (Block 540). In one embodiment, the “TotalVideoRate” is a fixed or predetermined value, which can be 384-kb/s, for example. In one embodiment, the “BitRateSplitPercent” is also a fixed or predetermined percentage, which can be 80-percent, for example. It is understood that these values can be different for a particular implementation. In an alternative embodiment, the controller 150 can vary the “TotalVideoRate” and the “BitRateSplitPercent” based on monitored operating parameters of the control unit, based on user-selected values, or based on other variables. For example, a user can use a graphical user interface to choose the “BitRateSplitPercent.” In another example, the videoconferencing unit 100 can have software that monitors the available bandwidth, which is used to adjust the “TotalVideoRate” during operation.
Once the “newBitRate” is calculated, the controller 150 then determines whether the bit rate currently assigned to encoder-A is already equal to the “newBitRate” (Block 542). If so, then no change to the bit rate to encoder-A is needed so the process 500 ends (Block 544). Once ended, the process 500 is repeated for a subsequent frame. Repeating the process 500 can be scheduled or immediate depending on the implementation.
If the bit rate currently assigned to encoder-A is not equal to the “newBitRate”, the controller 150 sets the bit rate for encoder-A to the “newBitRate” as calculated (Block 546) and sets the bit rate for encoder-B to a sum of the “TotalVideoRate” minus the “newBitRate.” Then, the time value representing when the last adjustment to the bit rate has been made is set to the current time and the current process 500 ends (Block 544) so the process 500 can be repeated for calculating differences associated with new incoming frames.
As noted previously at Block 516, the difference (SAD_Diff-A) associated with encoder-A may be less than “four” in which case the controller 150 instead determines if encoder-B needs more bandwidth.
Next, the controller 150 determines whether the difference (SAD_Diff-B) is greater than or equal to the value of “four” (Block 560). If the difference (SAD_Diff-B) is greater or equal to “four” at Block 560, the controller 150 determines whether the difference (SAD_Diff-B) is less than or equal to “seven” (Block 562). If so, the controller 150 allows the bit rate to be adjusted every two seconds (Block 570). Accordingly, the controller 150 determines if the time since the last bit rate adjustment has been made is less than two seconds (Block 572). If so, the process 550 does nothing because the process 550 has not been in the current state of adjusting the bit rate every two seconds long enough. Accordingly, the current process ends so that it can return to receiving the next SAD values for a subsequent frame (Block 564). If the time since the last bit rate adjustment has been made is more than two seconds at Block 572, then the controller 150 proceeds with steps for updating the bit rate discussed later.
At Block 580, the controller 150 allows the bit rate to be adjusted every one second if the difference (SAD_Diff-B) is greater than “seven” at Block 562. Accordingly, the controller 150 determines if the time since the last bit rate adjustment has been made is less than one second (Block 582). If so, then the process 550 does nothing and returns to receiving SAD values for a subsequent frame because the process 550 has not been in the current state of adjusting the bit rate every one second long enough (Block 564). If the time since the last bit rate adjustment has been made is more than one second at Block 582, then the controller 150 proceeds with steps for updating the bit rate discussed below.
To adjust the bit rate assigned to the encoders 120A-B, the controller 150 calculates a new bit rate value (“newBitRate”) that is equal to the product of the total video rate (“TotalVideoRate”) times a percentage of bit rate split between the encoders (“BitRateSplitPercent”) (Block 590). In one embodiment as noted previously, the “TotalVideoRate” is a predetermined value of 384-kb/s, and the “BitRateSplitPercent” is set at 80-percent.
The controller 150 then determines whether the bit rate currently assigned to encoder-B is equal to the “newBitRate” (Block 592). If so, then no change to the bit rate to encoder-B is needed so that the process 550 ends to be repeated for later incoming frames (Block 594). Otherwise, the controller 150 sets the bit rate for encoder-B to the “newBitRate” just calculated (Block 546) and sets the bit rate for encoder-A to a sum equal to the “TotalVideoRate” minus the “newBitRate.” Then, the time value representing when the last adjustment to the bit rate has been made is set to the current time and the current process ends (Block 594) so that the process can be repeated for calculating differences associated with new incoming frames.
If the difference (SAD_Diff-B) is less than “four” at Block 560, then the controller 150 preferably determines whether the time since the last rate adjustment is greater than two seconds (Block 566). If the time since the last adjustment is more than two seconds, then the controller 150 splits the available bandwidth between the two encoders 120A-B (Block 568). Thus, each of the encoders 120A-B is assigned 50% of the “TotalVideoRate.” If the time since the last adjustment is less than two seconds, however, the controller 150 does not adjust the bit rates for the encoders 120A-B because the process 550 has not been in this state long enough to warrant changing bit rates. Thus, the current process 550 ends (Block 569) so that it can be repeated for calculating differences associated with new incoming frames.
The foregoing description of preferred and other embodiments is not intended to limit or restrict the scope or applicability of the inventive concepts conceived of by the Applicants. In exchange for disclosing the inventive concepts contained herein, the Applicants desire all patent rights afforded by the appended claims. Therefore, it is intended that the appended claims include all modifications and alterations to the full extent that they come within the scope of the following claims or the equivalents thereof.
