Patent Application
20070053428
The present invention relates generally to media data transmission and more particularly to reducing bandwidth overload.
A number of media playback systems use continuous media streams, such as video image streams, to output media content. However, some continuous media streams in their raw form often require high transmission rates, or bandwidth, for effective and/or timely transmission. In many cases, the cost and/or effort of providing the required transmission rate is prohibitive. This transmission rate problem is often solved by compression schemes that take advantage of the continuity in content to create highly packed data. Compression methods such Motion Picture Experts Group (MPEG) methods and its variants for video are well known in the art. MPEG and similar variants use motion estimation of blocks of images between frames to perform this compression. With extremely high resolutions, such as the resolution of 1920×1080i used in high definition television (HDTV), the data transmission rate of such a video image stream will be very high even after compression.
One problem posed by such a high data transmission rate is data storage. Recording or saving high resolution video image streams for any reasonable length of time requires considerably large amounts of storage that can be prohibitively expensive. Another problem presented by a high data transmission rate is that many output devices are incapable of handling the transmission. For example, display systems that can be used to view video image streams having a lower resolution may not be capable of displaying such a high resolution. Yet another problem is the limitations on continuous media streaming in systems with a fixed bandwidth or capacity. For example, in a local area network with multiple receiving/output devices such a network will often have a fixed bandwidth or capacity, and hence be physically and/or logistically incapable of simultaneously supporting multiple receiving/output devices.
Given the limitations, as discussed, it is apparent that a method and/or system that overcomes at least some of these limitations would be advantageous.
In accordance with at least one embodiment of the present invention, a display data is received. It is determined if a predetermined criteria is met by a first representation of the display data, wherein the first representation of the display data includes a first plurality of display streams to be transmitted to a second plurality of display devices. A first display stream of the first plurality of display streams is compressed in a first manner when it is determined that the first representation of the display does not meet the predetermined criteria. An advantage of the present invention is that networks for broadcasting of media streams are implemented more efficiently. Another advantage of the present invention is that multiple media streams may be transmitted to multiple users on a fixed bandwidth network by managing degradation in transmission quality.
Referring to
It will be appreciated that a channel's bandwidth can become unacceptable for transmission of a display data stream because the bandwidth of the channel changes, or because a portion of the channel's bandwidth that is available to transmit the display data stream changes, and because the amount of data to be transmitted changes. For example, even when a display data stream is allocated the entire bandwidth of a communications channel, such as a wireless network (e.g. an 802.11 compliant network), changes in bandwidth of the communication channel will affect the amount display data that can be transmitted. Similarly, when a display data stream is allocated a portion of a given network's bandwidth, a change in the portion allocated to the data stream will affect the amount of display data that can be transmitted. Information relating to such changes in the available bandwidth can be obtained from a transmission module, which can included the controller 395 described herein, that empirically or deterministically monitors the available bandwidth, such as through the use of adaptive bandwidth footprint matching as described herein at states 100 and 110 of
It will be further appreciated that the amount of data needed to represent a display data stream can vary depending upon the specific display information being transmitted. An encoding device can monitor changes in the amount of data needed to represent a specific portion of the display data stream.
The method of
At block 52, a determination is made whether a change in the amount of data compression of the display data stream should occur. For example, in one embodiment, if the amount of data reduction to a target display data stream that is needed is less than a specific threshold it will be determined that further data reduction through the use of compression techniques, such as the use of larger quantization factors, is appropriate. For example, data reduction techniques using compression, such as quantization, as described at state 130 of
However, once the amount of data reduction sought is greater than a threshold that indicates further data reduction would likely trigger blocking or other visual artifacts, then an alternate data reduction technique is used. It will be appreciated that other data compression techniques besides quantization can be used, such as reducing an available color depth of each pixel. Note that the threshold values can be predetermined values set based on user preference.
Once it is determined at block 52 that the level of compression to be applied to a display data stream is to change, flow proceeds to block 57 where a compression variable, such as a set of quantization levels, is provided to an encoder to set a level of compression to be applied to the target display data stream. The compression level is increased to reduce the bandwidth used to transmit a representation of the display data, and the compression level is decreased to increase the bandwidth used to transmit a representation of the display data when there is sufficient available bandwidth to support providing more data to represent a higher quality representation of the data stream. As illustrated in
At block 53, a determination is made whether a change in the current frame size (i.e. the number rows and columns of pixels) of the display data stream should occur. For example, in one embodiment, if the amount of data reduction to a target display data stream exceeds a specific threshold typically 30 to 50% (TH1 in
Once it is determined that the level of scaling to be applied to a display data stream is to change, flow proceeds to block 58 where a scaling variable is provided to an encoder to set one or more scaling factors to be applied. For example, if the scaling level is to be increased to further increase the amount of data reduction, the number of pixels representing the height can be reduced, as can the number of pixels representing the width. Alternatively, when operating in range 187 the scaling level can be decreased when there is sufficient available bandwidth to support providing more data to represent the display data stream. It will be appreciated that in one embodiment the largest possible resolution is selected that will accommodate providing a display data stream that is capable of being transmitted at the available bandwidth within the threshold. Note that in one embodiment, range 187 of
The current scaling and compression factors set at blocks 57 and 58 are used to encode the received display data stream target displays stream for transmission over an available bandwidth. In addition, the flow of
Referring now to
The desired tolerance Dj (or maximum acceptable delay time) can be calculated using a variety of methods. In one embodiment, the method used is to take into consideration the buffering size of each client receiver unit, and ensure that the client receiver unit will not run out of media content to decode. A typical formula to calculate Dj is to take the size of the buffer and estimate a lower bound (in units of time) to consume, or fill, the buffer. As it is often desirable to keep the buffer of each client receiver unit as full as possible, a typical Dj will be calculated as Dj=Tj(estimate)/2. Where Tj(estimate) is the estimated lower time bound to completely consume the input buffer of a receiver unit associated with stream j. Alternately, instead of using ½ of Tj(estimate), a more aggressive approach would be to use ¾ of Tj(estimate), and a more conservative approach might take ⅓ of Tj(estimate). In cases where Tj(estimate) is small for receiver devices that are incapable of providing considerable buffer space, a conservative approach may be more appropriate. In one embodiment, Tj(estimate) is obtained by taking observed peak (highest) data rate (in bytes/second) of stream j and the smallest size of the buffers (in bytes) of all the devices receiving stream j. In this case, Tj(estimate) can be evaluated as Bp/Rp, where Bp is the receive buffer size of device p and Rp is the peak data rate of stream j associated with device p, where device p receives stream j and has the smallest receive buffer. Alternately, Rp can be associated with any value between the mean (average) and the peak. In one embodiment, the peak data rate (Rp) can be based on the largest compressed frame. If the receiving client unit does not have enough buffering capability for at least one compressed frame then it is unlikely to be able to display the video smoothly without dropping frames.
At the commencement of each unit of media, such as a frame of video, the ABFM state machine transitions to state 110. In state 110, the actual transmit time Tj (the actual time of frame transmission completion) is compared against the estimated transmit time T′j (the expected time of frame transmission completion) at the start of each frame of stream j. In one embodiment, if the actual time of frame transmission completion exceeds the estimated time by less than the desired tolerance Dj (i.e. Tj−T′j<Dj), the ABFM state machine returns to steady state 100. Otherwise, if the actual transmit time exceeds the estimated time by at least desired tolerance Dj (i.e. Tj−T′j>=Dj), the ABFM state machine enters state 120. It will be appreciated when a video stream is being represented by a group of pictures, and not just I-frames, that the ABFM state machine would transition to state 110 at the commencement of each group of pictures.
In state 120, a victim stream v is selected from the plurality of video streams. In one embodiment, victim stream v is selected using a predetermined selection method, such as by round robin selection where each video stream is chosen in turn. In another embodiment, the victim stream v is selected based on a fixed priority scheme where lower priority streams are always selected before any higher priority scheme. In yet another embodiment, the victim stream v is selected based on a weighted priority scheme where the stream having the greatest amount of data and/or the priority of each stream plays a role in its probability of selection. It will be appreciated that when a single video stream is being transmitted, such as to a destination over a wireless connection, it will always be selected at state 120.
Regardless of the method of selecting a victim stream v, in one embodiment, each stream j has a count, herein referred to as A(j), that refers to the current degradation value of the modified stream of stream j. In this case, the current degradation value of victim stream v, A(v), is evaluated in state 120. If A(v) is 0, in one embodiment, the one or more quantization factors of the re-encoding process for the victim stream v are changed in state 130, thus resulting in a decreased in the amount of data transmitted in victim stream v when greater reduction in transmitted data is needed. In one embodiment, escalation of the degree of data compression occurs when the quantization factors are increased resulting in a decrease in the amount of data transmitted in victim stream v. For example, the MPEG algorithm uses quantization factors to reduce the amount of data by reducing the precision of the transmitted video stream. MPEG relies on quantization of matrices of picture elements (pixels) or differences in values of pixels to obtain as many zero elements as possible. The higher the quantization factors, the more zero elements produced. Using algorithms such as run-length encoding, video streams (or their associated matrices) containing more zeros can be more highly compressed than video streams having fewer zeros.
For example, the MPEG algorithm for compression of a video stream has a stage in the algorithm for a discrete cosine transform (DCT), a special type of a Fourier Transform. The DCT is used to transform blocks of pixels in the time domain to the frequency domain. As a result of this transformation, the elements in the frequency domain, post-DCT, that are closest to the top left element of the resulting matrix with indices (0,0) are weighted more heavily compared to elements at the bottom right of the matrix. If the matrix in the frequency domain were to use less precision to represent the elements in the lower right half of the matrix of elements, the smaller values in the lower right half will get converted to zero if they are below a threshold based on a quantization factor. Dividing each element by a quantization factor is one method utilized to produce more zero elements. MPEG and related algorithms often apply larger quantization values to decrease the precision of the matrices in the frequency domain, resulting in more zero elements, and hence a decrease the data transmission rate. It will be appreciated that while state 130 specifically discusses compression using quantization, other types of compression can be used, such as changing the pixel depth of each pixel.
After reducing the data transmission of the victim stream v by modifying the quantization factor (state 130), in one embodiment, the ABFM state machine transitions to state 160, where the degradation value A(v) is increased by one and then a modulus of 3 is applied, i.e. A(v)current=(A(v)previous+1) mod 3. As a result, the value of A(v) can cycle from 0 to 2 for a given stream. Since A(v) was previously determined to be 1 in state 120, the new A(v) value would be 1 (0+1 mod 3). After modifying the degradation value A(v) for victim stream v in state 160, the ABFM state machine transitions back to state 100. Note that the degradation value 160 is incremented when it is desirable for further data reduction of the current display data stream to use the scaling technique of state 140, as opposed to the compression technique of state 130. Therefore, the degradation value can remains unchanged at state 160 if it is desirable for additional data reduction through the use of the compression of the display data stream using the compression technique of state 130. For example, referring to table 2, when the amount of desired data reduction is less than 30% to 50% depending on system parameters, the degradation value A(v) remains zero for the selected stream to indicate further data reduction can occur by modifying the compression variables. When at or near 30% to 50% threshold of data reduction, the degradation value A(v) is incremented to facilitate further data reduction by modifying scaling variables at state 140 while maintaining the compression variables. When at or near 30% to 50% data reduction threshold, the degradation value A(v) can be incremented to facilitate further data reduction to occur by modifying scaling variables at state 150 while maintaining the previously set compression and scaling variables.
If A(v) is determined to be 1 for victim stream v in state 120, the ABFM state machine enters state 140. In one embodiment, the height of the reencoded data stream is reduced by a predetermined amount, ½ for example, in state 140, resulting in a decreased amount of data to be transmitted. One method used to scale blocks of pixels by half is to blend and average pixels. Another method used is to drop every other pixel. In cases where the video stream is interlaced, halving the height can be achieved by dropping alternate fields, such as dropping all of the odd horizontal display rows or all of the even horizontal display rows. It will be appreciated that in some formats, particularly those in the National Television System Committee (NTSC) and the Advanced Television System Committee (ATSC) formats, video streams are interlaced where the even horizontal display rows for an entire frame are displayed first and then the odd horizontal display rows are displayed next. In other embodiments, the height of the reencoded data stream is reduced by a factor other than a half, such as ⅓, using similar methods as appropriate. Note that the amount of compression, as previously set by state 130, can be changed or remain the same during state 140. For example, if the degradation value A(v) is set to one once the data compression of the target stream is 40%, it would be possible for state 140 to eliminate data compression and remove every other line of pixels of the target data stream to incrementally reduce the amount of data representing the target stream from 40%, using just compression, to 50% using just scaling. Alternatively, all or some data compression can be maintained while performing less scaling to achieve a 50%, for example, total data reduction.
After reducing the data transmission of the victim stream v by reducing the resolution of the stream (state 140), in one embodiment, the degradation value A(v) is modified in state 160, as discussed previously. The resulting value for A(v) is 2 (1+1 mod 3). After modifying the degradation value A(v) for victim stream v in state 160, the ABFM state machine transitions back to state 100. Note that the degradation value 160 is incremented when it is desirable for further data reduction of the current display data stream to use the scaling technique of state 150, as opposed to the scaling technique of state 140. Therefore, the degradation value remains unchanged at state 160 if it is desirable for additional data reduction through the use of the compression of the display data stream using the compression technique of state 130.
If A(v) is determined to be 2 for victim stream v in state 120, the ABFM state machine enters state 150. In one embodiment, the width of the reencoded data stream is reduced by a predetermined amount in state 150 using methods similar to those discussed previously with reference to state 140, such as dropping every other pixel. It will be appreciated that for a same reduction factor, the reduction methods of state 140 or state 150 are interchangeable. In cases where the victim stream v is interlaced, halving the height before the width is generally more appropriate as it is more efficient to completely skip alternating fields, saving substantial processing requirements. As discussed previously, the previously applied compression and scaling previously set at states 130 and 140 can be maintained or changed at state 150.
After the reducing the data transmission of the victim stream v by reducing the resolution of the stream (state 150), in one embodiment, the degradation value A(v) is modified in state 160, as discussed previously. The resulting value for A(v) is 0 (2+1 mod 3). After modifying the degradation value A(v) for victim stream v in state 160, the ABFM state machine transitions back to state 100. As discussed previously, the previously applied compression and scaling previously set at states 130 and 140 can be maintained or changed at state 150.
In one embodiment, as a result of the cycling between 0 to 2 of the degradation value A(v) for a victim stream v, the ABFM state machine cycles through three different kinds of degradation of the resolution and/or the precision of victim stream v each time it is selected for degradation in state 120. Although an ABFM state machine utilizing three kinds of data degradation has been discussed, in other embodiments, fewer or more steps of data degradation may be used according to the present invention. For example, in one embodiment, an ABFM state machine utilizes multiple step degradation involving more than one state of changing of quantization factors. It will also be appreciated that scaling factors of width and height other than ½ (e.g.: ¾) may be used. For example, in one embodiment, the amount by which the resolution and/or precision of a victim stream v is based on the degree to which the actual frame transmission completion time for a frame of video in victim stream v exceeds the estimated frame transmission completion time. For example, if the actual frame transmission completion time is 10% greater than the estimated frame transmission completion time, then the resolution of victim stream v could be scaled down by 10%, thereby causing the actual frame transmission completion time to likely come closer to the estimated frame transmission completion time.
Referring next to
Gateway media server 210, in one embodiment, accepts one or more input data streams, such as digital video or display data, from satellite dish 201 and/or WAN 200. Each input data stream can include a plurality of multiplexed channels, such as MPEG data channels. Gateway media server 210 broadcasts the data streams and/or channels over a common medium (local data network 220) to one or more receiving client units, such as laptop 230, computer 240, or viewing unit 250. In one embodiment, there is a one-to-one correspondence between the number of data channels input to gateway media server 210 and the number of client receiver units to receive output data channels or streams. In another embodiment, there are fewer data channels or streams than there are receiver client units. In this case, two or more client receiver units may need to share one or more data channels or streams. Local data network 220 can include a local area network, a wide area network, a bus, a serial connection, and the like. Local data network 220 may be constructed using cable, twisted pair wire, fiber optic cable, etc. During broadcast of the data streams to the receiving client units, gateway media server 210, in one embodiment, applies the ABFM algorithm, as discussed previously with reference to
In at least one embodiment, the ABFM algorithm is utilized by gateway media server 210 to attempt to ensure that a representation of the display data meets a predetermined criteria. For example, gateway media server 210 may transmit the display data to receiver client units, where a video sequence displayed on the receiver client units is a representation of the displayed data. If the video sequence is simultaneously displayed properly in real time (the predetermined criteria) on a number receiver client units, gateway media server 210 may not need to take further action. Else if the video sequence is choppy, is not synchronized, is delayed, or is not received by all the designated receiver client units, the representation of the display data does not meet the predetermined criteria, and gateway media server 210, in one embodiment, uses the ABFM method previously discussed to modify one or more of the data streams of display data to improve the display of the video sequence.
As discussed previously, in at least one embodiment, an ABFM algorithm is implemented to maintain the data transmission rate of ABFM server system 205 within a fixed bandwidth. In one embodiment, the bandwidth of ABFM server system 205 is fixed by the maximum bandwidth of the transmission medium (local data network 225) between gateway media server 210 and the client receiver units (laptop 230, computer 240, or viewing unit 250). For example, if local data network is a local area network having a maximum transmission rate of 1 megabit per second, the bandwidth of ABFM server system 205 may be fixed at a maximum of 1 megabit per second. Alternately, in another embodiment, the bandwidth of ABFM server system 205 could be a predetermined portion of the available bandwidth of the transmission medium (local data network 225). For example, if there are four ABFM server systems 205 connected to local data network 225 having a maximum transmission rate of 1 megabit per second, each ABFM server systems 205 could be predetermined to have a fixed bandwidth of 0.25 megabits per second (one fourth of the maximum available transmission rate). As previously discussed, the AFBM algorithm can also be implement to maintain the data transmission rate within a varying bandwidth, such as is common with wireless communication channels.
Although the transmission medium between gateway media server 210 and client receiver units is often the factor which limits or fixes the bandwidth of ABFM server system 205, in one embodiment, the bandwidth of ABFM server system 205 is fixed by the rate at which gateway media server 205 is able to input one or more data streams, compress one or more of the data streams, and output the compressed (and uncompressed) data streams or channels to the client receiver units. For example, if gateway media server 205 can only process 1 megabits of data per second, but local data network 225 has a transmission rate of 10 megabits per second, the bandwidth of ABFM server system 205 may be limited to only 1 megabits per second, even though local data network 225 can transmit at a higher transmission rate. It will be appreciated that the bandwidth of ABFM server system 205 could be limited by other factors without departing from the spirit or the scope of the present invention.
Referring to
Referring next to
Decode module 612 will decode the selected display stream (target stream) and provide relevant control information, such as the resolution and compression variables associated with the displays stream, if any, to control module 614. Control module 614 also receives information from a transmit/receive module, such as a wireless transmission module, as to the available bandwidth over transmission medium 630, information from encode module 613 relating to the amount of data associated with display data as it is currently being encoded, and information from the client 620 indicating its ability to process information. Based upon this information, the control module 614 can modify compression variables used by the encode module 613 on a frame-by frame basis, and to modify scaling variables used by a scalar of the encode module 613 to change the amount of data used to represent the selected display stream to continually provide the highest quality video possible while maintaining a high degree of confidence that the video can be successfully transmitted within the available bandwidth.
The client 620 will receive information transmitted over transmission medium 630 at transmit receive module 621. The received display data stream will be provided to the decode module 622, while control information such as compression and resolution information will be provided to the control module 623, which in turn will configure decode module 622. Status information relating to decoding of the display data can be provided from the module 622 to the control module 623. For example, buffer fullness information can be provided from the decoder to the control module 623 to control the rate at which information is received at the transmit/receive module 621. In addition, control module can provide feed back information to the server 610 about its video requirements. For example, information relating the resolution and supported video formats can be communicated from the control module 623 to the server 610 through the transmit/receive module 621.
FIG.8 illustrates a method in accordance with a specific embodiment of the present disclosure. At block 811, a level of data reduction to be applied to a video stream is monitored. For example, as previously discussed, if a bandwidth available to transmit a data stream falls below an acceptable level, or the amount of data needed to be transmitted over an available bandwidth increases, it is possible that data associated with a data stream can be lost. At block 812, a first data reduction technique is applied to a first portion of a video stream to obtain the level of data reduction desired in response to a desired level of data reduction being below a first threshold. For example, referring to
At block 882, a determination is made whether or not a change in available bandwidth has occurred. If not, the flow returns to block 822 until a change is detected. The flow proceeds to block 823 when a change is detected. In accordance with a specific embodiment of the present disclosure, a change in available bandwidth is determined by obtaining a current available bandwidth from a transmission module and comparing this value to previous values. Once a change in bandwidth is detected, it can be further determined whether the first portion of the video stream can be transmitted at the current available bandwidth. Determining the change in available bandwidth can be accomplished by determining if the first portion of the video stream can be transmitted at the current available bandwidth. At block 823, a determination is made whether or not the available bandwidth is less than a threshold value. If so, flow proceeds to block 825, otherwise the flow proceeds to block 824. At block 824, a second portion of the video stream is scaled for wireless transmission at a second frame size during a second time period in response to determining that the bandwidth is greater than the threshold value. In accordance with a specific embodiment of the present disclosure, the second frame size can be selected to be the largest frame size suitable for transmission at the available bandwidth. During the second time period, the first data compression is maintained.
At block 825, if the response to determining the available bandwidth is less than a threshold value, a second portion of the video stream is provided for video transmission at the first frame size during the second time period. Referring to
At block 826, the video stream is wirelessly transmitted.
One implementation of the invention is as sets of computer readable instructions resident in the random access memory of one or more processing systems configured generally as described in
In the preceding detailed description of the figures, reference has been made to the accompanying drawings which form a part thereof, and in which is shown by way of illustration specific preferred embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, mechanical, chemical and electrical changes may be made without departing from the spirit or scope of the invention. To avoid detail not necessary to enable those skilled in the art to practice the invention, the description may omit certain information known to those skilled in the art. Furthermore, many other varied embodiments that incorporate the teachings of the invention may be easily constructed by those skilled in the art. Accordingly, the present invention is not intended to be limited to the specific form set forth herein, but on the contrary, it is intended to cover such alternatives, modifications, and equivalents, as can be reasonably included within the spirit and scope of the invention. The preceding detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.
This application is a continuation-in-part of U.S. patent application having application Ser. No. 11/344,512, filed on Jan. 31, 2006 (Attorney Docket Number 1459-VIXS002C), which is a continuation of application Ser. No. 09/823,646, filed on Mar. 30, 2001(Attorney Docket Number 1459-VIXS002). U.S. patent applications having application Ser. No. 11/344,512 and application Ser. No. 09/823,646 are hereby incorporated in their entirety by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 09823646 | Mar 2001 | US |
| Child | 11344512 | Jan 2006 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11344512 | Jan 2006 | US |
| Child | 11553210 | Oct 2006 | US |
