This invention relates to the video or audiovisual transmission arts. It finds particular application when a sender-box supplies audiovisual content to one or multiple receivers utilizing a video format having independent reference frames, such as I-pictures in the MPEG2 format. However, it is to be appreciated that the invention will find application in other formats and applications.
Multimedia devices, such as analog TV-link and digital TV-link systems, have become popular with consumers in recent years. Home networking has recently become less expensive and more popular with consumers, particularly wireless home networking such as wireless LANs using IEEE 802.11 standards. The combination of these two recently popular technologies make it possible to have a set top box receiving a video broadcast and to act as a sender-box, providing the video over a local network to receivers dispersed throughout the home, and even throughout the premises such as in a garden or a detached garage for example. Consumers, however, prefer not to run wires throughout their home and, since powerful electronic chips have become inexpensive enough, it is economical to incorporate MPEG2 encoding in consumer entertainment devices networked via wireless home networking.
A difficulty is encountered when transmission errors occur between the sender-box and one or more receivers around the home. For example, a standard mechanism for non-streaming data connections is for the receiver to send a signal back to the sender-box with a request for re-transmission of the data that was lost or corrupted due to the transmission error. In a multimedia environment, particularly when viewing, this creates objectionable delays and momentary freezes of the display.
A better solution typically used in set top boxes is to wait passively for the next reference frame to be transmitted, I-frame in an MPEG2 encoded format. The I-frame has the necessary video information to construct a complete video frame, however, it is on average, half the interval between successive I-frames for the next I-frame to be received, typically 0.5 seconds. This delay again causes objectionable momentary freezes of the display, but only to receivers that experienced the problem. Another solution that can be adapted to set top boxes is for the sender-box to send only I-frames, for MPEG2 encoding, which eliminates the aforementioned delay. However, a disadvantage of this solution is that the bit-rate of the stream is typically too high for the network to handle or the quality is too low if the bit-rate is reduced.
It is desirable, therefore, to provide a system and method that provides a shorter period of video degradation or freezing following errors or other transmission interruptions without increasing the bit rate of the video transmission. It is also desirable to provide these improvements in such a manner that a standard video decoding such as MPEG2 may be used in the receivers.
It is further desirable to use a mechanism of I-picture insertion to provide better overall quality by inserting fewer I-frames. As I-frames typically require more bits of information than P or B frames, having fewer I-frames means a higher average bit-rate per frame, thus an overall quality improvement. In the extreme case, there would be no I-frames at all, except those requested as the result of a transmission error or if a new decoder is activated.
In accordance with one aspect of the present invention, a video display method is provided. The method includes receiving a digital or analog audio/video stream at a sender-box, encoding, re-encoding or transcoding the received digital or analog audio/video stream into a video stream of independent video frames and intervening dependent video frames, transferring the stream to one or more receivers, sensing a condition indicative of a transmission defect to at least one receiver, in response to sensing the condition, generating a request for an independent frame, and in response to the request, inserting an independent video frame into the video stream. The sensing a condition indicative of a transmission defect is performed by at least one of a picture defect detector on the receiver, a communications interface in the sender-box or receiver-box, a multiplex/de-multiplex section in the receiver and an encoding processor in the sender-box.
In accordance with another aspect of the present invention, a consumer entertainment system is provided. The consumer entertainment system includes an input means for receiving a video input, an encoding means for encoding the received video input into a digital audio/video stream including independent video frames and dependent video frames, and a means for transferring the video stream to one or more receivers. The consumer entertainment system also includes a means for sensing a condition indicative of a potential display defect on one or more of the receivers, a means for generating a request for an independent frame with the request being communicated by the transferring means to the encoding means which responds to the request by inserting an independent frame into the video stream.
One advantage of the present invention is that it provides a reduction in the time a video is degraded or frozen after a transmission interruption to below a level at which the average viewer will notice.
Another advantage is that the invention provides video at a reduced bit rate compared to typical prior art methods.
Yet another advantage is that the invention provides video at an improved quality compared to typical prior art methods when there are no transmission errors.
Still yet another advantage is that the invention utilizes well known video encoding standards such as MPEG2 which permit the use of commonly available receivers on the network, as well as receivers configured to request I-frame insertion.
Still further advantages of the present invention will become apparent to those of ordinary skill in the art upon reading and understanding the following detailed description of the preferred embodiments.
The invention may take form in various parts and arrangements of parts. The drawings are only for purposes of illustrating a preferred embodiment and are not to be construed as limiting the invention.
With reference to
The sender-box 10, including alternate devices such as PDAs, mobile phones laptop computers, video capture devices, cameras, CCD devices, web-cams or similar devices, includes an input video section 18, a digital or analog audio/video stream compression processor, encoder, re-encoder or transcoder 20, and a communications interface 22. The input section 18, in preferred embodiments, receives an analog input stream and converts it to a raw internal digital video format for further processing either by the input section 18 or by the digital audio/video stream encoder 20. In alternate embodiments, the input section receives video that has been encoded at the video source and converts it to a raw internal video format for further processing by the digital audio/video stream encoder 20.
The video encoder 20 converts the raw digital audio/video stream to a compressed format such as MPEG2 for transmission to one or more receivers 16. While the present application will be described primarily with respect to MPEG2 encoding, other encoding formats such as MPEG4 or DIVX, and future encoding formats, fall within the scope of the present application. The sender-box 10 may include a time-shift buffer 24 for recording encoded digital audio/video streams for delayed viewing, and may also include other features and controls typically found in set top boxes. However, these features and controls are not discussed in detail since knowledge of these is not necessary for understanding concepts of the embodiments described herein.
The MPEG2 format generally groups multiple video frames into a group of pictures (GOP). Each GOP begins with an I-frame, normally followed by a number of P and B frames. Each GOP can be as small as a single I-frame, and is typically less than 15 frames in length. I-frames are intra-coded frames with an average 7 to 1 reduction ratio. I-frames can be looked at as reference pictures that can be decoded without reference to previous frames. By distinction, P-frames and B-frames use data from previous or succeeding frames to decode a picture correctly. Therefore, the term I-frame as used herein, is defined to include video frame formats that include all of the data necessary to construct a complete picture frame without reference to earlier frames, hereinafter also referred to as independent frames or I-pictures. A JPEG or JPEG2000 image is an example of an independent frame. Also, a P-frame containing all I-macroblocks, for example, is an independent frame. It is also to be understood that the method of using 2 consecutive P-frames, in which e.g. the first updates the upper half of the picture, and the second updates the lower half, and other similar methods are included within the scope of the definition of I-frame as used herein. One can think of many variants, but they all come down to “rewriting the entire screen with independent picture data.”
Likewise, the terms P-frame and B-frame include video frame formats that are dependent on data from earlier or later frames in order to construct a complete video frame, hereinafter also referred to as dependent frames. P-frames are predicted based on prior I or P frames with the addition of data for changed macro blocks. P-frames average a 20 to 1 reduction ratio or about half the size of I frames. In one example, the P frame represents the difference between a current frame and an immediately preceding frame. B-frames are bi-directional predicted frames based on appearance with positions of past and future frame macro blocks. B-frames have less data than P-frames averaging about a 50 to 1 reduction ratio.
I-frames may be looked at as reference pictures that can be decoded without reference to previous frames. P-frames and B-frames require data from previous or succeeding frames to decode a picture correctly. While embodiments are described with respect to MPEG2 other formats similar in concept to MPEG2 may be employed and fall within the scope of the present application. In the case where MPEG4 is the selected format for the video encoder 20, each GOP can be as large as the maximum key frame interval, usually 200 to 300 frames.
In embodiments utilizing MPEG4, a decoder can use multiple frames as reference frames. If a transmission error occurs, an alternative to requesting an I-picture insertion is to send information to the encoder to avoid using the lost pictures as reference frames. In this manner, the encoding is still relatively efficient, even without the use of I-pictures.
Also, when a B-frame is lost or corrupt, the receiver decoding means can simply skip the B-frame and continue with the next frame without any harm and, therefore, it is not necessary to request I-frame insertion in such cases. It should also be understood that transmission errors can extend over a relatively prolonged period of time, wherein multiple frames are lost. In this case, a request for an I-frame insertion is usually needed to improve overall quality and provide faster error recovery.
In a typical set top box, the encoder sends I-frames on regular intervals, 1 I-frame for every 15 P/B-frames for example. To achieve a fixed bit rate for transmission, many systems have the encoder allocate and average the transmission rate over a GOP. In this scenario, when transmission data is lost due to communications problems, video degradation will continue to exist until the next I-frame transmission which may take as long as 0.5 seconds, or even longer. Under concepts of the present application, however, an improved method and apparatus for restoring video quality in a shorter time are provided without corrupting the images for users of other receivers.
It is to be understood that, while the illustrated embodiment only sends I-frames as needed, sending only P/B-frames when possible, other embodiments may send I-frames both at fixed intervals and as needed. However, the temporal frequency of the I-frames is advantageously reduced in order to lower the transmission bit rate without sacrificing video quality.
It is also to be emphasized that embodiments of the present application maintain a continuous stream of frames to the receivers and that the stream remains fully compliant with MPEG2 standards. This is important in the case of multiple receivers so that receivers not experiencing a transmission error are not affected by the requesting of an I-frame by another receiver. The quality of the stream is not noticeably affected by the I-frame insertion and each of the multiple receivers produces an improved overall viewing quality.
With reference again to
In response to detecting a defect condition, a transmitter 54 signals the communications module 22 of the box 10 requesting the insertion of an I-frame as soon as possible. The video compression processor 20 responds by inserting an I-frame, or other reference pictures, into the digital audio/video stream that is being sent to the receivers.
Each receiver 16 also includes a main control section 56 in communication with the detector 52 and the transmitter 54 and a multiplexer/demultiplexer unit 58. The multiplexer/demultiplexer unit separates audio and video portions of streams for separate processing in an I/O section 59. The multiplexer/demultiplexer unit 58 is also capable of detecting defects and requesting I-frame insertion.
While steps 64 and 66 are shown as separate steps, in practice they may be combined in a re-encoder, or when using only partial decoding/decompression. At step 68, the decoded digital audio/video stream is sent to the encoder module for further processing and, at step 70, if there is more video input, processing returns to step 64. It is to be understood that the flow charts presented in
While steps 84 and 86 are shown as separate steps, in practice they may be combined in a single chip or module. At step 88, the decoded digital audio/video stream is sent to the encoder module for further processing and, at step 90, if there is more video input, processing returns to step 84.
In all cases, processing continues at step 104 where the audio/video stream is sent to the encoder module for further processing and, at step 106, if there is more video input, processing returns to step 94. As with the previously described methods, steps 94-102 may be combined into a single function or chip in practice.
In the case where a transmission error is detectable by communications interface 22 as determined at step 128, an I-frame insertion is requested at step 130 in order to restore stream quality as soon as possible. Step 132 returns to step 110 for successive video processing if there is more audio/video stream data to be received from the video decoder 18. If the sender-box 10 is displaying buffered digital audio/video stream data from time-shift buffer 24, step 134 returns to step 116 to acquire additional digital audio/video stream data from the time-shift buffer.
While the method diagrammed in
While the invention has been described with respect to I-frames and P/B-frames, it is to be appreciated that, as previously described, I-frames may be looked at as reference pictures that can be decoded without reference to previous frames, whereas P-frames and B-frames require data from previous or succeeding frames to decode a picture correctly. Therefore, various embodiments incorporating any video encoding method utilizing similar concepts fall within the scope of the present application.
Further, while the invention has been described with respect to receivers connected to a wireless network, it is to be appreciated that the invention is applicable to wired connections between the encoder and receivers. Therefore, various embodiments incorporating video encoders connected to one or multiple distant receivers with decoders connected via either a wired or wireless network fall within the scope of the present application.
Still further, while the invention has been described with respect to an in-home application having a sender-box, a set top box in particular, connected to one or more receivers it is to be appreciated that the scope of this application includes other uses of the concepts described herein. For example, uses may include converting video in a format that does not incorporate the concept of I-frames to a format such as MPEG2 that incorporates the concept of I-frames, thus enabling I-frame insertion according to the methods described herein. Another use might be the modification of a video format that incorporates the concept of I-frames by reducing the number of I-frames sent to connected receivers according to concepts of the present application.
The invention has been described with reference to the preferred embodiments. Obviously, modifications and alterations will occur to others upon reading and understanding the preceding detailed description. It is intended that the invention be construed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Number | Date | Country | Kind |
---|---|---|---|
60429670 | Nov 2002 | US | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/IB03/04896 | 10/29/2003 | WO | 5/25/2005 |