Patent Grant
10511849
The present invention is directed towards video encoders and decoders (CODECs), and more particularly, towards an apparatus and method for reducing the perceived delay for the initial display of decoded video content following a channel change.
Popular video compression standards, such as MPEG-2 and JVT/H.264/MPEG AVC, use intra and inter coding. For proper decoding, a decoder decodes a compressed video sequence beginning with an intra-coded (I) picture, and then continues to decode the subsequent inter-coded (P and B) pictures. A Group of Pictures (GOP) may include an I picture and several subsequent P and B pictures. I pictures typically require many more bits to code than does a P or B picture of equivalent video quality.
When a receiver initially begins receiving a program on a particular channel, such as following a channel change or initial turning on of the receiver, it must wait until an I picture is received to begin decoding properly, which causes a delay. To minimize channel change delay in digital video broadcast systems, I pictures are typically sent frequently, such as every N pictures. For example, to enable ½ second delay of the video decompression portion of the system, it is common to use N=15 for 30 fps content. Because compressed I pictures are so much larger than compressed P and B pictures, this considerably increases the bitrate over what would be required if I pictures were not inserted so frequently.
Most broadcast systems transmit I pictures frequently, for example every ½ second, in order to limit the channel change delay time due to the video decoding system. In some systems, instead of sending full I pictures frequently, a technique called “progressive refresh” is used, where sections of pictures are intra coded. Typically, all macroblocks in the picture are intra-coded at least once during an N-picture period.
In the JVT/H.264/MPEG AVC compression standard, P and B pictures may be predicted using multiple reference pictures, including the pictures before a preceding I picture. The standard identifies random access points as Independent Decoder Refreshes, or IDRs, which constrain that no reference pictures before each IDR are used in predicting pictures following the IDR.
The JVT/H.264/MPEG AVC compression standard includes a tool called redundant pictures, defined in the standard as:
Accordingly, what is needed is an apparatus and method for reducing the perceived delay for the initial display of decoded video content following a channel change.
These and other drawbacks and disadvantages of the prior art are addressed by an apparatus and method that provide for fast channel change of compressed video content.
A video decoder for receiving compressed stream data and providing decompressed video output includes a demultiplexor for receiving the compressed stream data and separating the normal stream and the channel change stream, a normal decoding portion in signal communication with the demultiplexor for receiving the compressed normal and channel change streams, and providing decompressed video output, and a normal frame store in signal communication with the normal decoding portion for storing reference pictures.
These and other aspects, features and advantages of the present invention will become apparent from the following description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings.
The present invention may be better understood in accordance with the following exemplary figures, in which:
Embodiments of the present invention provide allow channel change delay at any desired rate with a lower bitrate than prior art methods. That is, the invention enables low delay channel change time in a compressed video broadcast system, while significantly reducing the bitrate over prior methods of enabling low-delay channel change. In the JVT/H.264/MPEG AVC standard, individual P and B pictures are coded using one or more different slice types (I, P and/or B), while I pictures are coded using only I slices. Accordingly, in the description that follows, the term “slice” may be substituted for the term “picture” depending on the context and applicable standard. Prior art systems broadcast I pictures frequently to enable channel change, for example every N pictures. In embodiments of the present invention, normal I pictures are sent less frequently, and additional lower quality I pictures are sent more frequently.
In accordance with the principles of the present invention, a desired channel change delay can be achieved without requiring I pictures to be sent as frequently as is done in prior art systems. Instead, additional lower quality coded pictures, included in what is herein called the channel change stream, are sent in addition to the normal quality coded pictures. In the channel change stream, lower quality I pictures are sent at the desired channel change frequency, and are used at the decoder during the initial period following a channel change. Normal quality I pictures are sent in the normal stream at a lower frequency, and are used at the decoder once they are available.
For example, consider a system that sends I pictures in the normal stream every N*K pictures and lower quality I pictures in the channel change stream every N pictures, with K>1. Each coded picture in the channel change stream corresponds to a normal stream coded picture. Thus, when a coded picture is present in the channel change stream, two coded representations of that picture are actually transmitted.
When a channel change occurs, a decoding system starts decoding the compressed video as soon as it receives an I picture, either from the normal stream or from the channel change stream. If the first I picture to arrive is from the normal stream, the decoder continues normally. However, if the first I picture to arrive is a lower quality I picture from the channel change stream, the decoder decodes and uses the lower quality I picture. This causes lower quality video to be displayed until a normal quality I picture arrives. This period of lower quality video is not significantly noticeable to a viewer as it is of short duration and immediately follows a channel change. The human visual system takes some time to adjust to a new visual scene.
The channel change stream may either contain only lower quality I pictures, or may contain lower quality I, P and B pictures. The picture rate of the channel change stream may be lower than that of the normal stream. The lower quality pictures may be of the same resolution as the normal pictures but encoded at a lower bitrate, or may be of a lower resolution than the normal pictures. The bitstream size of the lower quality I coded pictures in the channel change stream are small compared with the size of normal quality coded I pictures in the normal stream. So even though additional coded representations of the same picture are being transmitted, overall bitrate savings occur because the size of a normal quality P or B picture plus the lower quality I picture is typically significantly less than that of a normal quality I picture alone.
If the channel change stream contains low quality I, P and B pictures, after a channel change the decoder system waits for the arrival of a lower quality I picture, and then it decodes and displays the lower quality pictures from the channel change stream until a normal quality I picture is received, at which point it switches to the normal quality stream.
If the channel change stream contains only I pictures, the decoding system waits for the arrival of an I picture in either the normal stream or channel change stream after a channel change. If the first I picture to arrive is in the channel change stream, the decoding system decodes and displays the lower quality I picture. Then this lower quality picture is stored in the normal decoder picture stores and the decoding system begins decoding the subsequent normal stream P and B pictures, using the lower quality I picture from the channel change stream as a reference. Because these normal quality P and B pictures were inter-coded based on prior pictures in the normal stream rather than the corresponding lower quality I picture from the channel change stream, this will cause some decoding drift.
Experiments have shown, however, that the visual impact of such drift is small in this situation, because it lasts for only a short duration and immediately follows a scene change corresponding to the channel change. The encoder can manage how much drift would occur and adjust coding parameters of the normal and/or channel change stream pictures appropriately such that drift does not exceed reasonable limits.
The instant description illustrates the principles of the invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the principles of the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.
Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.
The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (“DSP”) hardware, read-only memory (“ROM”) for storing software, random access memory (“RAM”), and non-volatile storage.
Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
In the claims hereof, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The invention as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. Applicant thus regards any means that can provide those functionalities as equivalent to those shown herein.
As shown in
Turning to
Thus,
Turning now to
Thus,
The multiplexor arranges the transmission time of the coded pictures such that the channel change stream I pictures are interspersed with the normal stream coded pictures. The channel change stream coded picture is preferably transmitted near the time that the normal stream picture corresponding to the same input picture is transmitted, and before any normal stream pictures that are inter-predicted with respect to that picture.
As shown in
Consider the case where a receiver tuned in to the channel while picture 5 was being received. The receiver would then wait until the first I picture in either stream arrived, which in this example is 12 in the channel change stream, and decode and display it. The decoded lower quality I picture 12 from the channel change stream would then be placed in the decoder's picture store, and used in decoding pictures 12-23 from the normal stream. These decoded pictures will contain drift. When normal quality P picture 24 arrives in the normal stream, the receiver may either choose to decode the normal stream's P picture 24 or the channel change stream's I picture 24. This could either be a receiver end decision, or a preference could be signaled by the encoder in the bitstream, based on which one will yield less drift. The normal stream's pictures 25-35 are then decoded, still with drift. Once the normal stream's picture 36 is received, which is an I picture, the decoder can start decoding properly without added drift for all subsequent pictures. From the viewer's perspective, for a short period after a channel change, here up to about 1.5 seconds, lower quality video is displayed, and then normal quality is displayed.
Bitrate savings versus a prior art system is achieved because the large normal stream I pictures are sent less frequently that they would be sent in a prior art system. The lower quality I pictures sent in the channel change stream are much smaller than the normal quality I pictures. An encoding system may send the lower quality I pictures in the channel change stream as frequently as desired, and with any pattern. I pictures in the normal stream also need not follow a regular pattern, and for example may be inserted whenever a scene change occurs. An encoding system does not need to insert channel change stream I pictures if the distance between I pictures in the normal stream does not exceed a desired value. The encoding system may choose to insert I pictures in the channel change stream whenever necessary to maintain a maximum I picture spacing, associated with a desired channel change delay limit, for example.
The channel change stream may contain pictures of different resolutions. For example, some of the I pictures in the channel change stream may be of the same resolution as the normal stream and others may be at a lower resolution. Alternately, two or more different lower resolutions for pictures in the channel change stream may be used.
In order to reduce the drift that occurs when decoding a normal stream picture using a channel change stream picture as a predictor, the encoder can restrict the range of allowable reference pictures for the P and B pictures that follow the normal stream picture that corresponds to the channel change stream I picture. In the JVT/H.264 video compression standard, P and B pictures may be predicted using multiple reference pictures, which provides a coding efficiency advantage over using a single reference picture. For the example in
Turning to
Turning now to
Thus,
In the decoding system, following a channel change, lower quality video is initially displayed, and once an I picture in the normal stream is received, normal quality video begins to be displayed. The abrupt transition from lower quality video to normal quality video may be more noticeable to a viewer than the lower quality video itself. To reduce the abruptness of the transition, a postprocessor may be added following the decoder to filter the decoded pictures. The filter strength may be adjusted over several pictures, to gradually increase the resolution or quality of the decoded pictures.
Turning now to
Thus,
As shown in
Turning to
There are several possible ways in which the multiplexing of the normal steam and channel change stream may be performed. To enable a backwards compatible system where the normal stream can be decoded without alteration to pre-existing decoders, one method to is place the channel change stream's lower quality coded pictures in user data associated with the corresponding picture of the normal stream.
This method allows the decoding system to identify the picture time of a channel change stream coded picture. If this method is used, an alteration to the unique picture start code of the coded pictures in the channel change stream is necessary, such as by using bit or byte stuffing, to help the pre-existing normal decoder avoid detecting the picture start code of the channel change stream picture inside of the user data. The bit or byte stuffing procedure may be reversed in the decoding system, before passing data to a standards compliant decoder.
An alternative multiplexing method is to use a different PID for the channel change stream than for the normal stream. In this case, the channel change stream will need to include timing information for the coded pictures, synchronized with the normal stream pictures. Also, an association must be made between the PIDs of the normal stream and the channel change stream.
If the JVT/H.264/MPEG AVC compression standard is used in this system, and the resolution of the normal pictures and channel change pictures are identical, the redundant pictures syntax of JVT may be used for coding the channel change pictures by setting the redundant_pic_cnt field in the slice header to 1 for the channel change pictures. In this case, in the decoding system, the channel change stream pictures may be identified by searching for pictures containing a redundant_pic_cnt field equal to 1 in the slice header.
These and other features and advantages of the present invention may be readily ascertained by one of ordinary skill in the pertinent art based on the teachings herein. It is to be understood that the principles of the present invention may be implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof.
Most preferably, the principles of the present invention are implemented as a combination of hardware and software. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPU”), a random access memory (“RAM”), and input/output (“I/O”) interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.
It is to be further understood that, because some of the constituent system components and methods depicted in the accompanying drawings are preferably implemented in software, the actual connections between the system components or the process function blocks may differ depending upon the manner in which the present invention is programmed. Given the teachings herein, one of ordinary skill in the pertinent art will be able to contemplate these and similar implementations or configurations of the present invention.
Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present invention is not limited to those precise embodiments, and that various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as set forth in the appended claims.
This application is a continuation of U.S. Non-Provisional application Ser. No. 10/559,643 filed on Dec. 2, 2005 and which claims the benefit, under 35 U.S.C. § 365 of International Application PCT/US04/18764, filed Jun. 15, 2004 which claims the benefit of U.S. Provisional Application Ser. No. 60/478,923 filed Jun. 16, 2003.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6118498 | Reitmeier | Sep 2000 | A |
| 6480541 | Girod et al. | Nov 2002 | B1 |
| 6549720 | Yamada et al. | Apr 2003 | B1 |
| 6587505 | Nozawa et al. | Jul 2003 | B1 |
| 6873368 | Yu et al. | Mar 2005 | B1 |
| 7143432 | Brooks | Nov 2006 | B1 |
| 7167479 | Christensen et al. | Jan 2007 | B2 |
| 7430222 | Green et al. | Sep 2008 | B2 |
| 7457415 | Reitmeier et al. | Nov 2008 | B2 |
| 7675972 | Laksono et al. | Mar 2010 | B1 |
| 9161033 | Boyce et al. | Oct 2015 | B2 |
| 20010011304 | Wesinger et al. | Aug 2001 | A1 |
| 20020039386 | Han et al. | Apr 2002 | A1 |
| 20030007562 | Kerofsky et al. | Jan 2003 | A1 |
| 20030072370 | Girod et al. | Apr 2003 | A1 |
| 20030169724 | Mehta et al. | Sep 2003 | A1 |
| 20030196211 | Chan | Oct 2003 | A1 |
| 20040034864 | Barrett et al. | Feb 2004 | A1 |
| 20040181813 | Ota et al. | Sep 2004 | A1 |
| 20040194134 | Gunatilake | Sep 2004 | A1 |
| 20050053086 | Mehta et al. | Mar 2005 | A1 |
| 20050053299 | Fuchs et al. | Mar 2005 | A1 |
| 20050138668 | Gray et al. | Jun 2005 | A1 |
| 20050229221 | Kerofsky et al. | Oct 2005 | A1 |
| 20060117360 | Cooper et al. | Jun 2006 | A1 |
| 20060140276 | Boyce | Jun 2006 | A1 |
| 20070140337 | Lim et al. | Jun 2007 | A1 |
| 20070174880 | Fite | Jul 2007 | A1 |
| 20070248165 | Boyce | Oct 2007 | A1 |
| 20080095247 | Ohno | Apr 2008 | A1 |
| 20080196061 | Boyce | Aug 2008 | A1 |
| 20090245393 | Stein | Oct 2009 | A1 |
| 20100064316 | Dai | Mar 2010 | A1 |
| 20100111197 | Jiang | May 2010 | A1 |
| 20110109810 | Li | May 2011 | A1 |
| 20120163476 | Gautier | Jun 2012 | A1 |
| 20130339997 | Farkash | Dec 2013 | A1 |
| Number | Date | Country |
|---|---|---|
| 0883299 | Dec 1998 | EP |
| 11289539 | Oct 1999 | JP |
| 2001527351 | Dec 2001 | JP |
| Entry |
|---|
| Karczewicz et al., A Proposal for SP-frames: ITU—Telecommunications Standardization Sector: VCEG-L27; Jan. 9, 2001; XP-002287038; pp. 1-9; Ebisee, Germany. |
| Horn et al., Robust Internet Video Transmission Based on Scalable Coding and Unequal Error Protection; Signal Processing: Image Communication; vol. 15; Sep. 1999; pp. 77-94. |
| Kurceren et al., “Synchronization-Predictive Coding for Video Compresseion:The SP-Frames Design for JVT/H.26L,” IEEE, Nokia Research Center, Nokia, Inc., Irving, TX 75039, IEEE ICIP 2002, pp. II-497-II-500. |
| Wakerly, “Digital Design Principles and Practices”, pp. 278-283, copyright 1990, Princeton-Hall. |
| Ramamurthy et al., “Modeling and Analysis of a Variable Bit Rate Video Multiplexer”, IEEE INFOCOM '92, pp. 817-827, copyright 1992. |
| Ugur et al., Combining Bitstream Switching and FGS for H.264 Scalable Video Transmission Over Varying Bandwidth Networks; 2003 IEEE Pacific Rim Conference on Communications Computers and Signal Processing; vol. 2; Aug. 28, 2003; pp. 972-975; University of British Columbia, Vancouver, British Columbia, Canada. |
| Search Report dated Nov. 9, 2004. |
| Number | Date | Country | |
|---|---|---|---|
| 20150373355 A1 | Dec 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60478923 | Jun 2003 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10559643 | US | |
| Child | 14842317 | US |
