The present invention relates to video decoding, and more particularly, to a method for adaptively performing video decoding, and to an associated adaptive complexity video decoder and an associated adaptive audio/video playback system.
When an end user is viewing a video program utilizing a home theater system at home, encoded data of the video program is decoded for being displayed on a display panel of the home theater system, and the decoded results of the encoded data can be scaled up or scaled down to meet the size and resolution of the display panel. Typically, a decoding device of the home theater system can decode the encoded data with ease, no matter how complicated the algorithm for decoding the encoded data is, and no matter whether the resolution of the video program is high or not.
It would be very convenient for the end user to own a portable electronic device (e.g. a mobile phone or a personal digital assistant (PDA)) that can be utilized for viewing the same video program anywhere. However, when a research and development (R&D) team of a manufacturer is designing such a portable electronic device, some problems may arise. For example, the decoding capability of the portable electronic device may be insufficient in a situation where the algorithm for decoding the encoded data is too complicated and/or the resolution of the video program is high. In another example, the power consumption may be too high when one or more processing circuits within the portable electronic device operate at the highest operation frequency available. Thus, there is a need for highly efficient and cost effective video decoders, in order to play back video with acceptable quality using portable electronic devices with limited computing capability.
It is therefore an objective of the claimed invention to provide a method for adaptively performing video decoding, and to provide an associated adaptive complexity video decoder and an associated adaptive audio/video playback system, in order to solve the above-mentioned problems.
It is another objective of the claimed invention to provide a method for adaptively performing video decoding, and to provide an associated adaptive complexity video decoder and an associated adaptive audio/video playback system, in order to selectively reduce the calculation load and the power consumption within a portable electronic device such as that mentioned above and/or perform adaptive audio/video playback adjustment.
It is another objective of the claimed invention to provide a method for adaptively performing video decoding, and to provide an associated adaptive complexity video decoder and an associated adaptive audio/video playback system, in order to selectively reduce complexity of decoding operations by selectively making one or more components therein operate in accordance with the resolution of reconstructed frames to be displayed, rather than the resolution of original frames, since the resolution of a display panel of the portable electronic device is typically limited, and is typically lower than the resolution of the original frames. As a result, scaling down the reconstructed frames sometimes can be avoided.
An exemplary embodiment of a method for adaptively performing video decoding comprises: performing decoding complexity management based upon bit stream information of an input bit stream of an adaptive complexity video decoder, in order to determine whether to reduce decoding complexity of at least one component of a plurality of components within the adaptive complexity video decoder; and selectively reducing decoding complexity of a portion of components within the adaptive complexity video decoder.
An exemplary embodiment of an associated adaptive complexity video decoder comprises a plurality of components and a decoding complexity manager. The plurality of components is arranged to perform operations related to video decoding. In addition, the decoding complexity manager is arranged to perform decoding complexity management based upon bit stream information of an input bit stream, in order to determine whether to reduce decoding complexity of at least one component within the adaptive complexity video decoder. Additionally, the decoding complexity manager selectively reduces decoding complexity of a portion of components within the adaptive complexity video decoder.
An exemplary embodiment of an associated adaptive audio/video playback system comprises an audio decoder and a video decoder. The audio decoder is arranged to perform audio decoding to generate audio information, and the video decoder is arranged to perform video decoding. In addition, the video decoder comprises a decoding manager, wherein the decoding manager delays audio playback of the audio information according to bit stream information of an input bit stream. In particular, the decoding manager of the video decoder can determine delaying the audio playback according to one or a combination of bit stream information of the input bit stream, a parameter representing processing capability, and a display buffer status. For example, according to one or a combination of bit stream information of the input bit stream, the parameter representing processing capability, and the display buffer status, the decoding manager of the video decoder determines whether to delay the audio playback. In another example, according to one or a combination of bit stream information of the input bit stream, the parameter representing processing capability, and the display buffer status, the decoding manager of the video decoder determines the delay amount of the audio playback.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
Certain terms are used throughout the following description and claims, which refer to particular components. As one skilled in the art will appreciate, electronic equipment manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not in function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is coupled to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.
Please refer to
According to this embodiment, the fast VLD and inverse quantization module 110 is arranged to perform fast VLD and inverse quantization on an input bit stream 108 to generate inverse quantization results 118, and the inverse transform unit 120 is arranged to perform inverse transform on the inverse quantization results 118 to generate inverse transform results 128. In addition, the motion compensation module 130 is arranged to perform motion compensation according to the input bit stream 108 and generate associated prediction outputs 138, where the temporal prediction unit 132 is arranged to perform temporal prediction and the spatial prediction unit 134 is arranged to perform spatial prediction. As shown in
Referring to
In particular, at least one portion of the adaptive complexity video decoder 100, such as the temporal prediction unit 132, the spatial prediction unit 134, the inverse transform unit 120, the reconstructed frame output unit 150, and/or the fast VLD and inverse quantization module 110, may operate in accordance with the resolution of the reconstructed frames 158, rather than the resolution of the original frames represented by the input bit stream 108, in order to reduce complexity of decoding operations. The reduced resolution frame can be achieved by down sampling the frame, for example, by selecting particular pixels to represent each block of the frame, such as selecting the bottom right pixels of each 2 by 2 pixels. In an embodiment, the temporal prediction unit 132 can operate in accordance with the resolution of the reconstructed frames 158, rather than the resolution of the original frames, in order to reduce complexity of temporal prediction. For example, the temporal prediction unit 132 can further estimate at least a portion of partial information that has been omitted for use of performing the temporal prediction. In addition, the spatial prediction unit 134 can operate in accordance with the resolution of the reconstructed frames 158, rather than the resolution of the original frames, in order to reduce complexity of spatial prediction. For example, the spatial prediction unit 134 can further estimate at least a portion of partial information that has been omitted for use of performing the spatial prediction. According to this embodiment, the inverse transform unit 120 can operate in accordance with the resolution of the reconstructed frames 158, rather than the resolution of the original frames, in order to reduce complexity of inverse transform. For example, the inverse transform unit 120 can omit a portion of calculations of the inverse transform by selecting one of a plurality of predetermined inverse transform functions FIT for use of the inverse transform. In addition, the reconstructed frame output unit 150 can operate in accordance with the resolution of the reconstructed frames 158, rather than the resolution of the original frames, in order to reduce complexity of generating the reconstructed frames. Based upon whether one or more conditions are satisfied, the in-loop filter 152 can selectively perform in-loop filtering for use of de-blocking corresponding to one of a plurality of levels of complexity, for example, full de-blocking, partial de-blocking, and skip de-blocking. Additionally, the fast VLD and inverse quantization module 110 can operate in accordance with the resolution of the reconstructed frames 158, rather than the resolution of the original frames, in order to reduce complexity of performing fast VLD and inverse quantization. More particularly, in order to reduce the complexity, the fast VLD and inverse quantization module 110 can utilize a lookup table comprising a main table and at least one sub-table (e.g. one or more sub-tables) during decoding, where the probability of utilizing the main table is higher than that of the at least one sub-table due to a predetermined arrangement of the lookup table in a design phase of the adaptive complexity video decoder 100.
According to some variations of this embodiment, at least a portion of the adaptive complexity video decoder 100 can omit partial information being processed by the portion of the adaptive complexity video decoder 100, in order to reduce complexity of decoding the input bit stream 108. Similar descriptions for these variations are not repeated in detail here.
According to some variations of this embodiment, at least a portion of the adaptive complexity video decoder 100-1 can omit partial information being processed by the portion of the adaptive complexity video decoder 100-1, in order to reduce complexity of decoding the input bit stream 108. Similar descriptions for these variations are not repeated in detail here.
By utilizing the architecture of any of the embodiments/variations disclosed above, the complexity of video decoding of respective components can be adjusted. For example, under control of the decoding complexity manager 100M, one or more components within the adaptive complexity video decoder 100 of the first embodiment (or one of the variations thereof) or within the adaptive complexity video decoder 100-1 of the second embodiment (or one of the variations thereof) can operate with a lower level of video decoding complexity. In another example, under control of the decoding complexity manager 100M, all of the components within the adaptive complexity video decoder 100 of the first embodiment (or one of the variations thereof) or within the adaptive complexity video decoder 100-1 of the second embodiment (or one of the variations thereof) can operate with the highest level of video decoding complexity when it is determined that reducing the decoding complexity of any component within the adaptive complexity video decoder 100 or the adaptive complexity video decoder 100-1 is not required. Referring to
In Step 912, the decoding complexity manager 100M performs decoding complexity management based upon the input bit stream 108 of the adaptive complexity video decoder under consideration (e.g. the adaptive complexity video decoder 100 of the first embodiment or the adaptive complexity video decoder 100-1 of the second embodiment), in order to determine whether to reduce decoding complexity of at least one component of a plurality of components within the adaptive complexity video decoder mentioned above. For example, the plurality of components may comprise the fast VLD and inverse quantization module 110, the inverse transform unit 120, the motion compensation module 130, the reconstructed frame output unit 150, the aforementioned de-blocking filter such as the in-loop filter 152, and the display time adjustment driver 175.
In Step 914, the decoding complexity manager 100M selectively reduces the decoding complexity of a portion of components within the adaptive complexity video decoder mentioned in Step 912. For example, the decoding complexity manager 100M can selectively reduce the decoding complexity of at least one portion (e.g. a portion or all) of the fast VLD and inverse quantization module 110, the inverse transform unit 120, the motion compensation module 130, the reconstructed frame output unit 150, the aforementioned de-blocking filter such as the in-loop filter 152, and the display time adjustment driver 175 at a specific time point. In another example, the decoding complexity manager 100M reduces the decoding complexity of none of the fast VLD and inverse quantization module 110, the inverse transform unit 120, the motion compensation module 130, the reconstructed frame output unit 150, the aforementioned de-blocking filter such as the in-loop filter 152, and the display time adjustment driver 175 at another time point.
According to this embodiment, the decoding complexity manager 100M can perform decoding complexity management by referring to one or a combination of a display buffer status of the display buffer 180, one or more parameter representing processing capability of the adaptive complexity video decoder mentioned in Step 912, and bit stream information of the input bit stream 108, where the bit stream information may comprise a bit rate, resolution, and/or frame type. Some other examples of the bit stream information comprise motion vectors, residuals such as direct current (DC) values, IDCT coefficients, a length of the bit stream, or the bit stream information that can be derived, for example, based on comparison and/or historical statistics, from the above exemplary bit stream information. In an embodiment, the decoding complexity manger 100M adjusts decoding complexity according to a variation of motion vectors. The parameter representing processing capability may comprise statistics of decoding time required per frame, for example, an average decoding time per frame can be used by the decoding complexity manager 100M to judge the current system capability and thus adjust decoding complexity adaptively. In particular, the decoding complexity manager 100M may perform decoding complexity management based upon all of the aforementioned display buffer status of the display buffer 180, the aforementioned at least one parameter, and the bit stream information of the input bit stream 108.
In this embodiment, based upon a first strategy of achieving the best quality available, the decoding complexity manager 100M can trigger the operation/procedure of standard decoding flow when the problem of video decoding delay is not an issue. However, when it is determined that the video decoding delay may occur (e.g. it is detected that the hardware resources are insufficient and/or the decoding workload is greater than a predetermined threshold), the decoding complexity manager 100M can trigger one or more operation/procedure, in order to prevent serious video decoding delay and achieve the best quality that the decoder may offer under current situation. For example, among the operations/procedures shown in
More specifically, in a situation where the buffer status of the display buffer 180 (and more particularly, the buffer level thereof) is between the predetermined threshold values TP and TB, the decoding complexity manager 100M turns off B frame de-blocking, and triggers the operation/procedure of adaptive P frame de-blocking (i.e. P frame de-blocking with adaptive complexity). In addition, in a situation where the buffer status of the display buffer 180 (and more particularly, the buffer level thereof) is between the predetermined threshold values TI and TP, the decoding complexity manager 100M turns off P frame de-blocking and B frame de-blocking, and triggers the operation/procedure of adaptive I frame de-blocking (i.e. I frame de-blocking with adaptive complexity). Additionally, in a situation where the buffer status of the display buffer 180 (and more particularly, the buffer level thereof) is between the predetermined threshold value TI and zero, the decoding complexity manager 100M turns off I frame de-blocking, P frame de-blocking, and B frame de-blocking.
In this embodiment, regarding the control mechanism of adaptive complexity de-blocking, the decoding complexity manager 100M can operate based upon a macroblock (MB) based de-blocking scheme. For example, the decoding complexity manager 100M can adaptively turn on/off I frame de-blocking, P frame de-blocking, and/or B frame de-blocking according to the buffer status and a bit stream length of at least one portion (e.g. a portion or all) of the input bit stream 108. This is for illustrative purposes only, and is not meant to be a limitation of the present invention. According to a variation of this embodiment, regarding the control mechanism of adaptive complexity de-blocking, the decoding complexity manager 100M can operate based upon a frame based de-blocking scheme. For example, the decoding complexity manager 100M can adaptively turn on/off I frame de-blocking, P frame de-blocking, and/or B frame de-blocking according to the buffer status, a bit stream length of at least one portion (e.g. a portion or all) of the input bit stream 108, a quantization step size (e.g. the so-called QP value), an MB type, and/or edge strength.
It is an advantage of the present invention that the present invention method and the associated adaptive complexity video decoder and the associated adaptive audio/video playback system of each embodiment/variations disclosed above can be applied to various kinds of digital video applications, where at least one portion (e.g. a portion or all) of the plurality of components within the adaptive complexity video decoder can operate with a lower level of complexity of video decoding. As the complexity of video decoding of one or more components can be adjusted, the decoding workload and the associated power consumption can be adaptively reduced. Therefore, in contrast to the related art, the present invention provides great flexibility of implementing digital video playback systems.
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
This application is a continuation-in-part application and claims the benefit of U.S. non-provisional application Ser. No. 12/732,177, which was filed on Mar. 25, 2010 and entitled “LOW COMPLEXITY VIDEO DECODER” and is incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5801975 | Thayer et al. | Sep 1998 | A |
7280595 | Lin | Oct 2007 | B1 |
7590059 | Gordon | Sep 2009 | B2 |
7630569 | DeCegama | Dec 2009 | B2 |
8446946 | Chiang | May 2013 | B2 |
20050240386 | Carballo et al. | Oct 2005 | A1 |
20060233259 | Chou | Oct 2006 | A1 |
20070274384 | Schwarz | Nov 2007 | A1 |
20080089417 | Bao et al. | Apr 2008 | A1 |
20090003447 | Christoffersen et al. | Jan 2009 | A1 |
20100053352 | Dekel | Mar 2010 | A1 |
20100226437 | Robertson | Sep 2010 | A1 |
Number | Date | Country |
---|---|---|
1902939 | Jan 2007 | CN |
101729883 | Jun 2010 | CN |
1 655 966 | May 2006 | EP |
1 655 966 | Apr 2011 | EP |
2007221697 | Aug 2007 | JP |
201008286 | Feb 2010 | TW |
02078353 | Oct 2002 | WO |
2008060262 | May 2008 | WO |
Entry |
---|
Thomas Wiegand et al:“Overview of the H.264/AVC Video Coding Standard”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, No. 7, Jul. 2003, pp. 560-576, XP011099249, IEEE Service Center, NJ, US. |
Yanmei Qu et al:“A Cost-effective VLD Architecture for MPEG-2 and AVS”, Journal of Signal Processing Systems, vol. 52, No. 1, Oct. 2007, pp. 95-109, XP019616654, Boston, US. |
Seong Hwan Cho et al:“A Low Power Variable Length Decoder for MPEG-2 Based on Nonuniform Fine-Grain Table Partitioning”, IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 7, No. 2, Jun. 1999, pp. 249-256, XP000833596, IEEE Service Center, NJ,US. |
Hyeong-Min Nam et al:“A Complexity Scalable H.264 Decoder with Downsizing Capability for Mobile Devices”, IEEE Transactions on Consumer Electronics, vol. 56, No. 2, May 2010, pp. 1025-1033, XP011312771, IEEE Service Center, NY,US. |
Number | Date | Country | |
---|---|---|---|
20110235713 A1 | Sep 2011 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 12732177 | Mar 2010 | US |
Child | 12830419 | US |