Patent Grant
9762901
The disclosed embodiments of the present invention relate to video coding, and more particularly, to a video coding method using at least evaluated visual quality determined by one or more visual quality metrics and a related video coding apparatus.
The conventional video coding standards generally adopt a block based (or coding unit based) coding technique to exploit spatial redundancy. For example, the basic approach is to divide the whole source frame into a plurality of blocks (coding units), perform prediction on each block (coding unit), transform residues of each block (coding unit) using discrete cosine transform, and perform quantization and entropy encoding. Besides, a reconstructed frame is generated in a coding loop to provide reference pixel data used for coding following blocks (coding units). For certain video coding standards, in-loop filter(s) may be used for enhancing the image quality of the reconstructed frame. For example, a de-blocking filter is included in an H.264 coding loop, and a de-blocking filter and a sample adaptive offset (SAO) filter are included in an HEVC (High Efficiency Video Coding) coding loop.
Generally speaking, the coding loop is composed of a plurality of processing stages, including transform, quantization, intra/inter prediction, etc. Based on the conventional video coding standards, one processing stage selects a video coding mode based on pixel-based distortion value derived from a source frame (i.e., an input frame to be encoded) and a reference frame (i.e., a reconstructed frame generated during the coding procedure). For example, the pixel-based distortion value may be a sum of absolute differences (SAD), a sum of transformed differences (SATD), or a sum of square differences (SSD). However, the pixel-based distortion value merely considers pixel value differences between pixels of the source frame and the reference frame, and sometimes is not correlated to the actual visual quality of a reconstructed frame generated from decoding an encoded frame. Specifically, based on experimental results, different processed images, each derived from an original image and having the same pixel-based distortion (e.g., the same mean square error (MSE)) with respect to the original image, may present different visual quality to a viewer. That is, the smaller pixel-based distortion does not mean better visual quality in the human visual system. Hence, an encoded frame generated based on video coding modes each selected due to a smallest pixel-based distortion value does not guarantee that a reconstructed frame generated from decoding the encoded frame would have the best visual quality.
In accordance with exemplary embodiments of the present invention, a video coding method using at least evaluated visual quality obtained by one or more visual quality metrics and a related video coding apparatus are proposed.
According to a first aspect of the present invention, an exemplary video coding method is disclosed. The exemplary video coding method includes: utilizing a visual quality evaluation module for evaluating visual quality based on data involved in a coding loop; and referring to at least the evaluated visual quality for performing de-blocking filtering.
According to a second aspect of the present invention, an exemplary video coding method is disclosed. The exemplary video coding method includes: utilizing a visual quality evaluation module for evaluating visual quality based on data involved in a coding loop; and referring to at least the evaluated visual quality for deciding a target coding parameter associated with de-blocking filtering.
According to a third aspect of the present invention, an exemplary video coding apparatus is disclosed. The exemplary video coding apparatus includes a visual quality evaluation module and a coding circuit. The visual quality evaluation module is arranged to evaluate visual quality based on data involved in a coding loop. The coding circuit has the coding loop included therein, and is arranged to refer to at least the evaluated visual quality for performing de-blocking filtering.
According to a fourth aspect of the present invention, another exemplary video coding apparatus is disclosed. The video coding apparatus includes a visual quality evaluation module and a coding circuit. The visual quality evaluation module is arranged to evaluate visual quality based on data involved in a coding loop. The coding circuit has the coding loop included therein, and is arranged to refer to at least the evaluated visual quality for deciding a target coding parameter associated with de-blocking filtering.
These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
Certain terms are used throughout the description and following claims to refer to particular components. As one skilled in the art will appreciate, manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is coupled to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.
The concept of the present invention is to incorporate characteristics of a human visual system into a video coding procedure to improve the video compression efficiency or visual quality. More specifically, visual quality evaluation is involved in the video coding procedure such that a reconstructed frame generated from decoding an encoded frame is capable of having enhanced visual quality. Further details of the proposed visual quality based video coding design are described as below.
As shown in
One key feature of the present invention is using the visual quality evaluation module 104 to evaluate visual quality based on data involved in the coding loop of the coding circuit 102. In one embodiment, the data involved in the coding loop and processed by the visual quality evaluation module 104 may be raw data of the source frame IMGIN. In another embodiment, the data involved in the coding loop and processed by the visual quality evaluation module 104 may be processed data derived from raw data of the source frame IMGIN. For example, the processed data used to evaluate the visual quality may be transformed coefficients generated by the transform module 113, quantized coefficients generated by the quantization module 114, reconstructed pixel data before the optional de-blocking filter 119, reconstructed pixel data after the optional de-blocking filter 119, reconstructed pixel data before the optional SAO filter 120, reconstructed pixel data after the optional SAO filter 120, reconstructed pixel data stored in the frame buffer 121, motion-compensated pixel data generated by the motion compensation unit 125, or intra-predicted pixel data generated by the intra prediction module 123.
The visual quality evaluation performed by the visual quality evaluation module 104 may calculate one or more visual quality metrics to decide one evaluated visual quality. For example, the evaluated visual quality is derived from checking at least one image characteristic that affects human visual perception, and the at least one image characteristic may include sharpness, noise, blur, edge, dynamic range, blocking artifact, mean intensity (e.g., brightness/luminance), color temperature, scene composition (e.g., landscape, portrait, night scene, etc.), human face, animal presence, image content that attracts more or less interest (e.g., region of interest (ROI)), spatial masking (i.e., human's visual insensitivity of more complex texture), temporal masking (i.e., human's visual insensitivity of high-speed moving object), or frequency masking (i.e., human's visual insensitivity of higher pixel value variation). By way of example, the noise metric may be obtained by calculating an ISO 15739 visual noise value VN, where VN=σL*+0.852·σu*+0.323·σu* Alternatively, the noise metric may be obtained by calculating other visual noise metric, such as an S-CIELAB metric, a vSNR (visual signal-to-noise ratio) metric, or a Keelan NPS (noise power spectrum) based metric. The sharpness/blur metric may be obtained by measuring edge widths. The edge metric may be a ringing metric obtained by measuring ripples or oscillations around edges.
In one exemplary design, the visual quality evaluation module 104 calculates a single visual quality metric (e.g., one of the aforementioned visual quality metrics) according to the data involved in the coding loop of the coding circuit 102, and determines each evaluated visual quality solely based on the single visual quality metric. In other words, one evaluated visual quality may be obtained by referring to a single visual quality metric only.
In another exemplary design, the visual quality evaluation module 104 calculates a plurality of distinct visual quality metrics (e.g., many of the aforementioned visual quality metrics) according to the data involved in the coding loop of the coding circuit 102, and determines each evaluated visual quality based on the distinct visual quality metrics. In other words, one evaluated visual quality may be obtained by referring to a composition of multiple visual quality metrics. For example, the visual quality evaluation module 104 may be configured to assign a plurality of pre-defined weighting factors to multiple visual quality metrics (e.g., a noise metric and a sharpness metric), and decide one evaluated visual quality by a weighted sum derived from the weighting factors and the visual quality metrics. For another example, the visual quality evaluation module 104 may employ a Minkowski equation to determine a plurality of non-linear weighting factors for the distinct visual quality metrics, respectively; and then determine one evaluated visual quality by combining the distinct visual quality metrics according to respective non-linear weighting factors. Specifically, based on the Minkowski equation, the evaluated visual quality ΔQm is calculated using following equation:
ΔQi is derived from each of the distinct visual quality metrics, and 16.9 is a single universal parameter based on psychophysical experiments. For yet another example, the visual quality evaluation module 104 may employ a training-based manner (e.g., a support vector machine (SVM)) to determine a plurality of trained weighting factors for the distinct visual quality metrics, respectively; and then determine one evaluated visual quality by combining the distinct visual quality metrics according to respective trained weighting factors. Specifically, supervised learning models with associated learning algorithms are employed to analyze the distinct visual quality metrics and recognized patterns, and accordingly determine the trained weighting factors.
After the evaluated visual quality is generated by the visual quality evaluation module 104, the evaluated visual quality is referenced by the de-blocking filter 119 to control/configure the operation of de-blocking filtering. As the evaluated visual quality is involved in making the video coding mode decision for de-blocking filtering, the source frame IMGIN is encoded based on characteristics of the human visual system to thereby allow a decoded/reconstructed frame to have enhanced visual quality.
For example, the de-blocking filter 119 may decide a boundary strength (BS) of a block boundary between adjacent blocks according to the evaluated visual quality, where the evaluated visual quality in this case may provide visual quality information for one or both of the adjacent blocks.
Please refer to
In contrast to the conventional video coding design, the present invention proposes using the evaluated visual quality VQ(C or R′) derived from data involved in the coding loop of the coding unit 102 to find the boundary strength value, where one evaluated visual quality VQ(C or R′) for deciding the boundary strength of a vertical/horizontal boundary of adjacent blocks may be obtained by a single visual quality metric or a composition of multiple visual quality metrics, C represents raw data of the source frame IMGIN (particularly, pixel data of one or both of the adjacent blocks), and R′ represents processed data derived from raw data of the source frame IMGIN (particularly, processed data derived from processing pixel data of one or both of the adjacent blocks by video coding). For example, one or more of the non-visual-quality criteria may be replaced by a visual quality based criterion, where the visual quality based criterion is checked to see if the evaluated visual quality satisfies a predetermined condition. In one exemplary design, some or all of the non-visual-quality criteria shown in
In an alternative design, both of the evaluated visual quality (e.g., VQ(C or R′)) and the checking result of at least one non-visual-quality based criterion are used to control the de-blocking filtering operation (e.g., boundary strength derivation). For example, the de-blocking filter 119 refers to the evaluated visual quality to find a first de-blocking filtering setting (e.g., a boundary strength value for a block boundary between adjacent blocks), refers to the checking result of at least one non-visual-quality based criterion to find a second de-blocking filtering setting (e.g., a second boundary strength value for the block boundary between the adjacent blocks), and finally selects one of the first de-blocking filtering setting and the second de-blocking filtering setting as a target de-blocking filtering setting (e.g., a boundary strength value actually assigned to the block boundary between adjacent blocks).
For another example, the de-blocking filter 119 performs a coarse decision according to one of the evaluated visual quality and the checking result of at least one non-visual-quality based criterion to select M coarse de-blocking filtering settings (e.g., coarse boundary strength values for a block boundary between adjacent blocks) from all possible N de-blocking filtering settings (e.g., BS=0, BS=1, BS=2, BS=3 and BS=4), and performs a fine decision according to another of the evaluated visual quality and the checking result of at least one non-visual-quality based criterion to determine P fine de-blocking filtering settings from the coarse de-blocking filtering settings (N>M & M>P≧1), wherein a target de-blocking filtering setting (e.g., a boundary strength value actually assigned to the block boundary between adjacent blocks) is derived from the P fine de-blocking filtering settings. In a case where P=1, the target de-blocking filtering setting is directly determined by the fine decision based on the non-visual-quality checking result if the coarse decision is made based on the evaluated visual quality; or the target de-blocking filtering setting is directly determined by the fine decision based on the evaluated visual quality if the coarse decision is made based on the non-visual-quality checking result.
Step 500: Start.
Step 502: Evaluate visual quality based on data involved in a coding loop, wherein the data involved in the coding loop may be raw data of a source frame or processed data derived from the raw data of the source frame, and each evaluated visual quality may be obtained from a single visual quality metric or a composition of multiple visual quality metrics.
Step 504: Check if at least one non-visual-quality based criterion should be used for de-blocking filtering decision. If yes, go to step 506; otherwise, go to step 510.
Step 506: Check at least one non-visual-quality based criterion used for controlling de-blocking filtering, and accordingly generate a checking result.
Step 508: Refer to both of the evaluated visual quality and the checking result for performing the de-blocking filtering. For example, both of the evaluated visual quality and the checking result may be used for deciding the boundary strength of a block boundary between adjacent blocks. Go to step 512.
Step 510: Refer to the evaluated visual quality for performing the de-blocking filtering. For example, the evaluated visual quality may be used for deciding the boundary strength of a block boundary between adjacent blocks.
Step 512: End.
As a person skilled in the art can readily understand details of each step in
As mentioned above, the evaluated visual quality determined by the visual quality evaluation module 104 can be referenced by the de-blocking filter 119 during de-blocking filtering. However, this is not meant to be a limitation of the present invention. In a second application, the de-blocking filter 119 may be arranged to refer to the aforementioned visual quality determined by the visual quality evaluation module 104 for deciding a target coding parameter associated with de-blocking filtering, where the target coding parameter may be a de-blocking filter parameter. In addition, the target coding parameter set based on the evaluated visual quality may be included in the bitstream BS generated by encoding the source frame IMGIN. That is, the target coding parameter is a signaling parameter which is transmitted to a video decoding apparatus to facilitate the decoder-side video processing operation. As the visual quality evaluation performed by the visual quality evaluation module 104 has been detailed above, further description directed to obtaining the evaluated visual quality based on one or more visual quality metrics is omitted here for brevity.
In an alternative design, both of the evaluated visual quality (which is obtained based on data involved in the coding loop) and the checking result (which is obtained by checking at least one non-visual-quality criterion) are used to decide a target coding parameter (e.g., a de-blocking filter parameter) associated with de-blocking filtering, where the target coding parameter set based on the evaluated visual quality and the checking result may be included in the bitstream BS and transmitted to a video decoding apparatus.
For example, the de-blocking filter 119 refers to the evaluated visual quality to decide a first parameter setting with best visual quality, refers to the non-visual-quality checking result to decide a second parameter setting, and finally selects one of the first parameter setting and the second parameter setting to set the target coding parameter. For another example, the de-blocking filter 119 performs a coarse decision according to one of the evaluated visual quality and the non-visual-quality checking result to determine a plurality of coarse parameter settings, and performs a fine decision according to another of the evaluated visual quality and the non-visual-quality checking result to determine at least one fine parameter setting from the coarse parameter settings, wherein the target coding parameter (e.g., the de-blocking filter parameter) is derived from the at least one fine parameter setting.
Step 600: Start.
Step 602: Evaluate visual quality based on data involved in a coding loop, wherein the data involved in the coding loop may be raw data of a source frame or processed data derived from the raw data of the source frame, and each evaluated visual quality may be obtained from a single visual quality metric or a composition of multiple visual quality metrics.
Step 604: Check if at least one non-visual-quality based criterion should be used for coding parameter decision. If yes, go to step 606; otherwise, go to step 610.
Step 606: Check at least one non-visual-quality based criterion used for controlling de-blocking filtering, and accordingly generate a checking result.
Step 608: Refer to both of the evaluated visual quality and the checking result for deciding a target coding parameter (e.g., a de-blocking filter parameter) associated with de-blocking filtering in video coding. Go to step 612.
Step 610: Refer to the evaluated visual quality for deciding a target coding parameter (e.g., a de-blocking filter parameter) associated with de-blocking filtering in video coding.
Step 612: End.
As a person skilled in the art can readily understand details of each step in
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.
This application claims the benefit of U.S. provisional application No. 61/776,053, filed on Mar. 11, 2013 and incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6160846 | Chiang | Dec 2000 | A |
| 6233283 | Chiu | May 2001 | B1 |
| 7742532 | Jeon | Jun 2010 | B2 |
| 7873727 | Pal | Jan 2011 | B2 |
| 8077775 | He | Dec 2011 | B2 |
| 8111300 | Hwang | Feb 2012 | B2 |
| 8345777 | Lee | Jan 2013 | B2 |
| 9282328 | Chen | Mar 2016 | B2 |
| 20030128754 | Akimoto | Jul 2003 | A1 |
| 20030206587 | Gomila | Nov 2003 | A1 |
| 20030206664 | Gomila | Nov 2003 | A1 |
| 20040114817 | Jayant | Jun 2004 | A1 |
| 20040156559 | Cheng | Aug 2004 | A1 |
| 20040208392 | Raveendran | Oct 2004 | A1 |
| 20050243915 | Kwon | Nov 2005 | A1 |
| 20060114997 | Lelescu | Jun 2006 | A1 |
| 20060215766 | Wang | Sep 2006 | A1 |
| 20060238445 | Wang | Oct 2006 | A1 |
| 20080069247 | He | Mar 2008 | A1 |
| 20080117981 | Lee | May 2008 | A1 |
| 20080240252 | He | Oct 2008 | A1 |
| 20090323803 | Gomila | Dec 2009 | A1 |
| 20100220796 | Yin | Sep 2010 | A1 |
| 20100296588 | Fujii | Nov 2010 | A1 |
| 20110033119 | Rezazadeh | Feb 2011 | A1 |
| 20110211637 | Blum | Sep 2011 | A1 |
| 20110222607 | An | Sep 2011 | A1 |
| 20110235715 | Chien | Sep 2011 | A1 |
| 20110255589 | Saunders | Oct 2011 | A1 |
| 20110280321 | Chou | Nov 2011 | A1 |
| 20120082241 | Tsai | Apr 2012 | A1 |
| 20120163452 | Horowitz | Jun 2012 | A1 |
| 20120177104 | Budagavi | Jul 2012 | A1 |
| 20120201475 | Carmel | Aug 2012 | A1 |
| 20120257681 | Sato | Oct 2012 | A1 |
| 20120328004 | Coban et al. | Dec 2012 | A1 |
| 20120328029 | Sadafale | Dec 2012 | A1 |
| 20130051454 | Sze | Feb 2013 | A1 |
| 20130051455 | Sze | Feb 2013 | A1 |
| 20130083844 | Chong | Apr 2013 | A1 |
| 20130094569 | Chong | Apr 2013 | A1 |
| 20130094572 | Van der Auwera | Apr 2013 | A1 |
| 20130177068 | Minoo | Jul 2013 | A1 |
| 20130243090 | Li | Sep 2013 | A1 |
| 20130318253 | Kordasiewicz | Nov 2013 | A1 |
| 20130343447 | Chen | Dec 2013 | A1 |
| 20140002670 | Kolarov | Jan 2014 | A1 |
| 20140056363 | He | Feb 2014 | A1 |
| 20140160239 | Tian | Jun 2014 | A1 |
| 20140254659 | Ho | Sep 2014 | A1 |
| 20140254662 | Ho | Sep 2014 | A1 |
| 20140254663 | Ho | Sep 2014 | A1 |
| 20140254680 | Ho | Sep 2014 | A1 |
| 20140254689 | Ho | Sep 2014 | A1 |
| 20140321552 | He | Oct 2014 | A1 |
| 20140334559 | Kim | Nov 2014 | A1 |
| 20160044332 | Maaninen | Feb 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 1471319 | Jan 2004 | CN |
| 1669338 | Sep 2005 | CN |
| 1694500 | Nov 2005 | CN |
| 1695164 | Nov 2005 | CN |
| 101090502 | Dec 2007 | CN |
| 101232619 | Jul 2008 | CN |
| 101325711 | Dec 2008 | CN |
| 101489130 | Jul 2009 | CN |
| 102150429 | Aug 2011 | CN |
| 102415088 | Apr 2012 | CN |
| 102685472 | Sep 2012 | CN |
| 2013030833 | Mar 2013 | WO |
| 2013074365 | May 2013 | WO |
| Entry |
|---|
| “International Search Report” mailed on Jun. 30, 2014 for International application No. PCT/CN2014/073176, International filing date:Mar. 11, 2014. |
| “International Search Report” mailed on Jun. 23, 2014 for International application No. PCT/CN2014/073146, International filing date:Mar. 10, 2014. |
| “International Search Report” mailed on Jun. 13, 2014 for International application No. PCT/CN2014/073171, International filing date:Mar. 11, 2014. |
| “International Search Report” mailed on Jun. 3, 2014 for International application No. PCT/CN2014/073178, International filing date:Mar. 11, 2014. |
| “International Search Report” mailed on Jun. 18, 2014 for International application No. PCT/CN2014/073167, International filing date:Mar. 11, 2014. |
| Chikkerur et al., “Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison”, IEEE Transactions on Broadcasting, vol. 57, No. 2, Jun. 2011, p. 165-182. |
| Chih-Ming Fu et al., “Sample Adaptive Offest for HEVC”, 2011 IEEE. |
| Number | Date | Country | |
|---|---|---|---|
| 20140254662 A1 | Sep 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61776053 | Mar 2013 | US |
