The present invention relates generally to compression of moving video data, and more particularly to the application of quantization of the three-dimensional Discrete Cosine Transform (DCT) representation of moving video data for the purposes of removing visually redundant information.
In accordance with one aspect of the invention, a method is provided for removal of all subjectively redundant visual information by means of calculating optimal visually-weighed quantizers corresponding to the decorrelating-transformed block decomposition of a sequence of video images. The contrast sensitivity of the human eye to the actual time-varying transform-domain frequency of each transform component is calculated, and the resolution of the transformed data is reduced by the calculated sensitivity.
A second aspect of the invention applies specifically to use of the DCT as the decorrelating transform.
It is well established in the literature of the field of video compression that video can be well-modeled as a stationary Markov-1 process. This statistical model predicts the video behavior quite well, with measured correlations over 0.9 in the pixel and line directions.
It is well-known the Karhunen-Loeve Transform (KLT) perfectly decorrelates Markov-distributed video. This means the basis of the KLT is an independent set of vectors which encode the pixel values of the video sequence.
It is a further result that many discrete transforms well approximate the KLT for large correlation values. Perhaps the best-known such function is the DCT, although many other functions (DST, WHT, etc.) serve as reasonable approximations to the KLT.
It is for this reason the DCT is used to decorrelate images in the JPEG standard, after which a uniform quantization factor individually chosen for each DCT component is applied to said component, removing visual information imperceptible to the human eye.
What is needed is a means of removing subjectively redundant video information from a moving sequence of video.
Many prior-art techniques are taught under the principle of guiding a design of a quantization matrix to provide optimum visual quality for a given bitrate. These techniques, being applicable to motion compensation-based compression algorithms, require a Human Visual Model-driven feedback loop to converge on the quantizers that will show minimal artifact on reconstruction. The use of this Human Visual Model is again limited to its application in the spatial domain. An example of this teaching is U.S. Pat. No. 8,326,067 by Furbeck, as illustrated in
The wavelet transform is another technique commonly used to perform compression. However, the wavelet does not decorrelate video, and thus optimal quantizers based upon a Human Visual Model cannot be calculated. A teaching by Gu et al, U.S. Pat. No. 7,006,568 attempts to address this issue by segmenting video sequences into similar-characteristic segments and calculating 2-D quantizers for each selected segment, chosen to reduce perceptual error in each subband, as illustrated in
The current invention improves the compression process by directly calculating the visually optimal quantizers for 3-D transform vectors by evaluating the basis behavior of the decorrelated transform space under a time-varying Human Visual Model, as represented by a Contrast Sensitivity Function.
As illustrated in
In the current embodiment, said configuration of video stream 5320 is elaborated in
In the current embodiment, said configuration of viewing conditions 5310 is elaborated in
In the current embodiment, said configuration of block-based decorrelating transform 5340 is elaborated in
In the current embodiment, said configuration of quantizer algorithm 5330 is elaborated in
Luminance quantizers are calculated as in
The equation 8110 of
The equation 9010 of
Equation 10010 of
The two-dimensional map of values assumes by said typical contrast sensitivity function CSF(u,w,l,X0,Xmax) (7010) for equally-weighted is depicted in
As illustrated in
Said quantizer Q (8020) gives optimal response for pure AC transform components, but produces sub-optimal results for pure DC or mixed AC/DC components, due to the extreme sensitivity of the human eye to DC levels. Pure DC transform components may be quantized by the value that the variance of the DC component is concentrated over the number of possible levels that can be represented in the reconstructed image, as the human eye is constrained to the capabilities of the display. Equation 15010 of
Mixed AC/DC components can be quantized by the minimum quantization step size apportioned over the variance of the DCT basis component. This process requires calculation of the per-component variance for the AC and DC components (i.e., the variance calculation in the number of dimensions in which each AC or DC component resides). Similarly, the value of the independent AC and DC quantizers must be calculated using the Contrast Sensitivity Function limited to the number of dimensions in which the AC or DC component resides. As illustrated in
The two-dimensional AC quantizer QACm,n,0 15220 is calculated directly from said typical generalized Contract Sensitivity Function CSF(u,w,l,X0,Xmax) 7010.
The maximum visual delta of 1/QACm,n,0 16110 calculated to apply to the variance-concentrated range Cx[m,m]*Cy[n,n] 16120 and 1/QDCm,n,0 16130 calculated to apply to the variance-concentrated range Cz[0,0] 16130 is calculated as 1/min(QACm,n,0,QDCm,n,0) 16210, and can be applied over the entire range Cx[m,m]*Cy[n,n]*Cz[0,0] 16220.
Said statistically optimal quantizer Q m,n,0 16310 may now be calculated following with the C language pseudocode excerpt 16320. It is to be understood that the process of calculating typical statistically ideal mixed AC/DC coefficients is illustrated in the general sense in
The worst-case degradation in visual quality caused by the Gibbs phenomenon as a result of quantization is illustrated in
This application is a continuation of U.S. application Ser. No. 14/266,757, filed on Apr. 30, 2014, which is a non-provisional and claims benefit of a U.S. Provisional Application No. 61/818,419, filed on May 1, 2013.
Number | Name | Date | Kind |
---|---|---|---|
4780762 | Nagasaki | Oct 1988 | A |
4894713 | Delogne et al. | Jun 1990 | A |
5629780 | Watson | May 1997 | A |
5737020 | Hall et al. | Apr 1998 | A |
5751358 | Suzuki et al. | May 1998 | A |
5768534 | Guillote et al. | Jun 1998 | A |
5933193 | Niesen | Aug 1999 | A |
5936669 | Niesen | Aug 1999 | A |
5953460 | Wu | Sep 1999 | A |
6023531 | Peters | Feb 2000 | A |
6222887 | Nishikawa et al. | Apr 2001 | B1 |
6239811 | Westwater | May 2001 | B1 |
6252989 | Geisler | Jun 2001 | B1 |
6256413 | Hirabayashi | Jul 2001 | B1 |
6314208 | Konstantinides | Nov 2001 | B1 |
6347157 | Chui | Feb 2002 | B2 |
6456744 | Lafe | Sep 2002 | B1 |
6801573 | Zheng | Oct 2004 | B2 |
6894628 | Marpe et al. | May 2005 | B2 |
6990247 | Schwartz | Jan 2006 | B2 |
7006568 | Gu | Feb 2006 | B1 |
7076104 | Keith et al. | Jul 2006 | B1 |
7123656 | Reznik | Oct 2006 | B1 |
7599840 | Mehrotra et al. | Oct 2009 | B2 |
7634145 | Keith et al. | Dec 2009 | B2 |
7684981 | Thumpudi et al. | Mar 2010 | B2 |
7889790 | Sun | Feb 2011 | B2 |
8179974 | Tu et al. | May 2012 | B2 |
8236068 | Aramata et al. | Aug 2012 | B2 |
8306340 | Ceperkovic et al. | Nov 2012 | B2 |
8326067 | Furbeck | Dec 2012 | B2 |
8351500 | Shimofure | Jan 2013 | B2 |
8432967 | Hosaka et al. | Apr 2013 | B2 |
8436755 | Korodi et al. | May 2013 | B2 |
20010004400 | Aoki et al. | Jun 2001 | A1 |
20020168022 | Kim | Nov 2002 | A1 |
20020186772 | Li et al. | Dec 2002 | A1 |
20030090591 | Concion et al. | May 2003 | A1 |
20030112333 | Chen | Jun 2003 | A1 |
20030142098 | Kim et al. | Jul 2003 | A1 |
20030169928 | Stanek | Sep 2003 | A1 |
20030190086 | Kim | Oct 2003 | A1 |
20040001611 | Celik et al. | Jan 2004 | A1 |
20040061709 | Park et al. | Apr 2004 | A1 |
20040062448 | Zeng | Apr 2004 | A1 |
20040066974 | Karczewicz et al. | Apr 2004 | A1 |
20040105590 | Akimoto et al. | Jun 2004 | A1 |
20040151243 | Bhaskaran | Aug 2004 | A1 |
20040190610 | Song et al. | Sep 2004 | A1 |
20040208362 | Suzuki | Oct 2004 | A1 |
20050008224 | Caruso et al. | Jan 2005 | A1 |
20050105608 | Coleman et al. | May 2005 | A1 |
20050157791 | Sun | Jul 2005 | A1 |
20050165517 | Reich | Jul 2005 | A1 |
20050190836 | Lu | Sep 2005 | A1 |
20050207660 | Edgar | Sep 2005 | A1 |
20050281479 | Song | Dec 2005 | A1 |
20050286629 | Dumitras et al. | Dec 2005 | A1 |
20060062304 | Hsia | Mar 2006 | A1 |
20060098897 | Dewaele | May 2006 | A1 |
20060155531 | Miller | Jul 2006 | A1 |
20060164268 | Lee et al. | Jul 2006 | A1 |
20060165311 | Watson | Jul 2006 | A1 |
20060292837 | Gomila et al. | Dec 2006 | A1 |
20070018906 | Visnovsky et al. | Jan 2007 | A1 |
20070071090 | Peng et al. | Mar 2007 | A1 |
20070081589 | Kim et al. | Apr 2007 | A1 |
20070121731 | Tanizawa et al. | May 2007 | A1 |
20070195737 | Walker | Aug 2007 | A1 |
20070230565 | Tourapis et al. | Oct 2007 | A1 |
20070263938 | Lee | Nov 2007 | A1 |
20070273939 | Kishida et al. | Nov 2007 | A1 |
20080071764 | Omi et al. | Mar 2008 | A1 |
20080186348 | Yanai et al. | Aug 2008 | A1 |
20080192824 | Lim et al. | Aug 2008 | A1 |
20080298636 | Rasheed et al. | Dec 2008 | A1 |
20090092099 | Gu | Apr 2009 | A1 |
20090232408 | Meany | Sep 2009 | A1 |
20100054342 | Jang et al. | Mar 2010 | A1 |
20100286991 | Hedelin et al. | Nov 2010 | A1 |
20100322597 | Gharavi-Alkhansari et al. | Dec 2010 | A1 |
20100324850 | Jung et al. | Dec 2010 | A1 |
20100329329 | Reznik et al. | Dec 2010 | A1 |
20110075897 | Dekel et al. | Mar 2011 | A1 |
20110255589 | Saunders | Oct 2011 | A1 |
20110268182 | Joshi | Nov 2011 | A1 |
20110286525 | Kamisli et al. | Nov 2011 | A1 |
20120057629 | Shi | Mar 2012 | A1 |
20120133361 | Gross | May 2012 | A1 |
20120140819 | Kim et al. | Jun 2012 | A1 |
20130003829 | Misra et al. | Jan 2013 | A1 |
20130089137 | Korman et al. | Apr 2013 | A1 |
20130114703 | DeForest et al. | May 2013 | A1 |
20130195206 | McCarthy | Aug 2013 | A1 |
20130287104 | Jeong et al. | Oct 2013 | A1 |
20130336594 | Dorairaj | Dec 2013 | A1 |
20140003728 | Aoki et al. | Jan 2014 | A1 |
20140051108 | Bernardi et al. | Feb 2014 | A1 |
20140115472 | Mochinaga et al. | Apr 2014 | A1 |
20140169407 | Terry | Jun 2014 | A1 |
20140267432 | Dong et al. | Sep 2014 | A1 |
20140270331 | Zhao et al. | Sep 2014 | A1 |
20140286434 | Lee et al. | Sep 2014 | A1 |
20140327737 | Westwater | Nov 2014 | A1 |
20140328406 | Westwater | Nov 2014 | A1 |
20140376620 | Kondo | Dec 2014 | A1 |
20150003707 | Amon et al. | Jan 2015 | A1 |
20160309189 | Westwater | Oct 2016 | A1 |
20160309190 | Westwater | Oct 2016 | A1 |
Entry |
---|
Nadenau et al., “Wavelet-Based Color Image Compression: Exploiting the Contrast Sensitivity Function,” IEEE Trans. Image Processing vol. 12, No. 1, Jan. 2003, pp. 58-70. |
Nadenau, “Wavelet-Based Color Image Compression: Exploiting the Contrast Sensitivity Function”, IEEE Transactions on Image Porcessing, vol. 12, No. 1, Jan. 2003, pp. 58-70. |
Office action for U.S. Appl. No. 15/189,242, dated Nov. 23, 2016, Westwater, “Method and Apparatus to Perform Correlation-Based Entropy Removal From Quantized Still Images or Quantized Time-Varying Video Sequences in Transform”, 46 pages. |
Office action for U.S. Appl. No. 14/266,757, dated Sep. 1 2016, Westwater, “Method and Apparatus to Perform Optimal Visually-Weighed Quantization of Time-Varying Visual Sequences in Transform Space”, 20 pages. |
Office action for U.S. Appl. No. 14/266,645, dated Sep. 28, 2016, Westwater, “Method and Apparatus to Perform Correlation-Based Entropy Removal from Quantized Still Images or Quantized Time-Varying Video Sequences in Transform”, 30 pages. |
Office action for U.S. Appl. No. 15/189,187, dated Sep. 8, 2016, Westwater, “Method and Apparatus to Perform Optimal Visually-Weighed Quantization of Time-Varying Visual Sequences in Transform Space”, 20 pages. |
Number | Date | Country | |
---|---|---|---|
20160309189 A1 | Oct 2016 | US |
Number | Date | Country | |
---|---|---|---|
61818419 | May 2013 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 14266757 | Apr 2014 | US |
Child | 15189187 | US |