Patent Application
20120044989
The subject matter disclosed herein relates generally to techniques for identifying and reducing blocking artifacts.
Video compression is employed for storage and transmission of video sequences to increase storage efficiency and to reduce bandwidth used to transmit video. Popular video standards such as MPEG and H.264 employ block-based compression techniques to achieve a reduction of bit-rate. In such techniques, each frame of the video is first partitioned into a set of disjointed blocks of fixed size. Advanced compression schemes are then applied to reduce the number of bits to code each block. An unwanted consequence of compression is the presence of visible blocking artifacts in the decoded video. In some cases, the greater the amount of compression of the video sequence, the stronger the blocking artifacts.
De-blocking is a post-processing process whose goal is to reduce or remove blocking artifacts and improve the visual quality of the decoded video. Some existing de-blocking techniques assume that the blocks are of a known size (such as 8×8 pixels) and/or that information about the strength of the blocking artifacts is available from the decoder in the form of quantization parameters. However, in video post-processing, block size and/or quantization parameters may not always be available from the decoder. Moreover, the size of the blocks may not necessarily be fixed and may depend on several factors, including:
Accordingly, such techniques may not perform effective de-blocking of images or videos containing blocks of non-standard sizes or those containing shifted blocks due to motion compensation in compressed video sequences. In some cases, there may be residual blockiness where block artifacts were not detected or the output image may suffer from loss of detail resulting from applying smoothing throughout the image.
Various techniques to perform de-blocking are described in the following articles: Hoon Paek, Rin-Chul Kim, and Sang-Uk Lee, “On the POCS-Based Postprocessing Technique to Reduce the Blocking Artifacts in Transform Coded Images,” IEEE Transactions On Circuits And Systems For Video Technology, Vol. 8, No. 3, June 1998, pp. 358-367.
Amir Z. Averbuch, Alon Schclar, and David L. Donoho, “Deblocking of Block-Transform Compressed Images Using Weighted Sums of Symmetrically Aligned Pixels,” IEEE Transactions On Image Processing, Vol. 14, No. 2, February 2005, pp. 200-212.
Shuanhu Wu, Hong Yan, and Zheng Tan, “An Efficient Wavelet-Based Deblocking Algorithm for Highly Compressed Images,” IEEE Transactions On Circuits And Systems For Video Technology, Vol. 11, No. 11, November 2001.
Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the drawings and in which like reference numerals refer to similar elements.
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase “in one embodiment” or “an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in one or more embodiments.
Various embodiments attempt to reduce blocking artifacts in compressed images and videos. Various embodiments estimate the location and strength of the blocking artifacts in each image and attempt to distinguish between blocks in smooth regions of an image and blocks in regions of high detail. Consequently, blockiness can be reduced in compressed images and videos that have smooth or detailed regions or non-uniform sized blocks while at the same time retaining the level of sharpness of the input.
Blocking artifacts caused by compression can be modeled as true step-edges in either the horizontal or the vertical directions. The location of blocking artifacts can be determined by searching for such step edges. Accordingly, edges in natural images, which are rarely true steps, may not be incorrectly detected as blocks by this technique. Nonetheless, constraints are placed to attempt to detect true block artifacts with a high degree of confidence. Once the location of the artifacts is known, the strength of blockiness at each location is measured. The strength of blockiness can be used to control the parameters of an adaptive de-blocking filter or used in other ways.
G
H
[m,n]=|Y[m,n]−Y[m,n−1]|,
The horizontal gradient is an absolute value of a difference between adjacent pixels in the same row. The step-size of a block edge at (m,n) is the value of horizontal gradient GH at (m,n). Block 102 repeats for every pixel coordinate in a row to determine the horizontal gradient for every pixel in a row.
To determine a vertical gradient, an absolute value of a difference between adjacent pixels in the same column is determined.
Block 104 commences after the horizontal gradients of all pixels in a row have been determined. Block 104 includes determining a low-pass filtered version of horizontal gradients in the row of the pixel at location [m,n]. Applying a low-pass filter aids in the detection of step-edges that are representative of blocking artifacts. A low-pass filtered version of each pixel in the same row as that of the pixel at location [m,n] can be calculated using:
where
For detection and measurement of horizontal block edges, block 104 can be used except that a low pass filtered version of each pixel in the same column as that of the pixel at location [m,n] is calculated.
Block 106 includes determining a pixel-wise change in intensity for pixels in the same row. For example, a pixel-wise ratio of the horizontal gradient to the low-pass filtered version of the horizontal gradient can be determined. The pixel-wise ratio of the horizontal gradient, GH[m,n], to the low-pass filtered version of pixels in a row, AH[m,n], can be determined as follows:
Block 108 includes determining whether the ratio exceeds a threshold. Sudden changes in intensity in an image can be due to block artifacts or changes in textures in an image. It can be proven mathematically that the maximum value of the pixel wise ratio determined in block 106 is 2N+1 and is achieved for true step edges, where N is an integer. A value of N=2 is found to give satisfactory detection of block-edges. Many real world images may contain some amount of noise. In such case, a block-edge can be modeled as a step edge corrupted by additive noise. It can be proven that the value of Rv[m,n] for such corrupted edges is lower than the theoretical maximum. Accordingly, instead of identifying the pixels where Rv[m,n] attains its maximum value, a search is conducted for pixels where Rv[m,n] is greater than some threshold T, where T≦2N+1. These pixels are then identified as potential block edges. Accordingly, if the pixel-wise ratio is greater than a threshold T, then block 110 follows block 108. If the pixel-wise ratio is less than or equal to a threshold T, then block 109 follows block 108.
In some applications, T can be set to a fixed value specified by a user. Threshold T can be set as a function of the noise level in the image (which is either known or measured by other means). In video applications, T may be adaptively set to a value for each input video frame depending on the noise level in each frame.
For images or videos having only compression artifacts, the choice of T involves a trade-off between missed detections (true block artifacts that are not detected) and false alarms (pixels that are false detected as having block artifacts). The higher the value of T, the fewer the false alarms. However, there is a greater chance that true block-edges will be left out. For N=2, a value of T=4 is found to give satisfactory results. The presence of random noise in the compressed image sequence may adversely impact the performance of the blockiness detector. For images or video that contain both compression and random noise, the mean or expected value of RV[m,n] for step-edges of size h is given by:
Thus, in general the presence of random noise lowers the threshold T that should be used for detection of block edges. Using the above relation, T can be set as a function of the noise level in the image. The noise level itself may either be known or can be measured such as by estimating the standard deviation of the image in flat regions. In video applications, T may be adaptively set to a value for each input video frame depending on the noise level in each frame.
Block 109 indicates that the pixel at coordinates m and n do not include a blocking artifact.
Block 110 includes selecting block edges whose length meets a threshold length. Amongst the pixels identified by block 108, it is likely that there will be some isolated false positives. To eliminate these isolated false positives, block 110 performs a consistency check to determine if the adjacent pixels that may include blocking artifacts and are in the same column have a minimum length, CLENGTH. If the adjacent detected pixels have length less than CLENGTH, then these detected pixels are treated as false classifications. In some embodiments, a pixel location (m, n) is classified as a vertical block edge location if the following is true:
(m−k+1,n),(m−k+2,n), . . . ,(m−k+CLENGTH,n) are all potential block edges for any kε{1,2, . . . ,CLENGTH}.
For detection and measurement of horizontal block edges, adjacent detected pixels that include blocking artifacts and are in the same row are compared against value CLENGTH.
Fixing a value of CLENGTH involves a tradeoff between false positives and missed block-edges. Setting CLENGTH to 1 implies that every potential block-edge detected by block 108 is classified as a true block-edge. Setting CLENGTH to high values may result in some true block-edges not being classified as block-edges.
Block 112 forms a set of pixels that potentially have block artifacts (denoted ΩV). The pixels that pass the tests of blocks 108 and 110 are included in the set of pixels that potentially have block artifacts.
A textured image region can include many changes in color and image intensity and block boundary detection may be more difficult in textured image regions than in smooth image regions. In textured regions, block artifacts may not always manifest themselves as step edges and hence may not be detected using the process of
Block 202 includes determining a number of pixel locations in each column in which vertical blocking artifacts are present. The pixel locations in which vertical blocking artifacts are present can be provided from block 112 of
“Number of block artifacts in column”/“length of column”
The number of pixel locations in each column in which vertical block artifacts are present can be a number of pixels identified in block 112 of
Block 204 includes determining whether the ratio for the current column is greater than the ratios of previous and next columns. Block 204 determines if the current column is a spike relative to adjacent columns. A spike is a local maximum. A local maximum represents that column has blocking artifacts. BV[k−1] represents a ratio of the previous column and BV[k+1] represents a ratio of the next column. Block 204 includes determining if BV[k] is larger than both columns BV[k−1] and BV[k+1]. Some pixels of columns BV[k−1] and BV[k+1] could contain pixels that have blocking artifacts. Attempts are made to identify columns that have a greater percentage of pixels that have blocking artifacts as compared to their neighbors. If the ratio for the current column is greater than the ratios of previous and next columns, then block 206 follows block 204. If the ratio for the current column is not greater than the ratios of previous and next columns, then block 205 follows block 204.
Other types of comparisons can be performed such as whether the ratio for the current column is some percentage greater than the ratios of adjacent columns.
Block 206 includes determining whether the current column has a ratio BV[k] that is greater than a threshold TBV. If the threshold is met or exceeded, then the column is considered to include a blocking artifact.
If the image has been a priori classified into smooth or textured regions via a segmentation process, then this threshold may be set in a region-adaptive fashion. In the absence of such information, a single threshold may be set for the entire image. Such a global threshold can be determined experimentally or may automatically be assigned some reasonable value such as 0.15 (15%). In some embodiments, for each column whose ratio exceeds a threshold, block 208 follows block 206. In some embodiments, for each column whose ratio does not exceed a threshold, block 205 follows block 206.
Block 208 aggregates a set of columns whose local ratio of BV[k] exceeds a threshold, TBV. The columns in this set are the ones most likely to correspond to true blocking artifacts. In this set of columns, there will be locations that were not detected as block artifacts by the procedure of
For colored images, detection schemes can be applied to each individual color plane. A colored image includes three color planes (RGB, or YUV). The procedure described here applies to a single color plane (say R of RGB, or Y of YUV). The entire process can be repeated for each of the colored planes.
Thus, by applying the processes of
The following describes a manner to determine strength of blocking artifacts. If a potential block edge is detected at location (m,n), then the step-size of that block edge is used to classify the edge as STRONG, MEDIUM, or WEAK. The step-size of the block edge at (m,n) can be the value of the gradient GH at (m,n) (determined in block 102 of
The default value of LOW_TH is chosen to be 6 and the default value of MED_TH is chosen to be 16. Variable BCLASS stores the strengths of the detected blocks.
The strength of the block artifacts can be used to set the strength of a de-blocking filter. This may be done on a per-image basis by determining the average strength of blockiness in an image, or may be done on a per-pixel basis by using the strength of blockiness at each pixel to adaptively determine the strength of deblocking for that pixel.
The average value of the gradient GH[m,n] can be calculated for pixels that have been classified as strong block artifacts (BCLASS[m,n]=STRONG). This average value, which is denoted as VSTRONG_METRIC, represents the average strength of blockiness for pixels that are classified as strong vertical block artifacts. Variable VSTRONG_METRIC can be used in the filtering of strong blocking artifacts. For weak strength artifacts, the value of LOW_TH=6 is used and for medium strength artifacts, the value of MEDIUM_TH=16 is used.
A de-blocking filter is to convert a sharp block edge to a smoother ramp edge. If a block artifact is present at location (m,n), then the de-blocking filter can be applied to a neighborhood of pixels at that location of the detected block artifacts. For vertical block artifacts, this neighborhood of pixels is defined as follows:
N
V(m,n)={(m,n+k),−FWIDTH≦k≦FWIDTH}.
The default value of FWIDTH is chosen to be four (4), although other values can be used. Similarly, for horizontal block artifacts, a neighborhood of pixels are vertically offset from the pixel of interest.
If the strength of the blocking artifact at (m,n) is given by BCLASS[m,n], then all the points in the neighborhood of pixels Nv(m,n) are also classified as having strength BCLASS[m,n]. In the event that a pixel (x, y) belongs to the neighborhood of more than one block artifact, then, the conflict is resolved by giving the higher strength to pixel (x,y). The following pseudo-code can be applied.
If BCLASS [m1,n1]>BCLASS [m2,n2]
else
end.
The complete set of points where the vertical de-blocking filter is applied therefore includes the points where a block artifact has been detected, along with the corresponding neighborhoods of such points. Mathematically,
Removal of blocking artifacts can involve applying a bilateral filter in a neighborhood of points around the pixel locations with detected blocking artifacts. The parameters of the bilateral filter applied at a particular pixel location are adaptively determined by the strength of the blocking artifact present at that location (i.e., weak, medium, or strong). One possible procedure is described next.
For each pixel in the set of pixels that potentially has block artifacts (ΩV), a bilateral filter is applied along the horizontal direction (for a row of pixels) to obtain the output filtered pixel value yo[m,n] using the following relationship:
where,
w
d(n,k)=e−(n−k)
w
r(x[m,n]·x[m,k])=e−(x[m,n]−x[m,k])
The amount of smoothing produced by the filter is controlled by the estimated artifact strength. The filter is designed such that step edges of strengths comparable to the estimated artifacts strength are filtered out (thereby reducing blockiness) but edges of greater strength are effectively left intact.
It was determined experimentally that σd=1.67 results in superior visual quality of the filtered output. Other values can be used.
Value σr can be chosen as follows:
VSTRONG_METRIC represents the average strength of gradient for pixels that are classified as being strong vertical block artifacts.
The removal of horizontal block artifacts involves the same filtering operations as those for vertical block artifacts, except that 1-D row filtering is replaced by 1-D column filtering.
It is known that the block artifacts are most visible in smooth regions and that the presence of textures/fine details masks to a certain degree the presence of blocking artifacts. Thus, based on the information obtained by the above procedure, the strength of a deblocking filter could be made stronger in smooth regions and reduced in textured details. This has the effect of significantly reducing the perception of blockiness while at the same time retaining the level of detail in the image or video.
If a de-blocking filter can be used between the decoder and encoder of a transcoder, the quality of the transcoded output be improved for a given compression level/bitrate.
Similar to vertical block-edge classification in blocks 302-308, respective blocks 312-318 perform operations to identify horizontal block-edge classifications for horizontal block edges. Vertical gradient block 312 can determine vertical gradient of a pixel. Gradient smoothing block 314 can apply low pass filtering on a vertical gradient. Ratio block 316 can determine a ratio of the vertical gradient over a low pass filtered version of the gradient in a similar manner to that of block 306. Horizontal block-edge classifier 318 can determine whether a row of pixels include a blocking artifact by performing operations similar to those described with regard to blocks 108, 110, and 112.
For horizontal block-edges, horizontal block-artifact counter 412 can determine a number of pixels in a row that include blocking artifacts. Maxima detector 414 can determine whether the row that includes blocking artifacts has a local maximum of number of blocking artifacts relative to adjacent rows. Textured block-edge classifier 416 can classify the row as having a blocking artifact if a ratio of number of blocking artifacts in the row to the length of the row exceeds a threshold.
In interlaced video, two fields captured at different time instants are woven together to form a single interlaced frame. Interlaced content presents additional challenges to de-blocking.
Filtering of horizontal blocking artifacts involves filtering in the vertical direction. Such an operation may not be performed on an interlaced frame (a frame in which both odd and even fields are interleaved). Filtering can take place on a single field, as opposed to a frame of interleaved fields, but there is a risk of causing excessive smoothing. This is because points that appear one pixel apart in the field are actually two pixels apart in the de-interlaced image. Thus, it may be desirable to perform vertical filtering after the de-interlacing operation.
On the other hand, de-interlacing can involve some type of interpolation of the missing field. A result of this is that blocking artifacts that appeared as true step-edges in a field may not be true step-edges after the de-interlacing operation is carried out. This may reduce the accuracy of detection of block artifacts. Thus, it may be preferable to perform detection before the de-interlacing operation.
Processor 510 may be implemented as Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors, x86 instruction set compatible processors, multi-core, or any other microprocessor or central processing unit.
Host memory 512 may be implemented as a volatile memory device such as but not limited to a Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), or Static RAM (SRAM). Storage 514 may be implemented as a non-volatile storage device such as but not limited to a magnetic disk drive, optical disk drive, tape drive, an internal storage device, an attached storage device, flash memory, battery backed-up SDRAM (synchronous DRAM), and/or a network accessible storage device.
Graphics subsystem 515 may perform processing of images such as still or video for display. An analog or digital interface may be used to communicatively couple graphics subsystem 515 and display 522. For example, the interface may be any of a High-Definition Multimedia Interface, DisplayPort, wireless HDMI, and/or wireless HD compliant techniques. Graphics subsystem 515 could be integrated into processor 510 or chipset 505. Graphics subsystem 515 could be a stand-alone card communicatively coupled to chipset 505. In various embodiments, processor 510 and/or graphics subsystem 515 performs instructions that identify blocking artifacts and potentially correct blocking artifacts based on techniques described herein.
Radio 520 may include one or more radios capable of transmitting and receiving signals in accordance with applicable wireless standards such as but not limited to any version of IEEE 802.11 and IEEE 802.16. For example, radio 520 may include at least a physical layer interface and media access controller.
The graphics and/or video processing techniques described herein may be implemented in various hardware architectures. For example, graphics and/or video functionality may be integrated within a chipset. Alternatively, a discrete graphics and/or video processor may be used. As still another embodiment, the graphics and/or video functions may be implemented by a general purpose processor, including a multi-core processor. In a further embodiment, the functions may be implemented in a consumer electronics device.
Embodiments of the present invention may be implemented as any or a combination of: one or more microchips or integrated circuits interconnected using a motherboard, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, an application specific integrated circuit (ASIC), and/or a field programmable gate array (FPGA). The term “logic” may include, by way of example, software or hardware and/or combinations of software and hardware.
Embodiments of the present invention may be provided, for example, as a computer program product which may include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines such as a computer, network of computers, or other electronic devices, may result in the one or more machines carrying out operations in accordance with embodiments of the present invention. A machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs (Compact Disc-Read Only Memories), magneto-optical disks, ROMs (Read Only Memories), RAMs (Random Access Memories), EPROMs (Erasable Programmable Read Only Memories), EEPROMs (Electrically Erasable Programmable Read Only Memories), magnetic or optical cards, flash memory, or other type of media/machine-readable medium suitable for storing machine-executable instructions.
The drawings and the forgoing description gave examples of the present invention. Although depicted as a number of disparate functional items, those skilled in the art will appreciate that one or more of such elements may well be combined into single functional elements. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, orders of processes described herein may be changed and are not limited to the manner described herein. Moreover, the actions of any flow diagram need not be implemented in the order shown; nor do all of the acts necessarily need to be performed. Also, those acts that are not dependent on other acts may be performed in parallel with the other acts. The scope of the present invention, however, is by no means limited by these specific examples. Numerous variations, whether explicitly given in the specification or not, such as differences in structure, dimension, and use of material, are possible. The scope of the invention is at least as broad as given by the following claims.
