Patent Application
20130235942
In video coder/decoder systems, a video coder may code a source video sequence into a coded representation that has a smaller bit rate than does the source video and, thereby may achieve data compression. The video coder may code processed video data according to any of a variety of different coding techniques to achieve compression. One common technique for data compression uses predictive coding techniques (e.g., temporal/motion predictive coding). For example, some frames in a video stream may be coded independently (I-frames) and some other frames (e.g., P-frames or B-frames) may be coded using other frames as reference frames. P-frames may be coded with reference to a single previously coded frame (called, a “reference frame”) and B-frames may be coded with reference to a pair of previously-coded reference frames, typically a reference frame that occurs prior to the B-frame in display order and another reference frame that occurs subsequently to the B-frame in display order. The resulting compressed sequence (bit stream) may be transmitted to a decoder via a channel. To recover the video data, the bit stream may be decompressed at the decoder by inverting the coding processes performed by the coder, yielding a recovered video sequence.
To achieve high compression, the video coding processes typically are “lossy;” they permit a video decoder to recover a video sequence that is a replica of the source video sequence but has some errors. Thus, video coding systems often produce images with various types of coding artifacts including loss of detail, blockiness, ringing and banding. Such artifacts generally are more noticeable in still image content than in image content that exhibits a high degree of motion. Designers of video coding systems endeavor to provide coding systems that maintain high quality at appropriate bitrates and, therefore, avoid such display artifacts.
Embodiments of the present invention provide video coding systems and methods that protect against banding artifacts in decoded image content. According to the method, a video coder may identify from the content of pixel blocks of a frame of video data which pixel blocks are likely to exhibit banding artifacts from the video coding/decoding processes. The video coder may identify regions of the frame that are likely to exhibit banding artifacts based on the identified pixel blocks' locations with respect to each other. The video coder may apply anti-banding processing to pixel blocks within one or more of the identified regions and, thereafter, may code the processed frame by a compression operation.
In
The video source 210 may provide video to be coded by the rest of the system 200. In a media serving system, the video source 210 may be a storage device storing previously prepared video. In a videoconferencing system, the video source 210 may be a camera that captures local image information as a video sequence. Video data typically is provided as a plurality of individual frames that impart motion when viewed in sequence. The frames themselves typically are organized as a spatial array of pixels.
The pre-processor 220 may perform various analytical and signal conditioning operations on video data. The pre-processor 220 may parse input frames into color components (for example, luminance and chrominance components) and also may parse the frames into pixel blocks, spatial arrays of pixel data, which may form the basis of further coding. The pre-processor 220 also may apply various filtering operations to the frame data to improve efficiency of coding operations applied by a video coder 230.
The pre-processor 220 also may search for video content in the source video sequence that is likely to generate artifacts when the video sequence is coded by the system 200, then decoded and displayed by another terminal. “Banding” is one of the artifacts that the pre-processor 220 may identify. Banding may occur in areas of source frames that are generally smooth and exhibit a gradual transition within the area. When such frames are coded, then decoded and displayed, a gradual transition may not be preserved; instead, the transition may be displayed as a plurality of discrete changes in the area of the reconstructed frame. The pre-processor 220 may identify to the controller 250 portions of the video sequence in which banding artifacts may arise.
The video coder 230 may perform coding operations on the video sequence to reduce the video sequence's bit rate. The video coder 230 may include a coding engine 232, a local decoder 233, a reference picture cache 234, a predictor 235 and a controller 236. The coding engine 232 may code the input video data by exploiting temporal and spatial redundancies in the video data and may generate a datastream of coded video data, which typically has a reduced bit rate as compared to the datastream of source video data. As part of its operation, the video coder 230 may perform motion compensated predictive coding, which codes an input frame predictively with reference to one or more previously-coded frames from the video sequence that were designated as “reference frames.” In this manner, the coding engine 232 codes differences between pixel blocks of an input frame and pixel blocks of reference frame(s) that are selected as prediction reference(s) to the input frame.
The local decoder 233 may decode coded video data of frames that are designated as reference frames. Operations of the coding engine 232 typically are lossy processes. When the coded video data is decoded at a video decoder (not shown in
The predictor 235 may perform prediction searches for the coding engine 232. That is, for a new frame to be coded, the predictor 235 may search the reference picture cache 234 for image data that may serve as an appropriate prediction reference for the new frames. The predictor 235 may operate on a pixel block-by-pixel block basis to find appropriate prediction references. In some cases, as determined by search results obtained by the predictor 235, an input frame may have prediction references drawn from multiple frames stored in the reference picture cache 234.
The controller 236 may manage coding operations of the video coder 230, including, for example, selection of coding parameters to meet a target bit rate of coded video. Typically, video coders operate according to constraints imposed by bit rate requirements, quality requirements and/or error resiliency policies; the controller 236 may select coding parameters for frames of the video sequence in order to meet these constraints. For example, the controller 236 may assign coding modes and/or quantization parameters to frames and/or pixel blocks within frames.
The transmitter 240 may buffer coded video data to prepare it for transmission to the far-end terminal (not shown). The transmitter 240 may merge coded video data from the video coder 230 with other data to be transmitted to the terminal, for example, coded audio data and/or ancillary data streams (sources not shown).
The controller 250 may manage operation of the system 200. During coding, the controller 250 may assign to each frame a certain frame type (either of its own accord or in cooperation with the controller 236), which can affect the coding techniques that are applied to the respective frame. For example, frames often are assigned as one of the following frame types:
As discussed above, the pre-processor 310 may parse each frame into pixel blocks. The pre-processor 310 also may identify regions within frames in which banding is likely to occur and it may identify such regions to a controller 370.
The block-based coder 320 may include a subtractor 321, a transform unit 322, a quantizer 323 and an entropy coder 324. The subtractor 321 may generate data representing a difference between the source pixel block and a reference pixel block developed for prediction. The subtractor 321 may operate on a pixel-by-pixel basis, developing residuals at each pixel position over the pixel block. Non-predictively coded blocks may be coded without comparison to reference pixel blocks, in which case the pixel residuals are the same as the source pixel data.
The transform unit 322 that may convert the source pixel block data to an array of transform coefficients, such as by a discrete cosine transform (DCT) process or a wavelet transform. The quantizer unit 323 may quantize (divide) the transform coefficients obtained from the transform unit 322 by a quantization parameter Qp. The entropy coder 324 may code quantized coefficient data by run-value coding, run-length coding or the like. Data from the entropy coder may be output to the channel as coded video data of the pixel block. The reference frame decoder 330 may decode pixel blocks of reference frames and assemble decoded data for such reference frames. Decoded reference frames may be stored in the reference picture cache 340.
The predictor 350 may generate and output prediction blocks to the subtractor 321. The predictor 350 also may output metadata identifying type(s) of predictions performed. For inter-prediction coding, the predictor 350 may search among the reference picture cache for pixel block data of previously coded and decoded frames that exhibits strong correlation with the source pixel block. When the predictor 350 finds an appropriate prediction reference for the source pixel block, it may generate motion vector data that is output to the decoder as part of the coded video data stream. The predictor 350 may retrieve a reference pixel block from the reference cache that corresponds to the motion vector and may output it to the subtractor 321. For intra-prediction coding, the predictor 350 may search among the previously coded and decoded pixel blocks of the same frame being coded for pixel block data that exhibits strong correlation with the source pixel block. Operation of the predictor 350 may be constrained by a mode selection provided by the controller 370. For example, if a controller selects an inter-coding mode for application to a frame, the predictor 350 will be constrained to use inter-coding techniques. If the controller selects an inter-prediction mode for the frame, the predictor may select among inter-coding modes and intra-coding modes depending upon results of its searches.
A transmit buffer 360 that accumulates metadata representing pixel block coding order, coded pixel block data and metadata representing coding parameters applied to the coded pixel blocks. The metadata can include prediction modes, motion vectors and quantization parameters applied during coding. Accumulated data may be formatted and transmitted to the channel.
A controller 370 to manage coding of the source video, including selection of a coding mode for use by the predictor 350 and selection of quantization parameters to be applied to pixel blocks.
To detect pixel blocks that are at risk for banding, the method 400 may search for pixel blocks that have smooth video content. Several techniques are available to detect such pixel blocks. The method 400 may compute means and/or variances of image data of each pixel block and compare those values to predetermined thresholds to determine whether individual pixel blocks should be designated as candidates. In addition, the method 400 may consider differences among these values between pixel blocks to determine whether individual pixel blocks are sufficiently smooth that they should be designated as candidates. As another technique, the method 400 may perform frequency analyses of pixel blocks' content, for example, by DCTs and/or wavelet transforms, and may consider distribution of energies among high frequency and low frequency coefficients of such transforms. A pixel block whose DC and low frequency coefficients have high energy as compared to low frequency coefficients of the pixel block or whose high frequency coefficients are lower than a predetermined threshold may be considered appropriate candidates.
As discussed, during operation of box 430, the method 400 may assemble candidate pixel blocks into regions. Any two candidate pixel blocks may be joined to a common region if they share a border in common. The method 400 may operate recursively, adding increasing numbers of candidate pixel blocks to a given region until the region has no other pixel blocks on its edges that are candidates for banding artifacts. Four regions, R1-R4, are illustrated in
In indicated, once the pixel blocks are assembled into regions, the method 400 may consider statistics of the regions to determine whether to disqualify certain regions from further anti-banding processing. Several metrics may be used for this purpose, for example:
Size: The method may disqualify regions when the number of pixel blocks in the respective region is less than a predetermined threshold. In the example of
Shape: The method may disqualify regions based on their shape. For example, the method may require that each region have height and a width that exceeds certain thresholds. For example, if a region were required to have a width and a height that are both greater than three pixel blocks, regions R1-R3 would be disqualified.
DC Differences Among Blocks: The method may compare DC coefficients among pixel blocks within a region to determine whether to disqualify the region from further processing. DC coefficients may be obtained by averaging image content of each pixel block. If block-to-block DC coefficient changes exceed a predetermined threshold or if the DC coefficient changes occur in a non-uniform manner (e.g., they change signs from block to block), then pixel blocks of a region may be disqualified from further processing.
Mean and Variance Comparisons: The method may calculate the means and variances for image content of individual pixel blocks and compare those values to a mean and variance of all image content in the region. If pixel blocks' means and/or variances differ from the mean and/or variance of the region as a whole, the region may be disqualified from further anti-banding processing.
Temporal Persistence: The method 400 may calculate each region's persistence over time. Candidate regions that are present across a relatively small number of adjacent frames may not be present for a sufficiently long time to be perceived as an annoying artifact by a viewer. By contrast, candidate regions that are present for a sufficient number of frames—for example, for a second or longer after decoding and display—may remain candidates for anti-banding processing.
Of course, the method 400 may combine these different metrics to determine when to disqualify regions from anti-banding processing.
In another embodiment, the method 400 may consider some of its metrics as part of the process of growing regions from individual candidate pixel blocks. Consider region R3 as an example. The region R3 may be created by considering pixel blocks PB3.1 and PB3.2 in a first iteration, then expanding to consider pixel blocks PB3.3, PB3.4, etc. As a new pixel block (say, PB3.3) is considered for admission to the region R3, the method 400 may compare metrics of the candidate pixel block PB3.3 to metrics of the pixel blocks in the region R3 and may decide whether to accept or reject the pixel block for admission to the region R3 on the basis of the comparison. For example, the method may consider the mean and variance of pixel content within the candidate pixel block PB3.3 to the mean and variance of pixel content of other pixel blocks (PB3.1, PB3.2, in this example) already admitted to the region R3. If the means and variances are sufficiently similar, the candidate pixel block PB3.3 may be admitted.
Even when the method 400 considers pixel block metrics as part of the process of growing regions, the method 400 may consider still other metrics after conclusion of the region growing process. For example, disqualification processes that are based on a region's shape, size and/or temporal persistence likely will be performed after the region growing process concludes.
In other embodiments of the present invention, the method 400 may use the foregoing metrics to determine whether to remove certain pixel blocks from a region. In such embodiments, pixel blocks may be removed from a region but remaining pixel blocks of the region may be subject to anti-banding processing. Using size and shape as an example, the method may determine whether a portion of a region includes a projection that has a size and shape that are unlikely to exhibit banding artifacts even though the remainder of the region could exhibit such artifacts (shown as P1, for example, in region R4). In this case, the pixel blocks belonging to the projection may be removed from the regions but the remainder of the regions still may be a candidate for anti-band processing.
In another embodiment, rather than adding noise to pixel blocks of regions subject to anti-banding processing, the method 400 may apply different types of filtering to anti-banding regions that are applied to other pixel blocks. Generally, a pre-processor may apply filtering, including denoising filtering, to source video to render video coding more efficient. When regions are identified that are to be subjected to anti-banding processing, application of denoising filters to anti-banding regions of a frame may be altered to preserve any dither noise that appears naturally in the source frame. In one example, the denoising filter may be disabled entirely for such regions but, in others, a strength of filtering may be diminished as compared to a strength of filtering applied to image content outside the anti-banding regions.
As discussed, once the region growing process is completed, the method 400 may apply anti-banding processes to pixel blocks within the region to protect against banding artifacts. Several techniques are available to protect against banding, including for example:
A video coding system 300 may add an amount of dither noise to pixel blocks within the region prior to coding.
A video coder 300 may alter quantization of pixel blocks that are subject to anti-banding processing. Pixel blocks that fall within regions may be assigned relatively lower quantization parameters than pixel blocks that do not fall within regions. By lowering quantization parameters for such pixel blocks, it increases the likelihood that pixel block residuals will not be truncated to zero by the quantization parameter. Thus, when the pixel block is coded, a controller 370 may alter a quantization parameter as applied to the quantizer 323.
A video coder 300 may vary an amount of filtering to be applied to pixel blocks that are members of such regions. In this manner, when a video coder 300 determines that a given pixel block is a member of a region for which anti-banding processing is to be performed, the video coder 300 may disable or lower the strength of filtering that otherwise might be applied by a pre-processor 310 to the pixel block. As noted, a pre-processor 310 ordinarily may filter pixel block data to condition it for coding. Anti-noise filters are members of one such class of filters that a pre-processor 310 may apply. Such filters typically improve coding efficiency by diminishing high frequency image content of pixel blocks. In the case of smooth image regions, however, such filters may increase the likelihood that banding artifacts may arise and, therefore, they may be disabled or diminished in filtering strength. Moreover, a pre-processor 310 may vary an amount of filtering applied to pixel blocks within an anti-banding region based on an amount of noise that is estimated to be present in the source signal.
A pre-processor 310 may vary an amount of filtering applied to portions of frame data based on characteristics and viewing conditions of a display as estimated by an encoder. For example, an encoder may revise filtering strengths based on information representing size of a display at a decoder. Thus, a different level of filtering may be applied when an encoder codes video for a handheld device with a 2-3 inch diagonal display than when the encoder codes the same video for a tablet or personal computer having a 10-15 inch diagonal display.
A pre-processor 310 may vary an amount of filtering applied to portions of frame data based on perceptual masking value estimates for each portion of image data, with the perceptual mask indicating how easily a loss of signal content can be observed.
The regions may constitute a detection map with multiple levels of granularity, which further may be linear or nonlinear lowpass filtered or morphologically filtered from an initial binary detection map. Combined with the alpha blending technique, a detection map with multiple levels of granularity may prevent generating undesirable yet visible abrupt transitions between a detected banding region and its neighboring non-detected region.
The detection map further may be filtered temporally to increase consistency in time.
A pre-processor may vary an amount of filtering applied to portions of frame data based on indicators from a face detector (not shown)—portions that are identified as representing frame data may have relatively higher quality filtering than other regions of the frame. Faces often represent areas of image content having the highest level of interest when recovered video is rendered at a display of a decoder. Therefore, a different level of filtering may be applied for areas detected to represent human faces than for other areas that may represent background or other content.
A pre-processor may vary an amount of filtering applied to portions of frame data based on indications from a rate controller, identifying a bit budget of a given picture and bitrate generated in a previous encoding pass. As indicated, stronger filters may reduce noise in image content at a greater rate than weaker filters but may incur a coding cost at a lower bitrate than may be required to code the noisier image content. If a rate controller indicates that a relatively low number of bits are available for coding, filtering strengths may be increased whereas; if the rate controller indicates that a relatively high number of bits are available for coding, filtering strengths may be lowered.
The amount of the pre-processing filtering may be controlled by the quantization parameter and/or other bitstream syntax statistics collected in a previous encoding pass of a multi-pass video coder. For example, in a multi-pass coder, an encoder may estimate image statistics (such as spatial complexity) and generate quantization parameters for various portions of the video sequence. Before performing another pass of coding the same sequence, filtering strengths may be adjusted based on observed costs of coding in the first pass. Filtering strengths may be increased for portions of the video sequence that were observed to require relatively high bit rates in a prior coding pass, whereas filtering strengths may be lowered for portions of the video sequence that were observed to require relatively low bit rates.
The foregoing discussion has described operation of the embodiments of the present invention in the context of coders and decoders. Commonly, video coders are provided as electronic devices. They can be embodied in integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors. Alternatively, they can be embodied in computer programs that execute on personal computers, notebook computers or computer servers. Similarly, decoders can be embodied in integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors, or they can be embodied in computer programs that execute on personal computers, notebook computers or computer servers. Decoders commonly are packaged in consumer electronics devices, such as gaming systems, DVD players, portable media players and the like and they also can be packaged in consumer software applications such as video games, browser-based media players and the like.
Several embodiments of the invention are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations of the invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.
This application claims the benefit of priority afforded by provisional application Ser. No. 61/607,484, filed Mar. 6, 2012, entitled “Improvements in Video Preprocessors and Video Coders.”
| Number | Date | Country | |
|---|---|---|---|
| 61607484 | Mar 2012 | US |
