Patent Application
20130223525
The present invention relates to video coding and, in particular, to predictive coding techniques based on prediction patches.
Motion compensation is commonly used as prediction in modern video compression standards to exploit temporal redundancy across frames. A first encoded frame serves as a prediction reference for one or more subsequently-coded frames. In such systems, a video coder codes the first frame and transmits the coded data to a remote video decoder. The video coder itself decodes the first coded frame and stores the recovered frame locally for use as a prediction reference for later frames. Since video coding techniques typically are lossy, the recovered frame likely will have some coding-induced errors when compared to its source. However, the video coder and video decoder each may obtain a common copy of the recovered frame, absent transmission errors, which provide an adequate basis for coding.
A video coder uses locally decoded video (called, “reference frames” herein) to generate prediction data when coding a later-received frame. The video coder parses frames into pixel blocks and codes the frame on a pixel block-by-pixel block basis. The video coder searches for a prediction reference among locally-stored reference frames and, when one is found, the video coder codes a difference between the predicted data and source content of the later-received frame.
High coding efficiency can be achieved using motion compensated prediction techniques. However, temporal correlation in the video sequence can be broken, for example, on a scene change, when there is camera movement, in the presence of high level noise, etc. Further, a video coder and decoder typically store a limited number of reference frames which may fail to provide good prediction when temporal correlation is low. Accordingly, there is a need in the art for a video coding system that can exploit benefits of predictive coding techniques when reference frames become poor predictors for a video sequence being coded.
Embodiments of the present invention provide a coding system in which an encoder may include a coding engine for predictively coding input video, a decoder to reconstruct reference pictures generated by the coding engine, a reference picture cache to store the reconstructed reference pictures, a patch cache to store prediction patches generated from other sources, and a prediction search unit to search among the reference picture cache and the patch cache to generate prediction references for use by the coding engine while coding input video. The prediction patches may be assembled from a variety of sources including: predefined image content, reference pictures being evicted from the reference picture cache, image content of prior coding sessions and image data stored by applications on a common terminal where the encoder resides. The prediction patches are likely to persist in the encoder for a time longer than reference pictures stored in the reference picture cache and, as such, might provide good correlation to input video data in circumstances where low correlation is provided by the reference picture cache. A decoder may store prediction patches in its own patch cache for synchronous decoding.
In
As illustrated, the video coder 220 may include a pre-processor 221, a coding engine 222, a local decoder 223, a reference picture cache 224, prediction search unit 225, a patch cache 226 and a controller 227. The pre-processor 221 may accept source video from the camera 210 and may perform various processing operations on the source video to condition it for coding. The coding engine 222 may perform compression operations on the pre-processed video to reduce spatial and/or temporal redundancies therein. The coding engine 222 may output coded video data to the transmitter 230. The decoder 223 may decode coded video data of select frames, designated as reference frames, and may store the recovered reference frame data in the reference picture cache 224 for use in coding later-received video. The patch cache 226 may store other data that has been designated for use as prediction references for the video sequence.
As part of the coding operation, the prediction search unit 225 may establish prediction references for frames being coded. Specifically, the prediction search unit 225 may search among data stored in the reference prediction cache 224 and/or the patch cache 226 to find appropriate prediction reference(s) for new frames being coded. The prediction search unit 225 also may generate metadata, such as frame/patch identifiers and motion vectors, to identify a source of prediction data within the reference picture cache 224 or the patch cache 226. The metadata may be integrated with the coded video data and transmitted to a decoder (not shown) for use in decoding the coded video data.
The pre-processor 221 may perform a variety of video processing operations on the source video output from the camera to condition the source video for coding. The pre-processor 221 may include an array of filters (not shown) such as de-noising filters, sharpening filters, smoothing filters, bilateral filters and the like, that may be applied dynamically to the source video based on characteristics observed within the video. The pre-processor 221 may coordinate with the controller 227 to review the source video data from the camera and select one or more of the filters for application. Typically, the pre-processor 221 conditions the source video data to render compression more efficient or to preserve image quality in light of data losses that may be incurred as the coding engine 222 operates.
The coding engine 222 may code input video data according to a variety of different coding techniques to achieve compression. The coding engine 222 may compress the images by a motion-compensated prediction. Frames of the input video may be assigned a coding type, such as intra-coding (I-coding), uni-directionally predictive coding (P-coding) or bi-directionally predictive coding (B-coding). The frames further may be parsed into a plurality of pixel blocks and may be coded by transform coding, quantization and entropy coding. The coding engine 222 may select quantization parameters in cooperation with the controller 227, which truncate low-energy transform coefficients. Pixel blocks of P- and B-coded frames may be coded predictively, in which case, the coding engine may calculate motion vectors identifying pixel blocks of decoded frames stored in the reference picture cache 224 or patch cache 226 that serve as predictions of the pixel blocks being coded and may generate prediction residuals prior to engaging the transform coding.
The coding engine 222 further may designate that certain coded frames may be used as reference frames for use in coding later-received input frames. The decoder 223 may decode coded video data of the reference frames and store the video data recovered therefrom in the reference picture cache 224. In this manner, the video coder 220 has a decoded copy of the reference frame as it will be decoded at the video decoder (not shown in
The patch cache 226 may store various “prediction patches,” image content that are stored both at the video coder 220 and a video decoder (not shown in
The transmitter 230 may transmit the coded video data to the channel 250. In so doing, the transmitter 230 may multiplex the coded video data with other data to be transmitted such as coded audio data and control data (sources not illustrated in
The video decoder 320 may include a decoding engine 321, a post-processor 322, a predictor 323, a reference picture cache 324 and a patch cache 325 operating under control of a controller 326. The decoding engine 321 may perform decoding operations that invert coding operations performed by the coding engine 222 (
When the decoding engine 321 decodes new reference frames, it may store the recovered reference frames in the reference picture cache 324 for use in decoding subsequently-received coded video data.
The post-processor 322 may perform additional video processing to condition the recovered video data for rendering, commonly at a display device. Typical post-processing operations may include applying deblocking filters, edge detection filters, ringing filters and the like. The post-processor 322 may output recovered video sequence for rendering on the display 330 or, optionally, stored to memory for later retrieval and display.
The functional blocks illustrated in
During coding, the prediction search unit 225 (
If an adequate prediction reference is not found within the reference picture cache, the method 400 may perform a search for a prediction match among the prediction patches stored in the patch cache (box 450). Following the search, the method 400 may determine whether an adequate prediction match was found (box 460). If so, the method 400 may code the pixel block with reference to the matching data from the selected prediction patch (box 470). If no adequate prediction match was found from either search, the method 400 may code the pixel block by another coding technique such as I coding (box 480). Alternatively, the method 400 may code the pixel block predictively using the reference frame or prediction patch uncovered by the searches of boxes 420, 450 that minimizes coding error (box 490).
In another embodiment, the method 400 may perform searches among the reference picture cache and the prediction cache (boxes 420, 450) in all cases and, thereafter, may select a prediction reference between the two searches based on coding efficiency.
Searches also may be augmented by data provided by local motion detectors (for example, gyroscopes, GPS systems and/or accelerometer), which provide indicators of camera speed during image capture and, by extension, degrees of motion in the video content. For example, each stored reference frame and/or patch may have stored with it data representing motion of the respective frame and/or patch. Searches among the reference picture cache and the prediction cache may be constrained to reference pictures and/or patches that exhibit similar degrees of motion as the pixel block being coded.
In another embodiment, a search method may select caches 224, 226 (
In an embodiment, searches among the patch cache 226 may be augmented with metadata information (such as information about camera pose, camera motion, object type(s) detected in video, etc.) to reduce search complexity. Pixel patches may be transformed (such as by noise filtering, enhancement, 3D transformations, etc.) before being applied in coding as reference pixels
As indicated, contents of the patch caches 226, 325 (
Active coding may include, at terminal 110, capturing source video, coding the video using reference frames and reference patches and transmitting coded video data to the other terminal 120 (boxes 530-540). The terminal 110 also may determine whether to build a new patch (box 545) and, if so, may mark coded data from which the patch is to be extracted (or simply may transmit the patch data expressly) in the coded video data (box 550). The coding process may operate recursively and the operations represented by boxes 530-550 may repeat until the coding session is terminated.
Patches also may be transmitted between terminals 110, 120 using implied signaling. In this embodiment, the terminals 110, 120 may operate according to a common rule set that defines triggering conditions for generation of new patches. The triggering conditions may be based on other elements of the coded video data. In a simple example, the triggering condition for patch generation may be based on a number of frames (or reference frames) being coded since a prior patch was created. When the triggering condition is met, both terminals 110, 120 may generate a new patch from predetermined elements of the coded channel data, also defined in the rule set, and may store the new patch in the terminals' respective patch caches. In this embodiment, it becomes unnecessary to send express designators in the channel data to identify when patches are to be created.
During active coding, the second terminal 120 may receive the coded video (box 555) and store any patches contained in the coded video to the patch cache (box 560). The terminal 120 also may decode video using references frames and prediction patches as indicated in the coded video data (box 565) and may display the recovered video (box 570). The coding process may operate recursively and the operations represented by boxes 555-570 may repeat until the coding session is terminated.
Thus, the patch library can be updated and synced between encoder and decoder by signaling the changes. Transformations on the patches can be transmitted via the bitstream. As an alternative to transmission of patches at start up (message 510), the terminals 110, 120 can start with known patch libraries which can be adapted or augmented during the course of transmission.
As frames continue to be coded and the reference cache fills, the method 600 also may evict reference frames from the reference picture cache. When this occurs, the method 600 may determine whether to build a prediction patch from a reference picture to be coded. For example, the method 600 may compare the utilization count of the reference picture being evicted to a threshold and determine whether the utilization count exceeds the threshold (boxes 650-660). If the utilization count is exceeded, the method 600 may build a new patch from the reference frame to be evicted (box 670). Thereafter, or if the utilization count does not exceed the threshold, the method 600 may evict the reference picture from the cache (box 680).
Prediction patches may be constructed from reference pictures in a variety of ways. In a simple embodiment, the reference picture simply may be copied to the prediction cache (operation not shown). In another embodiment, the method 600 may attempt to detect objects or textures within the reference frame being evicted and may build a new prediction patch from the detected objects or detected textures (operation not shown). In yet another embodiment, illustrated in
As discussed, contents of the patch caches 226, 325 may represent a reference library of prediction patches that provide alternative sources of prediction for pixel blocks being coded. The prediction patches, therefore, may improve coding efficiency when the reference frames stored in the reference caches 224, 324 do not provide adequate prediction references for new frames to be coded. The prediction patches can be of arbitrary size; they need not be the same size as the frames being coded. The prediction patches can (but need not) be derived from one or more frames of the input video sequence. The prediction patches may be generated from a variety of different sources.
In another embodiment, the prediction patches can be derived from image content stored locally in the terminal that performs the encoding. A patch may be derived from a single frame or from multiple frames. The reference library may be constructed from data of previously coded frames or prior coding sessions. Representative examples of patch libraries follow:
One example of a reference library is a face library. Such a library can be constructed by storing data of faces detected in previously-coded frames or in prior coding sessions. In this embodiment, an encoder 200 (
During coding, face references in the patch cache library can be transformed through 3D warping, scaling, rotating, filtering, etc., to match the size of a detected face in a new frame being coded to the size of a matching face in the patch cache library. Due to face movement and camera motion, an encoder 200 may not always identify prediction references of a face using traditional motion searches within a reference picture cache 224 of limited depth. Application of a patch cache 226 that stores a large number of reconstructed faces can improve the quality of an encoder's prediction. In this example, only the face region need be stored and searched and, therefore, the performance impact and memory requirements of the patch cache 226 is expected to be lower than doing traditional motion searches over a large number of reference frames in a reference picture cache 224. The performance impact can be further reduced by pre-selecting likely matching faces using additional information that may be provided by a face detector 260 such as face orientation and smile/blink detection and metadata provided by a camera 210 or pre-processor 221 such as motion estimates, estimates of motion blur and the like.
Another example of a reference library is a texture library. The patch cache 226 may store a library of texture patterns that may be artificially constructed or be constructed by identifying texture from a current coding session or from previous coding sessions (video communication sessions). When the textures are artificially constructed, textures patterns may be predefined at the encoder 200 (
The texture patterns can be grouped into different classes based on certain characteristics. For example, text characters can be identified as a first class, object shapes (vehicles, human body, etc.) may be identified as different classes, and the like. Texture patterns can be classified also by the location in the image as they occur; this can be helpful in improving the perception of resolution of salient textures. The performance impact can be further reduced by pre-selecting likely matching textures and texture classes using side-information that may be provided by the pre-processor 221, such as geographic location, neighboring texture classes, correlation among different texture classes, etc.
During coding, stored texture data may be subject to processing, which may include scaling, rotation, smoothing, sharpening, etc., prior to use as a prediction reference. Indices of the texture patterns as stored in the patch cache 226 can be sorted by how frequently they match the coded blocks in previous frames, in order to reduce the overhead of signaling the indices.
The principles of the present invention find application with object detectors of other types, beyond the face and texture recognition systems discussed hereinabove. Moreover, such detectors and patch coders may be augmented by data provided by local motion detectors (for example, gyroscopes, GPS systems and/or accelerometer), which provide indicators of camera speed during image capture and, by extension, degrees of motion in the video content.
In another embodiment, the patch cache may be initialized with data known privately to the terminals 110, 120 (
As part of their operation, applications 770 may store data that includes video elements that can be leveraged for use as prediction patches. For example, a terminal 700 may include a contacts application 770.1 storing data of the personal contacts of an operator of the terminal. Further, the terminal 700 may include a photo manager application 770.2 storing tagged images of persons known to the operator. And, indeed, the images of these applications may be synched with images stored on terminals of other users.
When a video coding session is initiated between terminals of two users, the terminals may exchange identifiers of the other terminals and may populate the picture caches 224, 324 (
The terminals also may use data from the patch caches in response to aberrant system behavior. For example, channel conditions may fluctuate during the course of a video coding session which can affect the amount of bandwidth available to the terminals. Under extreme low bandwidth, an encoder may enter a mode in which it does not code any residual information to a decoder. Instead, the encoder can send information that indicates transformation of reference frames or reference libraries of the patch cache, to construct video frames to be displayed at the receiver side.
The foregoing discussion has described operation of the embodiments of the present invention in the context of coders and decoders. Commonly, video coders are provided as electronic devices. They can be embodied in integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors. Alternatively, they can be embodied in computer programs that execute on personal computers, notebook or tablet computers or computer servers. Similarly, decoders can be embodied in integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors, or they can be embodied in computer programs that execute on personal computers, notebook computers or computer servers. Decoders commonly are packaged in consumer electronic devices, such as gaming systems, smartphones, DVD players, portable media players and the like, and they also can be packaged in consumer software applications such as video games, browser-based media players and the like.
Several embodiments of the invention are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations of the invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.
This application claims the benefit of U.S. Provisional Application Ser. No. 61/603,047 filed on Feb. 24, 2012, the content of which is incorporated herein it its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 61603047 | Feb 2012 | US |
