This relates generally to processing video information.
Video may be supplied with a given frame rate. The video is made up of a sequence of still frames. The frame rate is the number of frames per second.
Some displays use frame rates different than the frame rate of the input video. Thus, frame rate conversion converts the frame rate up or down so that the input frame rate matches the display's frame rate.
Frame rate conversion is used to change the frame rate of a video sequence. A typical frame rate conversion algorithm application is to convert film content from 24 frames per second to 60 frames per second for the National Television Systems Committee (NTSC) system or from 25 frames per second to 50 frames per second for the phase alternating line (PAL) system. High definition television supports 120 or 240 frames per second display, which also needs frame up conversion. In accordance with some embodiments, the frame rate conversion algorithm may adaptively compensate for the motion depicted in the video sequence.
In one embodiment, bi-directional, hierarchical local and global motion estimation and motion compensation is used. “Bi-directional” means that the motion is estimated between two anchor frames in the forward and backward directions. “Hierarchical motion estimation” refers to the fact that motion estimation is refined with each increasing resolution of the supplied video information. The bi-directional hierarchical local and global motion estimation is followed by a final motion compensation stage that integrates the two anchor frames and all motion estimation elements into one interpolation stage.
In accordance with one embodiment, an input series of two video frames may be received. The frames may include a series of pixels specified by x, y, and time t coordinates. Motion vectors may be determined from a first to a second frame and from the second to the first frame or, in other words, in the forward and backward directions. The algorithm creates an interpolated frame between the two frames using the derived local and global motion, the time stamp provided, and the consecutive frame data. The time stamp corresponds to the frame rate and, particularly, to the frame rate desired for the output frame.
Thus, a previous frame P may have pixels specified by x, y, and t variables and a next frame N may have pixels with x, y, and t+1 variables. The output frame C has pixels with x, y, t′ variables. Interpolated output frame C may have a time t+q, where q is less than 1 and greater than 0. Pixel positions may be indicated by p in an x and y coordinates. A motion vector MVAB (x,y) is the motion vector, at coordinates x and y in screen space, from a frame A to a frame B. A global motion vector GMAB is the dominant motion vector from frame A to frame B.
In accordance with some embodiments, the frame rate conversion is amenable to highly parallelized throughput computer devices such as multicore processors. It is also compatible with block search hardware accelerators. The frame rate converter, shown in
The preprocessing stage 100 includes letter box detection unit 106, pyramid building unit 108, and scene cut detection unit 110 in one embodiment. The letterbox detection unit 106 finds the black margins of the frames, if any such black margins exist. The scene cut detection unit 110 detects scene changes based on histogram features in one embodiment. The pyramid build 108 builds the pyramid structure of lower resolution versions of the frames.
In the letterbox detection unit 106, the black margins may be removed, which may otherwise interfere with the accuracy of motion estimation and, hence, are ignored during the block search process. The unit 106 computes independently the four coordinates of the active image regions inside the frame: firstX, firstY, lastX, and lastY, where X and Y are pixel coordinates. The firstX algorithm computes for each column, starting from the left one, in one embodiment, the average of the maximum and minimum grey level value of each pixel in YUV representation. The firstX algorithm tests a few criteria for a black color column. When, for the first time, one of these criteria is not met, the algorithm stops and the index of this column is output as the firstX. One set of criteria for a column to be part of a margin includes the determination that the maximum grey level value is not bigger than a threshold, the average grey level value is smaller than a threshold, the difference between the maximum and minimum values is smaller than a threshold, the difference between the average grey level value and the average value of the previous column is smaller than a threshold, and the difference between the average Y value and the average value of the same column in the previous frame is smaller than a threshold. The algorithms for lastX, firstY, lastY are essentially the same.
The scene cut detection unit 110 detects scene cuts by comparing the histogram of the next frame with that of the previous frame. When a scene cut is detected, the algorithm resets motion estimation buffers and changes the mode to frame duplicate. In the frame duplicate node, frames are simply duplicated instead of being interpolated. Thus, frames are duplicated at module 130 of the motion compensated interpolation stage 104 and the motion estimation stage 102 is simply skipped.
The build pyramid unit 108 is a standard image processing operation. The number of pyramid levels depends on the original frame resolution and may be determined in such a way that the lowest resolution level is not too small for block search operations.
The build pyramid unit 108 builds a pyramid structure for the Y component of a YUV depiction for an image where the lowest image of the pyramid is the original image, the second or lower resolution image is a quarter the size of the base unit or original image, and the third image is a still lower resolution image of the second image, a quarter of its size.
The motion estimation procedure may be the same in both the forward and backward directions. The motion estimation stage 102 uses the build pyramid block module 108, having a given number of levels. In one embodiment, three levels are utilized, but any number of levels may be provided. In order to achieve a smooth motion field, motion vector predictors from the previous level of a pyramid and from the previous motion estimation are used. The motion estimation output may include one motion vector for each 8×8 block in one embodiment.
In the motion estimation stage 102, the pyramids of the previous frame P and the next frame N are provided to a forward block search unit 112 and a backward block search unit 114. The output of each block search unit 112 and 114 is a motion vector field, either from the previous frame P to the next frame N, in the case of forward block search unit 112 or from the next frame to the previous frame, in the case of the backward block search unit 114, as depicted in
The motion estimation stage 102 may implement forward motion estimation or the backward motion estimation. It may be implemented in software or hardware. In a hardware embodiment, a hardware accelerator may be used in some embodiments.
The input frames A and B include only the Y component of a Y,U,V color system, in one embodiment. Other color schemes may also be used. The input to the motion estimation unit may also include temporal predictors for each block at each of a plurality of pyramid levels of a hierarchical system. Temporal predictors are the expected locations of a source block in a reference frame according to the previous motion estimation compute. The outputs are the motion vectors, as indicated, for each block at each pyramid level and the global motion or dominant motion vector in the frame.
In some embodiments, an adaptive motion estimation may be implemented, as illustrated in
The frame is divided into non-overlapping blocks (
the horizontal variability: and the
and the vertical variability:
are computed for deriving a similarity measure (block 152).
The similarity measure Sima,b (block 154) is determined as follows:
where CA, CX and CY are empirical constants.
Based on these calculated similarity values, the original block is divided into sub-block configurations containing block sizes that are smaller than the original block size (block 156). The process may iterate, in some embodiments, to produce even smaller block sizes.
The sub-block size possibilities, in one embodiment, are illustrated in
Description:
The module decides about the division of a 16×16 block to 8×8 sub-blocks. It tries to keep the search blocks as large as possible as long as they have internal similarity.
Input:
B—the 16×16 block, consisting of B00, B10, B01 and B11, 8×8 sub-blocks
Output:
Div—(B1, B2, B3, B4)—division of the block B to four or fewer sub-blocks. When there are less than four sub-blocks the last ones are empty.
Process:
Compute the average gray level and the directional viabilities of the block B and the sub-blocks B00, B10, B01 and B11 according to:
Average gray value:
Horizonatal variability
Vertical variability
Set the three constants ca, cb, cc (see the Similarity3 procedure below) to: ca=AvgB, cb=VarXB, cc=VarYB.
Sim=Similarity3(al, a2, b1, b2, c1, c2):
Description:
Measurement of similarity, based on 3 factors for two different blocks. Lower result indicates more similar blocks.
Input:
a1, a2, b1, b2, c1, c2—the factors to compare
Output:
Similarity3—the similarity factor of the blocks
Process:
where ca, cb, cc are constants.
Next, the motion estimation stage 102 prepares predictors for the search blocks. A predictor is a base motion vector that determines the central location of the reference frame, around which the search will be done. For each search block, a set of up to nine predictors may be calculated. Up to six predictors are inherited from the previous pyramid level motion vectors of neighboring blocks. In the highest pyramid level, there is only one spatial predictor, the zero vector. Up to five predictors may be inherited from the corresponding (forward or backward) preceding motion estimation stage. One is the global motion vector and the others are projected motion vectors of the sub-block in the same spatial location. Identical motion vectors may be removed from the predictor's set in order to reduce the search compute time, in some embodiments.
For each sub-block, of the source frame, that may be a sub-block of the search block, the magnitude of a similarity measure, such as sum of absolute differences of its motion vectors, is compared to a threshold. If the similarity value is larger than the threshold, then the block is further subdivided into sub-blocks. For these sub-blocks, a new set of predictors is computed and, for each one of them, the best motion vector is recomputed based on a new block search. If there are a number of predictors/candidates for the best vector, the one with the best sum of absolute differences (SAD) is chosen. If there are several candidates with the same SAD—the shortest one is chosen. If there are several candidates with the same SAD and the same length—an arbitrary one is chosen.
The global estimation module 116 estimates the global or dominant motion vector in the motion vector field computed by the forward and backward motion estimation modules 112 and 114. This may be done by an iterative process that computes the average motion vector and then removes the motion vectors that are distanced in the L1 norm or Manhattan distance from the average. After the outliers are removed, the average is then computed again and another set of vectors is removed. The algorithm converges when no vectors are removed or when the majority of the vectors are very close to the average. The global motion vector is computed for the forward and backward directions. If these vectors differ, the one that has the smaller similarity measure on a small number of blocks is selected, in one embodiment.
The logo detection module 118 detects static solid or semitransparent logos or titles. The output of the logo detection module 118 is a logical array that indicates for each pixel in a frame if it belongs to a logo or title or not.
The frame repeat fallback module 120 makes a decision on the interpolation method of the motion compensated interpolation stage 104. Based on the distribution of motion vectors, the judder level and the scene cut indication, it computes a flag indicating frame repeat for the motion compensated interpolation stage 104. The criteria for using frame repeat, instead of motion compensated interpolation, may include very low judder level, large regions with high motion, and a large motion field variance. The judder level is estimated as the sum, over all the pixels in the frame, of the inner products of the gradient and the motion vector in one embodiment. However, pixels that have very fast motion or low inner product may be ignored. When the judder level is very low, motion compensated interpolation does not improve the video quality.
When a large percentage of the frame pixels have very high motion, the motion compensation interpolation may produce significant artifacts. Also, the human visual system is not sensitive to the judder artifacts when the motion is very high.
When the motion field has large variance, the motion compensated interpolation may produce significant artifacts that are more annoying than the judder artifacts. In order to produce smoother transitions between frame duplication and motion compensation modes and to avoid fast mode fluctuations, the algorithm may define a smooth decision region. The system may transition between the modes only if the mode indications are consistent for a few consecutive frames in one embodiment. In the beginning of a new scene, the frame rate conversion module is on (i.e. FRC on in
Thus, referring to
The motion compensated interpolation stage 104 includes a motion vector smoothing 122, pixel warping 124, median calculator 126, post processing 128, and frame duplicate 130. The motion vector smoothing 122 computes forward and backward motion vectors for each pixel of the interpolated frame on the basis of the relevant block motion vectors. The motion vector of a given pixel is a weighted average of the motion vector of the block to which it belongs and the motion vectors of its immediate neighbor blocks. The weights are computed for each pixel based on its location in the block.
The pixel warping 124 computes four interpolation versions for each color component (Y, U, and V. for example) of each pixel of the interpolated frame. The interpolation versions may be pixel A=N(p+(1−q)·MVPN(p)) from frame N in the location indicated by the corresponding motion vector from P to N and the time stamp q, pixel B=P(p+q·MVNP(p)) from frame P in the location indicated by the corresponding motion vector from N to P and the time stamp q, pixel D=N(p+(1−q)·GMPN) from frame N, in the location indicated by the global motion vector from P to N and the time stamp q, pixel E=P(p−q·GMPN) from frame P in the location indicated by the global motion vector from N to P and the time stamp q. The method of interpolation, in one embodiment, may be nearest neighbor interpolation or bi-linear interpolation, as well as any other interpolation method.
Where locations are not at integer positions, the candidate pixel may be computed by using bilinear interpolation.
If a motion vector points to a pixel that is located outside the valid frame region, then the motion vector is truncated such that the valid pixel in the closest location is taken, in one embodiment, and this pixel is returned with a non-valid indication. Also, if a motion vector points to a location that one of its neighbors has a logo indication, then the motion vector is truncated to an integer position and this pixel is returned with a non-valid indication. In the median calculator 126, a non-valid pixel is replaced by the pixel from the other frame. If “pixel A” is not valid, then it is replaced by “pixel B” and the other way around. If both pixels are not valid, then the values that were returned are used. Denoting the four pixel candidates describes above as A, B, D, and E, in the corresponding order and where C equals (A+B)/2, the interpolated pixel is computed as the median of A, B, C, D, and E.
For pixels moving inside moving objects, the motion field is trusted and the values of the pixels A, B, and C are very close and, hence, they dominate the medium procedure. This is also true for pixels belonging to the background. However, for the points nearby, the objects' boundaries, the points A, B, and C may differ significantly. To reduce the halo effect of occlusion regions, the pixels D and E may be incorporated into the median.
The median calculator 126 calculates the median of A, B, C, D and E pixels for each component, such as Y, U, V of each pixel, where C is the average of A and B pixels.
The motion compensation block uses the P and N frames, including all Y, U, and V color components in a YUV system. It uses the forward motion vectors from P to N for the blocks of the lowest pyramid level (i.e. the level having the original resolution) only and the backward motion vectors from N to P for the blocks of the lowest pyramid level only. The forward global motion vector from P to N, and the backward global motion vector from N to P are used, as well as q, which is the time stamp of the interpolated frame and is a value between 0 and 1. The output is an interpolated frame.
In the post processing module 128, the logo detection logical array is scanned. In each position where there is a logo indication, the previously computed interpolated pixel is replaced by the average of the pixel in the previous frame and the pixel in the next frame at the same location as the interpolated pixel in some embodiments.
Referring to
Referring to
The algorithm for choosing the spatial predictors is as follows: for each case you need to choose the spatial predictors for the blocks with light blue color. Look at all blocks in
A computer system 130, shown in
In some embodiments, the bi-directional approach and using adaptive block size with the global motion vector may reduce the artifacts near object edges since these image regions are prone to motion field inaccuracy due to an aperture problem that arises in the one directional method.
While the aperture problem itself is not solved by the bi-directional approach, the final interpolation is more accurate since it relies on the best results from the two independent motion fields.
The graphics processing techniques described herein may be implemented in various hardware architectures. For example, graphics functionality may be integrated within a chipset. Alternatively, a discrete graphics processor may be used. As still another embodiment, the graphics functions may be implemented by a general purpose processor, including a multicore processor.
References throughout this specification to “one embodiment” or “an embodiment” mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation encompassed within the present invention. Thus, appearances of the phrase “one embodiment” or “in an embodiment” are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be instituted in other suitable forms other than the particular embodiment illustrated and all such forms may be encompassed within the claims of the present application.
While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.