Patent Grant
7292283
The present invention relates generally to motion estimation, and more particularly, to an apparatus and method for generating refined sub-pixel vectors for motion estimation from vector correlation values and converged vector correlation values using quadratic approximations.
Feature films are projected at a rate of 24 frames per second. The process of projecting 24 still pictures every second creates the illusion of continuous motion on the screen. In contrast, video formats were designed for cathode ray tube television sets, which work in a completely different manner than film projectors. Televisions create still pictures, line by line, with an electron beam that passes over a phosphor coated screen. When television was first developed, the National Television Standards Committee (NTSC) defined a standard having 480 lines of resolution. In other words, the electron beam was passed over the phosphorous screen from left to right in 480 rows, from top to bottom, of the screen. When televisions were first developed, it was not feasible for the electron beam to scan all 480 rows in a single frame. As a consequence, televisions were initially designed to perform interlacing. With interlacing, every other row is scanned in a first frame and the skipped rows are scanned in the next frame. More recently progressive scan televisions have been developed, which enable all 480 lines to be scanned in each frame. Now with high definition television or “HDTV”, new resolution formats have been developed, such as SMPTE 274M-1995 (1920×1080 resolution) SMPTE 296M-1997 (1280×780 resolution), etc. Regardless of the video resolution used, the frame rate differs than that used by feature films.
Since video and film have different frame rates, a frame rate conversion must be performed before a feature film can be played on a video display. For example with the NTSC format, 30 frames or 60 fields are displayed per second. Thus if only two film frames are recorded for every five frames of video, a video copy of the movie can be created that plays at the correct speed.
With video displays, area flicker is an artifact that becomes visible if the image is not refreshed fast enough. This is particularly irritating with larger displays as the eye is more sensitive to flicker in the peripheral visual regions. To reduce flicker, the display needs to refreshed at a faster rate. A straight forward scheme for up conversion that repeats the same frame more than once is therefore useful for reducing flicker. For example, a conversion from 24 frames per second to 72 frames per second has generally been found to reduce the flicker problem. Unfortunately, the higher frame rate introduces another artifact called “motion judder” when the motion of an object is present in the film.
Motion vectoring is a technique used to alleviate motion judder. With motion vectoring, frames are constructed or predicted from a reference frame using motion vectors and a prediction error. The reference frame can be either unidirectional or bidirectional. With unidirectional motion prediction schemes, the predicted picture frame is constructed from a previous reference picture frame using a motion vector and a prediction error. The reference picture frame is typically a previous frame that has been compressed using an intra-frame coding technique. With bidirectional schemes, the predicted frame is constructed from the best matching of either a previous or a future picture frame using either forward or backward motion vectors and a prediction error.
Sub-pixel accuracy is a key performance issue with motion vectoring is. Sub-pixel accuracy is generally defined as the lighting of pixels having a designated point that lies inside a given polygon. For example, if the pixel center is the designated point, then any pixel with its center inside the polygon will be lit. Otherwise the pixel is not lit. The location of the designated point is arbitrary. If the designated point is shifted from the center to the lower left point of each pixel, then the display of the polygon will be shifted by the distance of approximately 0.5 pixels in the lower left direction.
Interpolating frames with just pixel resolution vectors does not provide very good resolution detail. To improve resolution, the interpolation of sub-pixel vectors has been performed to avoid a loss of detail. One known technique for determining sub-pixel resolution vectors is to generate a sub-pixel resolution image for both the current and previous frame images and then search and locate the best vector in the sub-pixel resolution image. This approach, however, provides a limited increase in resolution given the significant cost in generating and storing the sub-pixel resolution frame information.
An apparatus and method for generating refined sub-pixel vectors for motion estimation from vector correlation values and converged vector correlation values using quadratic approximations is therefore needed.
To achieve the foregoing, and in accordance with the purpose of the present invention, an apparatus and method for generating refined sub-pixel vectors for motion estimation from vector correlation values and converged vector correlation values using quadratic approximations respectively is disclosed. The apparatus and method includes defining a minimum vector position value of a converged vector and then determining a predetermined number of vector correlation samples around the minimum vector position value. The predetermined number of vector correlation samples provide a coarse correlation surface estimation of the minimum vector position value. A correlation surface fitting of the predetermined number of vector correlation samples using a quadratic approximation of the coarse correlation surface estimation of the minimum value is then performed. The correlation surface fitting resulting in a refined sub-pixel minimum vector position with horizontal and vertical components.
The invention, together with further advantages thereof, may best be understood by reference to the following description taken in conjunction with the accompanying drawings in which:
In the figures, like reference numbers refer to like components and elements.
Referring to
Referring to
The quadratic equation F(x) is defined by equation [1]:
Xmin=(½)[(P1−P3)/(P1−2P2+P3)] [1]
Xmin=−b/2a
where x=0 at the converged vector location.
The position of the minimum is where the slope disappears, that is where:
F′(x)=0;
F′(x)=2ax+b=0; or
Xmin=−b/2a
For the case where samples P1, P2 and P3 are integer pixels apart, the interpolation offset from the center integer minimum, in phase samples, is calculated from the following equations:
P1=F(−1)=a−+c [2]
P2=F(0)=c [3]
P3=F(1)=a+b+c. [4]
Solving for a and b, the value of Xmin is:
Xmin=(½)[(P1−P3)/(P1−2P2+P3)] [5]
For one eighth (⅛th) pixel resolution, the sub pixel offset to the minimum is:
Xmin=(4)[(P1−P3)/(P1−2P2+P3)] [6]
Equation [6] can be simplified as a function of variables (P1−P2 and (P3−P2). Therefore, Xmin can be represented by equation [7].
Xmin=(4)[(P1−P3)−(P3−P2)/(P1−P2)+(P3−P2)] [7]
By setting (P1−P2) to the variable “U” and (P3−P2) to the variable “V”, Xmin can be represented by equation [8]:
Xmin=4(U−V)/(U+V). [8]
The function Xmin is thus represented in terms of two variables, U and V. By reducing the numerator and denominator of equation [8] to 4-bit resolution, an 8-bit address can be used in a look-up table that specifies refined minimum Xmin to ⅛ pixel resolution from the variable U and V each quantized to 4 bits.
Referring to
Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the described embodiments should be taken as illustrative and not restrictive, and the invention should not be limited to the details given herein but should be defined by the following claims and their full scope of equivalents.
The present application claims priority from U.S. Provisional Application Ser. No. 60/532,495 entitled “Sub-Pixel Vector Estimations Using Quadratic Approximations”, filed on Dec. 23, 2003, by Steve Selby, assigned to the assignee of the present invention, and incorporated by reference herein for all purposes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4838685 | Martinez et al. | Jun 1989 | A |
| 4873573 | Thomas et al. | Oct 1989 | A |
| 5652629 | Gonzales et al. | Jul 1997 | A |
| 5768438 | Etoh | Jun 1998 | A |
| 5784114 | Borer et al. | Jul 1998 | A |
| 5929913 | Etoh | Jul 1999 | A |
| 6040864 | Etoh | Mar 2000 | A |
| 6192080 | Sun et al. | Feb 2001 | B1 |
| 6414997 | Piccinelli et al. | Jul 2002 | B1 |
| 6424372 | Kaneda et al. | Jul 2002 | B1 |
| 6628845 | Stone et al. | Sep 2003 | B1 |
| 6633612 | Selby | Oct 2003 | B2 |
| 6738423 | Lainema et al. | May 2004 | B1 |
| 6784942 | Selby et al. | Aug 2004 | B2 |
| 7098958 | Wredenhagen et al. | Aug 2006 | B2 |
| 7194126 | Konolige | Mar 2007 | B2 |
| 7248741 | Akimoto et al. | Jul 2007 | B2 |
| Number | Date | Country |
|---|---|---|
| 2303513 | Feb 1997 | GB |
| WO 9401830 | Jan 1994 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20050135488 A1 | Jun 2005 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60532495 | Dec 2003 | US |
