Patent Application
20070110157
The present invention relates to a motion estimation method and device adapted to process a sequence of frames, a frame being divided into blocks of data samples.
The present invention relates to a predictive block-based encoding method comprising such a motion estimation method. It also relates to the corresponding encoder.
The present invention finally relates to a computer program product for implementing said motion estimation method.
This invention is particularly relevant for products embedding a digital video encoder such as, for example, home servers, digital video recorders, camcorders, and more particularly mobile phones or personal digital assistants, said apparatus comprising an embedded camera able to acquire and to encode video data before sending it.
In a conventional video encoder, most of the memory transfers and, as a consequence, a large part of the power consumption, come from motion estimation. Motion estimation consists in searching for the best match between a current block and a set of several candidate reference blocks according to a rate distortion criterion, a difference between the current block and a candidate reference block forming a residual error block from which a distortion value is derived. However, such a motion estimation method is not optimal, especially in the case of a video encoder embedded in a portable apparatus having limited power.
Several authors have developed low-power methods. Some of them propose computational simplifications: such methods are not sufficient anymore. Others try to minimize memory accesses.
In the spatial domain, the paper entitled “A Low Power Video Encoder with Power, Memory and Bandwidth Scalability”, by N. Chaddha and M. Vishwanath, 9th International Conference on VLSI Design, pp. 358-263, January 1996, proposes a technique based on hierarchical vector quantization which enables the ability for the encoder to change its power consumption depending on the available bandwidth and on the required video quality.
In the temporal domain, the paper entitled “Motion Estimation for Low-Power Devices”, by C. De Vleeschouwer and T. Nilsson, ICIP2001, pp. 953-959, September 2001, proposes to simplify the conventional motion estimation but at the cost of a lower compression performance.
Disadvantages of these states of the art are that either the motion estimation method reduces the video quality too much, or that it does not achieve a sufficient memory transfer saving.
It is an object of the invention to propose an efficient way to reduce memory transfer, while keeping satisfying visual quality.
To this end, the motion estimation method in accordance with the invention is characterized in that it comprises a step of computing a residual error block associated with a motion vector candidate on the basis of a current block contained in a current frame and of a reference block contained in a reference frame, said reference block having a same position in the reference frame as the current block has in the current frame, the motion vector candidate defining a relative position of a virtual block containing a first reference portion of the reference block with reference to said reference block, the residual error block being computed from:
a first difference between data samples of the first reference portion and corresponding data samples of a first current portion of the current block, and
a second difference between a prediction of data samples of a second reference portion of the virtual block, which is complementary to the first reference portion, and data samples of a second current portion of the current block, which is complementary to the first current portion.
On the one hand, the motion estimation method in accordance with the invention uses only a restricted set of data samples, which is a reference block having a same position in the reference frame as the current block has in the current frame. Said reference block is also called the collocated block. Thanks to the use of said reduced set of data samples, the motion estimation method according to the invention is an efficient way to reduce memory transfer at the encoder and at the decoder. Moreover, reducing the energy dissipation of a corresponding video encoding circuit increases the reliability of said circuit and allows a significant attenuation of the cooling effort. Therefore production costs are greatly lowered.
On the other hand, said motion estimation method is adapted to determine a motion vector between the first reference portion of the reference block and the first current portion of the current block, i.e. by only taking into account portions of said current and reference blocks which are similar. Said motion vector can vary from (−N+1,−N+1) to (N−1,N−1) if the reference block comprises N×N data samples. In addition, the motion estimation method is adapted to predict missing data samples, i.e. the data samples that belong to the second reference portion of the virtual block. As we will see in further detail later on, this prediction can be done according to different modes. Thanks to the determination of a motion vector and to the prediction of corresponding missing data samples, the motion estimation method according to the invention is capable of keeping a satisfying visual quality.
These and other aspects of the invention will be apparent from and will be elucidated with reference to the embodiments described hereinafter.
The present invention will now be described in more detail, by way of example, with reference to the accompanying drawings, wherein:
The present invention relates to a method of motion estimation for use in a device adapted to process a sequence of frames, a frame being divided into blocks of data samples, for example pixels in the case of video data samples. Said device is, for example, an encoder adapted to encode said sequence of frames.
The present invention is more especially dedicated to the encoding of video frames. It can be used within MPEG-4 or H.264 video encoder, or any equivalent distortion-based video encoder. However, it will be apparent to a person skilled in the art that it is also applicable to the encoding of a sequence of audio frames or any other equivalent encoding.
It is to be noted that the present invention is not limited to encoding but can be applied to other types of processing, such as for example, image stabilization wherein an average of the different data blocks of a video frame is computed in order to determine a global motion of said frame. Such an image stabilization process can be implemented in a camcorder, in a television receiver, or in a video decoder after the decoding of an image.
The motion estimation method may be implemented in handheld devices, such as mobile phones or embedded cameras, which have limited power and which are adapted to encode sequences of video frames.
These conventional encoders are based on DCT transformation, scalar quantization, and motion estimation/compensation (ME/MC). The latter is clearly the most power consuming. When a block is encoded, the motion estimation unit ME looks for the best match for a current block cb in a current frame CF, among several blocks belonging to a search area SA in reference frames RF1 to RF3, as shown in
The present invention proposes to replace the conventional motion estimation by a so-called ‘collocated motion estimation’, which is a restricted way of doing motion estimation, with a search area comprising a reduced set of pixels. In order to maintain a correct encoding efficiency while using less data, it is here proposed to modify the motion estimation process, and to mix it with a spatio-temporal prediction of missing pixels.
Said motion estimation method comprises a step of dividing a frame into blocks of pixels of equal size, for example of N×N pixels, where N is an integer.
Then it comprises a step of computing a residual error block associated with a motion vector candidate MV on the basis of a current block cb contained in a current frame CF and of a reference block rb contained in a reference frame RF. According to the invention, the reference block has the same position (i,j) in the reference frame as the current block has in the current frame. In other words, the reference block is collocated to the current block. The motion vector candidate MV defines a relative position of a virtual block vb containing a first reference portion rbp1 of the reference block rb with reference to said reference block.
The residual error block is then computed from:
a first difference between data samples of the first reference portion rbp1 and corresponding data samples of a first current portion cbp1 of the current block, the first current portion cpb1 corresponding to a translation of the projection in the current frame of the first reference portion according to the motion vector candidate MV, and
a second difference between a prediction of data samples of a second reference portion pred of the virtual block, which is complementary to the first reference portion, and data samples of a second current portion cbp2 of the current block, which is complementary to the first current portion.
In other words, let us note r(x,y) the residual error block value of a pixel of position (x,y) that will be encoded. The residual error block value is computed as follows:
r(x,y)=if(x+vx,y+vy)εrb
rb(x+vx,y+vy)−cb(x,y)
else
pred(rb,cb(x,y))
In general, values of pixels of the second reference portion pred are predicted from values of pixels of the reference block rb but this is not mandatory, as we will see later on.
Such a motion estimation method is called collocated motion estimation method. With said collocated motion estimation, the best match of the current block cb, i.e. the block to be encoded, is searched in the reference block rb. To this end, said motion estimation method is adapted to test different motion vector candidates MV between a first reference portion of the reference block and a first current portion of the current block, a predetermined motion vector candidate corresponding to portions of predetermined size. Said motion vector candidate can thus vary from a motion vector Mvmin of coordinates (−N+1, −N+1) to a motion vector Mvmax of coordinates (N−1, N−1) if the reference block comprises N×N pixels.
The step of computing a residual error block is repeated for a set of motion vector candidates. The motion estimation method in accordance with the invention further comprises a step of computing a distortion value for the motion vector candidates of the set on the basis of their associated residual error block values. The motion estimation method finally comprises a step of selecting the motion vector candidate having the smallest distortion value.
This process is called block matching and is based, for example, on the computing of the sum of absolute differences SAD according to a principle known to a person skilled in the art. The computing step is based, as other examples, on the computing of the mean absolute error MAE on the computing of the mean square error MSE. It will be apparent to a person skilled in the art that the distortion value can be computed using other equivalent calculations. For example, it can be based on a sum of an entropy h of the residual error block and on the mean square error MSE.
The residual error block and the selected motion vector are transmitted according to a conventional encoding scheme.
Except for the motion vector candidate (0,0), some pixels are always missing for the computation of the distortion value. Several ways of predicting the missing pixels can be used.
pred(rb,cb(x,y))=rb(x,y)−cb(x,y).
It is to be noted in FIGS. 4 to 6 that the arrow diff1 represents the computation of the first difference between pixels of the first reference portion rbp1 and corresponding pixels of the first current portion cbp1 and that the arrow diff2 represents the computing of the second difference.
pred(rb, cb(x,y))=rb(proj(x),proj(y))−cb(x,y),
where the proj( ) function is adapted to determine the symmetric p″ of the pixel p′ of the second reference portion pred with reference to a horizontal and/or vertical edge of the reference block and to take the value of said symmetric pixel p″ as the reference value rb(x″,y″), as shown in
According to another embodiment of the invention, a single prediction value pred_value is derived from the reference block rb. The corresponding residual error block value is computed as follows:
r(x,y)=cb(x,y)−pred_value
pred_value is set to the mean of the reference block rb values or the median of said values.
Still according to another embodiment of the invention, strictly spatial prediction is performed. In that case, the reference block is not used. The prediction value pred_value is an average or a median value of a line L of pixels on top of the current block or of a column C of pixels at the left of the current block as shown on
It will be apparent to a person skilled in the art that other methods can be proposed to determine the prediction value. For instance, it can be the most frequent value, i.e. the peak of an histogram of the reference block rb, or a value related to the line L, the column C and/or the reference block rb.
The drawings and their description hereinbefore illustrate rather than limit the invention. It will be evident to a person skilled in the art that there are numerous alternatives that fall within the scope of the appended claims.
For example the motion estimation method in accordance with the invention can be used either with only one prediction function, or with several prediction functions as above described, each prediction function being concurrent, as well as motion vectors are themselves concurrent, and selected via the distortion criterion.
The collocated motion search can be based on a three-dimensional recursive search 3DRS, or a Hierarchical Block Matching Algorithm HBMA algorithm. Sub-pixel refinement can be adopted in the same way. The motion is not restricted to a translation; it can support affine models for instance.
The proposed invention can be applied in any video encoding device were accesses to an external memory represent a bottleneck, either because of limited bandwidth or because of high power consumption. The latter reason is especially crucial in mobile devices, where extended battery lifetime is a key feature. It replaces the conventional motion estimation in any kind of encoder. It can be used, for example, in net-at-home, or transcoding applications.
The motion estimation method in accordance with the invention can be implemented by means of items of hardware or software, or both. Said hardware or software items can be implemented in several manners, such as by means of wired electronic circuits or by means of an integrated circuit that is suitable programmed, respectively. The integrated circuit can be contained in an encoder. The integrated circuit comprises a set of instructions. Thus, said set of instructions contained, for example, in an encoder memory may cause the encoder to carry out the different steps of the motion estimation method. The set of instructions may be loaded into the programming memory by reading a data carrier such as, for example, a disk. A service provider can also make the set of instructions available via a communication network such as, for example, the Internet.
Any reference sign in the following claims should not be construed as limiting the claim. It will be obvious that the use of the verb “to comprise” and its conjugations do not exclude the presence of any other steps or elements besides those defined in any claim. The word “a” or “an” preceding an element or step does not exclude the presence of a plurality of such elements or steps.
| Number | Date | Country | Kind |
|---|---|---|---|
| 03300179.3 | Oct 2003 | EP | regional |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/IB04/03469 | 10/20/2004 | WO | 4/21/2006 |
