This invention relates to post processing of motion vectors for video compression and in particular to cluster-based post processing.
Motion Estimation (ME) and Motion Compensation (MC) are used in video coding to exploit temporal redundancy in a moving image sequence and, hence, to assist in achieving compression for transmission or storage. A number of schemes exist to calculate Motion Vectors (MV) that describe motion in a given image. In real-time embedded systems, block matching ME algorithms are used because of their ease of implementation.
Because of its ease of implementation on embedded/DSP platforms, a most popular measure of similarity between current macroblocks and target macroblocks in a reference frame is a Sum of Absolute Differences (SAD) of their respective pixel values. The computation takes each pixel in the current source macroblock 11 and compares it to a pixel in a corresponding position in the reference macroblock 21, taking no account of the sign of the difference value, so that a running sum of moduli of the differences is calculated over the whole macroblock. A reference macroblock 21 within the search area 22 having a least value of this accumulated difference is selected as a best match to the current source macroblock 11. The search is performed over the predetermined search area 22 and using a predefined search pattern. In exhaustive search algorithms, also known as Full Search, the search is performed pixel by pixel on the entire search area 22, and a MV 12, 13, denoting the displacement of the reference macroblock 21 in the reference frame 20 with respect to the source macroblock 11 in the current frame 10 is computed, based on minimal SAD over the whole search. Although SAD is not an exact measure of similarity it provides a good approximation and is readily implemented. Other statistical methods of measuring similarity between macroblocks may also be used but they usually entail additional computational complexity.
Although Full Search algorithms may seem intuitively to provide a best approach to finding a MV, Full Search algorithms often tend to capture sub-optimal MVs. For example, for homogeneous backgrounds of slow or medium motion sequences and in the presence of excessive noise, full search often leads to large MVs partly due to multiple false matches with low SAD values. These spurious MVs are suboptimal as they are optimized based only on similarity and they are not necessarily coherent with MVs in the same neighbourhood. If they do not follow the motion of neighbouring macroblocks, spurious vectors tend to consume more bits in coding the MV differentials, as required in many popular compression algorithms, thus they are sub-optimal in terms of a combined bit rate and similarity measure.
It is an object of the present invention at least to ameliorate the aforesaid disadvantages in the prior art.
According to the invention there is provided a method of detecting and replacing spurious motion vectors in a video signal compression process. The method comprises determining whether a motion vector of a current macroblock is spurious by determining whether there is a significant difference between the motion vector and motion vectors of neighbouring macroblocks and replacing the motion vector as spurious with a motion vector derived from neighbouring macroblocks if the difference exceeds a predetermined threshold.
Advantageously, comparing the motion vector with motion vectors of neighbouring macroblocks comprises: seeking to form at least one valid cluster of motion vectors of neighbouring macroblocks; and if at least one valid cluster cannot be formed, leaving the motion vector of the current macroblock unchanged.
Advantageously, seeking to form at least one valid cluster of motion vectors of neighbouring macroblocks comprises determining whether the motion vector of a neighbouring macroblock is a member of a cluster containing a motion vector of a first neighbouring macroblock.
Conveniently, determining whether a motion vector is a member of a cluster containing a first member comprising a motion vector of a first neighbouring macroblock comprises determining whether a Manhattan distance between the motion vector and the motion vector of the first member of the cluster is greater than a first threshold and, if not, including the motion vector in the cluster.
Advantageously, the first threshold is 12 pixels.
Conveniently, determining whether a valid cluster can be formed comprises determining whether a cluster including at least three motion vectors can be formed.
Advantageously, the method further comprises, if at least one valid cluster can be formed: determining whether the motion vector of the current macroblock falls within a valid cluster so formed; and, if so, leaving the motion vector of the current macroblock unchanged.
Conveniently, determining whether the motion vector of the current macroblock falls within a valid cluster comprises determining whether coordinates of the motion vector of the current macroblock fall between minimum and maximum values of coordinates of motion vectors which are members of the cluster.
Advantageously, the method further comprises, if the motion vector of the current macroblock does not fall in a valid cluster, determining which valid cluster comprises a best valid cluster for replacing the current motion vector with a motion vector derived from the motion vector of the best valid cluster by confining search areas to restricted regions associated with each valid cluster and conducting a full search over each such search area, calculating an associated SAD and motion vector to determine which search area has a smaller SAD for the current macroblock to define a refined motion vector for the cluster.
Advantageously, the method further comprises determining whether a difference between the current macroblock motion vector and the refined motion vector derived for a best cluster motion vector exceeds a second threshold; and if not, leaving the motion vector of the current macroblock unchanged.
Advantageously, determining a difference between the motion vector and a best cluster motion vector comprises determining a Manhattan distance between the motion vector and the best cluster motion vector.
Conveniently, the method comprises determining whether a Manhattan distance between the motion vector and a best cluster motion vector exceeds eight pixels.
Advantageously, the method further comprises determining, when the motion vector of the current macroblock does not fall within a cluster, whether a product of a predetermined parameter and the SAD of a current macroblock and a target macroblock located by the motion vector is greater than a SAD of the current macroblock and a target macroblock located by the best cluster motion vector. The predetermined parameter is dependent upon a quantization parameter used to encode the current macroblock. If the product is greater, the motion vector of the current macroblock is replaced by the best cluster motion vector and if not the motion vector of the current macroblock is left unchanged.
Conveniently, the predetermined parameter is defined by the equation:
QP is the quantisation parameter, best_MV_SAD is a SAD of a current macroblock and a target macroblock located by the motion vector and best_cluster_SAD is a SAD of the current macroblock and a target macroblock located by a best cluster motion vector.
According to a second aspect of the invention, there is provided a system for detecting and replacing spurious motion vectors in video signal compression. The system comprises: means for determining whether a motion vector of a current macroblock is spurious by determining whether there is a significant difference between the motion vector and motion vectors of neighbouring macroblocks; and means for replacing the motion vector as spurious with a motion vector of a neighbouring macroblock if the difference exceeds a predetermined threshold.
Advantageously, the means for comparing the motion vector with motion vectors of neighbouring macroblocks comprises means for seeking to form at least one valid cluster of motion vectors of neighbouring macroblocks.
Advantageously, the means for seeking to form at least one valid cluster of motion vectors of neighbouring macroblocks comprises means for determining whether a motion vector of a neighbouring macroblock is a member of a cluster containing a motion vector of a first neighbouring macroblock.
Conveniently, the means for determining whether a motion vector is a member of a cluster containing a first member comprising a motion vector of a first neighbouring macroblock comprises means for determining whether a Manhattan distance between the motion vector and a motion vector of the first member of the cluster is greater than a first threshold.
Conveniently, the first threshold is 12 pixels.
Advantageously, a valid cluster comprises at least three motion vectors.
Advantageously, the system further comprises means for determining, if at least one cluster can be formed, whether the motion vector of the current macroblock falls within a valid cluster so formed.
Conveniently, the means for determining whether the motion vector of the current macroblock falls within a valid cluster comprises means for determining whether coordinates of the vector of the current macroblock fall between minimum and maximum values of coordinates of motion vectors which are members of the cluster.
Advantageously, the system further comprises means, if the motion vector of the current macroblock does not fall in a valid cluster, for determining which valid cluster comprises a best valid cluster for replacing the current motion vector with the motion vector derived from the motion vector of a best valid cluster. It is determined which valid cluster motion vector results in a smaller SAD for the current macroblock by restricting a search area associated with each valid cluster and conducting a full search over these areas and calculating an associated SAD and motion vector for the current macroblock to determine which cluster provides a smaller SAD and defining a refined motion vector for that best valid cluster.
Advantageously, the system further comprises means for determining, when at least one cluster can be formed, whether a difference between the current macroblock motion vector and the refined motion vector exceeds a second threshold and if not to leave the current macroblock motion vector unchanged.
Conveniently, the means for determining a difference between the motion vector and the best cluster motion vector comprises means for determining a Manhattan distance between the motion vector and the best cluster motion vector.
Conveniently, the system comprises means for determining whether a Manhattan distance between the motion vector and the best cluster motion vector exceeds eight pixels.
Advantageously, the system further comprises means for determining, when the motion vector of the current macroblock does not fall within a cluster, whether a product of a predetermined parameter and a SAD of a current macroblock and a target macroblock located by the motion vector is greater than a SAD of the current macroblock and a target macroblock represented by a best cluster motion vector. The predetermined parameter is dependent upon a quantization parameter used to encode the current macroblock. Means are provided for replacing the motion vector of the current macroblock if so by the best cluster motion vector and if not leaving the motion vector of the current macroblock unchanged.
Conveniently, the predetermined parameter is defined by the equation:
where QP is the quantisation parameter, best_MV_SAD is the SAD of a current macroblock and a target macroblock located by the motion vector and best_cluster_SAD is the SAD of the current macroblock and a target macroblock represented by a best cluster motion vector.
The object is achieved by the independent claims. The dependent claims relate to further embodiments.
The invention will now be described, by way of example, with reference to the accompanying drawings in which:
In the Figures, like reference numbers denote like parts.
Referring to
1) Formation, step 82, 83 of clusters of motion vectors of neighbouring macroblocks;
2) Refinement, step 84-85, of the motion vectors; and
3) Decision logic, steps 86-87, to determine whether to replace the motion vector.
A preliminary step is therefore, after determining, step 81, motion vectors of a macroblock and of neighbouring macroblocks, to form, step 82, motion vectors of macroblocks neighbouring a currently post-processed macroblock into clusters of motion vectors. Referring to
Referring to
Referring to
Referring to the flowcharts of
Step 33 determines whether a given macroblock has already been taken into a cluster and if so terminates the test and moves on to the next macroblock otherwise it is subjected to the examination process of steps 34-37.
Referring to
The process of
Referring again to
As indicated above, a cluster is considered to be valid if it has vectors of three or more macroblocks. If there are no valid clusters, that is, if there is no close Manhattan distance between any vectors, the process will terminate leaving, step 88, the original current macroblock MV unmodified as macroblock X in
If the current motion vector 51 for macroblock X in
So that current macroblock motion vectors, 61, which fall just outside the valid cluster area, 50, shown in
After such a restricted area search is used a determination is first made to which valid cluster of motion vectors to consider correcting the motion of the current macroblock. This is determined, step 85, by comparing the new refined SAD values for each valid cluster and selecting as a candidate for the current macroblock that motion vector which is associated with the smaller of these SADs. The lower value of the two SADs determines which of the two valid clusters is selected to provide an alternative candidate to the original MV and is defined as the “best cluster MV”. These SAD values may be greater than the original MV SAD for the current macroblock but may be preferable in being associated with vectors close to those within the valid clusters.
Step 3: Decision logic
In this final step a decision is made whether or not to replace, step 89, the original MV with a refined MV. The original MV is replaced with the best cluster MV if both the following conditions are satisfied:
(1+(QP−5)/100)*current_SAD>best_cluster_SAD
where
QP=Estimate of the quantisation parameter for the current macro block
current_SAD=SAD corresponding to the MV of the current macro block
best_cluster_SAD=the lowest of two SADs corresponding to the two extended area candidate clusters.
This QP-based threshold ensures that at higher QPs there is more bias towards cluster motion vectors compared with macroblocks quantised at lower QPs.
Thus, there has been described a cluster-based post processor algorithm of MVs to correct any spurious sub-optimal ME. A full search ME is followed by a cluster-based post-processor algorithm that improves the performance of exhaustive ME. Basically, a normal full search ME is carried out and in addition MVs are computed which are optimized in terms of their neighbourhood similarity. Cluster-based post processing analyses MVs of each macroblock together with the surrounding neighbourhood macroblock MVs to help identify possible spurious vectors and hence leads to a correction of the sub-optimal MVs and thus optimises the MVs in terms of rate and similarity. Thus the proposed ME of full search followed by cluster based post-processing of MVs gives a good approximation of rate distortion optimised MVs. As the post-processing algorithm is applied only when suspected spurious sub-optimum MVs are detected, the average computational requirements of the post-processor are very low.
There has been described a process comprising three steps.
Firstly all 8 neighbourhood macroblock motion vectors, shown in
Starting with V0, the Manhattan distance between V0 and each of the other clusters V1, V2 . . . V7 is calculated. If the Manhattan distance between V0 and any of the other clusters is less than or equal to 12 the two clusters are merged and considered as a single cluster. For example if the distance between V0 and V1 is 8 then V1 will be made part of cluster V0 and V1 ceases to exist as an independent motion vector and becomes a member of the cluster with a motion vector V0. A cluster is defined to be valid only if it has at least 3 elements.
For example let V0 be a valid cluster containing V0, V1, V3 and V5. In this example V1 and V3 and V5 are part of V0 and so they are all ignored when seeking to form a second cluster. Clustering continues for V2 with V4, V6, V7. If this succeeds in merging V2 with V4 and V6 then there will be no clustering based on V3, V4, V5 or V6 as the first member so the process will next try to cluster V7 which will be left alone and as such is invalid as a cluster.
After this clustering of neighbouring macroblocks the valid clusters, of which up to a maximum of two are possible, are counted and if there are none the process terminates.
If there are valid clusters each is compared to the motion vector of the current macroblock to establish whether the current MV is part of any of these valid clusters. The current MV is defined as the MV that the motion estimation process has associated with macroblock X in
This process identifies the best Cluster MV of the valid clusters by determining which valid cluster has the smaller SAD value after conducting a full search over a restricted area around each valid cluster to determine a new refined SAD and a motion vector. The motion vector of this valid cluster is the best valid cluster candidate for replacing the motion vector of the current macroblock. Once the best Cluster MV is found the following decision logic is applied.
If the current MV for macroblock X of
If current MV SAD*(QP based threshold)>Best Cluster MV SAD
then the current MV for macroblock X is replaced with the best Cluster MV.
Number | Date | Country | Kind |
---|---|---|---|
0710857.4 | Jun 2007 | GB | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/EP08/56762 | 6/2/2008 | WO | 00 | 6/1/2010 |