Patent Application
20100014715
The invention relates to an image processing apparatus and a method thereof and, more particularly, to an image processing apparatus for generating matching cost with texture consideration and a method thereof.
Nowadays, video information applied to various electronic devices has enriched our daily life. High quality video information having large data amounts, however, increases the level of the difficulty in processing, storage and transmission. Video compression is therefore needed for reducing the cost of transmitting or storing the video information. In video compression, one important technique is utilizing a motion vector to represent how an object shifts between successive frames. Consider a car moving across a silent field: after obtaining the information of the silent background, the information worth processing or transmission is the motion of the car rather than every entire frame.
In conventional methods, to recognize “the object” in successive frames, two successive frames are both partitioned into several blocks wherein each blocks contains several pixels (e.g., 8×8 pixels or 16×16 pixels). A reference block, “the object”, is selected from blocks of the previous frame, and a target block is selected from blocks of the instant frame. The difference between the target block and the reference block is measured between each pixel of the target block and the reference block. The measured difference represents the matching cost between the reference block and the target block. By selecting other blocks of the instant frame as new target frames and measuring each matching costs corrseponding to each new target frames, a target block having the lowest matching cost value is selected as the best matched block. The best matched motion vector, therefore, can be obtained as a shifting of the best matched block and the reference block. The above-mentioned image processing technique is also refered to as motion estimation.
However, in some applications, like tracking or frame rate conversion (FRC), etc., finding the “true motion” is more important than finding the best matched block.
To overcome the above-mentioned problem, one objective of the invention is to provide an image processing apparatus with texture consideration and a method thereof in order to find the “true motion” of the object.
According to one exemplary embodiment of the present invention, an image processing apparatus is provided. The image processing apparatus comprises a block matching unit, a texture information analyzing unit, and a matching cost generating unit. The block matching unit compares at least a target block and at least a reference block to generate a matching result. The texture information analyzing unit generates a texture analysis result corresponding to texture information of the target block and texture information of the reference block. The matching cost generating unit is coupled to the block matching unit and the texture information generating unit, and generates a matching cost according to the matching result and the texture analysis result.
According to another exemplary embodiment, an image processing method employed by the above-mentioned image processing apparatus is provided. The image processing method comprises comparing at least a target block and at least a reference block to generate a matching result, generating a texture analysis result corresponding to texture information of the target block and texture information of the reference block, and generating a matching cost according to the matching result and the texture analysis result. These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.
Certain terms are used throughout the description and following claims to refer to particular components. As one skilled in the art will appreciate, electronic equipment manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is coupled to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.
The steps listed above can be performed in any order, and any of the included steps can be integrated, separated, or omitted so as to obtain substantially the same result and goal of the method. Any such manipulation of the steps above should be considered within the scope of the invention.
Please refer to
One important piece of information is the edge information of a block. An edge indicates discontinuousness in terms of pixels, and implies that “an object” could be outlined by that edge. In other words, finding the edge is helpful towards finding the object and, in turn, the “true motion” of the object. In addition, other important information would include variance information and a frequency response of a block. The variance information indicates entropy or a complexity of a block. If the block has a higher entropy value or is more complex, this implies that the block might have some objects or some important information deserving to be processed. The variance information can be roughly defined as the absolute-sum of difference between nearby pixels making up the block, as shown in the following Equation (1). Or more precisely, the variance information can be defined as the square-sum of difference between each pixel and the pixel average, as in the following Equation (2). Either one can be employed by the present invention, depending on the design considerations.
In Equations (1) and (2), xij indicates each pixel consisting the block. x0 indicates an average value of pixels consisting the block. VR indicates the variance information.
Furthermore, the purpose of considering the frequency response is similar to that for the variance, which is also utilized to indicate the complexity of the block. The frequency response of the block discloses the percentage of each different frequency band from the information making up the block, such as high-band, middle-band and low-band information. The frequency response of the block can be derived from applying a Fourier transform to the block or from applying other transforms like Fourier-related transforms. If the frequency response discloses that the block consists of various frequency band information, this implies that the block has some important information deserving to be processed.
Under the teachings of the above disclosure, people skilled in the art can readily realize that to determine the matching cost corresponding to the blocks, not only the difference between the blocks has to be considered but also the content of the blocks. In other words, the texture information of the blocks has to be considered when determining the matching cost corresponding to the blocks. According to Step 320, the texture information analyzing unit 220 generates a texture analysis result TR corresponding to texture information of block 2n and block 4n-1. Certainly, the texture analysis result TR can comprise edge information, variance information, frequency response information, or any combination of those. The texture information analyzing unit 220 is implemented for determining the texture information of the blocks. In this embodiment, for example, the texture information analyzing unit 220 can be a prewitt filter or a soble filter for determining edge information of block 2n and block 4n-1. People skilled in the art can readily realize that the texture information analyzing unit 220 can be other units for determining the variance information or the frequency response of block 2n and block 4n-1.
After obtaining the matching result MR and the texture analysis result TR, the matching cost MC of block 2n and block 4n-1 can be determined from both of them. According to Step 330, the matching cost generating unit 230 generates a matching cost MC according to the matching result MR and the texture analysis result TR. A block having abundant contents should have a higher opportunity than blank or smooth blocks to be selected for determining the corresponding motion vector. The matching cost generating unit 230 can simply be implemented using a subtractor for subtracting the texture analysis result TR from the matching result MR to generate the matching cost MC, or can be implemented using other units for generating the matching cost MC by considering the influence of the texture analysis result TR. The generated matching cost MC can be utilized in different image processes, like generating a motion vector. It should be noted that using the matching costs generated from the image processing apparatus 200 shown in
In this exemplary embodiment, the mapping unit 222 is used to simplify the computation complexity. For example, when the texture analysis result TR indicates that the edge information of block 2n is higher than a predetermined value, the edge information of block 2n is mapped to logic “1”, representing that an edge exists in block 2n; and when the edge information of block 4n-1 is lower than the predetermined value, the edge information of block 4n-1 is mapped to logic “0”, representing that no edge exists in block 4n-1. Table 1 illustrates the edge information as matrix form to generate the new Edge information according to a texture information analyzing unit 220 and a mapping unit 222.
The weighting unit 224 is used to adjust the influence of the texture information, including the texture analysis result TR and the mapped texture analysis result TRm. The generated matching result MR and texture information can be generated based on different units of measurement. In addition, the texture information could have different meanings in different applications; for example, sometimes an edge is vital, and sometimes the edge can be omitted. The texture information, therefore, can be weighted into an appropriate amount. Please note that each piece of information comprised in the texture information (e.g., the edge information, the variance information and/or the frequency response of the target block or the reference block) can be weighted by different values independently.
The motion vector decision unit 240 can utilize the matching cost MC having texture consideration included therein to determine a corresponding motion vector. For example, the matching cost MC1′ corrresponding to block 2n-1 of frame n−1 and block 4n of frame n, the matching cost MC2′ corrresponding to block 3n-1 of frame n−1 and block 4n of frame n, and the matching cost MC3′ corrresponding to block 4n-1 of frame n−1 and block 2n of frame n are obtained as the above embodiments. The matching cost MC2′ will have a higher value compared to MC1′ and MC3′, since the difference between block 3n-1 and block 3n is bigger. The matching cost MC3′ is lower than the matching cost MC1′ since blocks 4n-1 and 2n have more abudant texture information than the blocks 2n−1 and 4n have. The motion vector decision unit 240 determines a motion vector consisting of a pair of bidirectional motion vectors directing from block 3n to blocks 4n-1 and 2n, respectively, for the interpolated block 3i according to the matching cost MC3′. In this way, the interpolated block 3i is no longer incorrectly determined as a blank block. The twinkle phenomenon, therefore, can be solved. Please note that any unit(s) of the mapping unit 222, the weighting unit 224, and the motion vector decision unit 240 can be omitted, depending on different application requirements. In other words, any image processing devices having texture information involved in determining a matching cost fall in the scope of the invention.
To conclude, the apparatus and method provided by the embodiments of the invention generate the matching cost having texture information consideration, which is helpful for finding a “true motion” of an object, and especially helpful in the FRC or tracking application.
Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention.
