Patent Application
20110205438
The present invention is related to a method and an apparatus for motion estimation and motion compensation in video image data and especially for line based field rate up-conversion motion estimation and motion compensation of video image data. The present invention is further related to a computer program product and a data carrier comprising a computer program.
The present invention relates to a motion estimation and compensation device and more particularly to a motion estimation and motion compensation device that estimates motion vectors and performs motion compensated predictions of an interlaced and non interlaced sequence of chrominance sub-sampled and non sub-sampled video frames.
Hereinafter, the present invention and its underlying problem is described with regard to the processing of a video signal for line-based motion estimation and motion compensation within a video processing apparatus such as a microprocessor or microcontroller having line memory devices, whereas, it should be noted, that the present invention is not restricted to this application, but can also be used for other video processing apparatus.
The market introduction of TV-sets based on 100/120 Hz frame rate or even higher required the development of reliable Field/Frame Rate Up-conversion (FRU) techniques to remove artefacts within a picture such as large area flickers and line flickers. Standard FRU methods, which interpolate the missing image fields to be displayed on Displays without performing an estimation and compensation of the motion of moving objects in successive image fields are satisfactory in many applications, especially with regard to a better quality of the image and with regard to the reduction of the above-mentioned artefacts. However, many pictures contain moving objects, like persons, subtitles and the like, which cause so-called motion judders.
This problem is better understood by referring to
In order to avoid such blurring effects and to reduce artefacts several methods for motion estimation and motion compensation—or shortly MEMC—are proposed. This MEMC provides the detecting of a moving part or object within the received image fields and then the interpolation of the missing fields according to the estimated motion by incorporating the missing object or part in an estimated field.
This motion of an object in successive image fields/frames can be represented by a so-called motion vector. The motion vector AB represents the motion of the object MO from position A in the previous field T to position B in the current field/frame T+1. This motion vector AB typically has a horizontal and a vertical vector component. Starting from point A in the previous field T and applying this motion vector AB to the object MO the object MO is then translated in position B in the current field/frame T+1. The missing position I of the object MO in the missing field/frame T+½ that has to be interpolated must be calculated by the interpolation of the previous field T and the current field T+1 taken account of the respective positions A, B of the moving object MO. If the object MO does not change its position between the previous field/frame and the current field/frame, e. g. if A and B are the same, position I in the missing field is obtained by the translation of A with a motion vector |AB|/2. In this manner the missing field T+½ is interpolated with a moving object in the right position with the consequence that blurring effects are effectively avoided.
Theoretically, for each pixel of a field a corresponding motion vector has to be calculated. However, this would increase the number of calculation needed and thus the memory requirements enormously. To reduce this enormous calculation and memory effort there exist basically two different approaches:
The first approach employs a so-called block-based MEMC. This first approach assumes that the dimension of the object in the image is always larger than that of a single pixel. Therefore, the image field is divided into several image blocks. For MEMC only one motion vector is calculated for each block.
The second approach employs a so-called line-based MEMC. In this second approach the algorithm is based on a reduced set of video input data of a single line of a field or a part of this line. The present invention is based on this second MEMC approach.
In present line-based MEMC systems, image data is usually stored in a local buffer or on chip memory, the so-called line memory, to which rather extreme bandwidth requirements are made. Many present MEMC systems, like the implementations described by Gerard de Haan in EP 765 571 B1 and U.S. Pat. No. 6,034,734, apply a cashe memory (e.g. a two-dimensional buffer) to reduce the bandwidth requirements and to store a sub-set of an image. The motion compensation device fetches video image data from this cashe while applying motion vectors. Typically, in MEMC systems this cashe covers the whole search range of the motion vectors. Usually, the cashe consists of a great amount of so-called line memories. This results in a relatively large amount of memory, e.g. 720 pixels wide and 24 lines (with an associated maximum vertical vector range of [−12-+12]. Such a cashe comprising a great amount of single line memories requires a huge memory needed only for MEMC data buffering. As a consequence, the memory portion within the processor covers a relatively sizable chip area.
Commonly used MEMC algorithms compensate the motion in two directions, i.e. the motion in the horizontal direction and as well in the vertical direction. For that operation a memory access should be randomly possible, which requires for an application in hardware sufficient embedded chip memory within the video processor for the different temporal incoming data streams. The size of this embedded chip memory strongly depends on the search range (i.e. search area) for the motion of an object, as already outlined above, where the motion estimation can match similar video patterns in two temporal positions and derive the velocity of the motion in terms of pixels per frame or per field.
However, this matching process does not always work perfectly, since methods to determine the quality of the measured motion vector are required. Therefore, for the internal storage of further temporal incoming video signals additional memory resources are required. This, however, increases the amount of embedded memory even further, which leads to an increase of the chip area since for an integrated circuit it is the chip internal memory which significantly determines the chip area. Consequently, the chip is getting more and more expensive. Especially in the mainstream market segment such as for modern Plasma- and LCD-TVs these additional costs typically form a limiting factor for an MEMC implementation.
Therefore, there is a need for an efficient internal memory included in a motion estimation device to store data from a reference frame.
The present invention is, therefore, based on the object to provide a more efficient use of the chip-internal resources and especially of the chip-internal memory with regard to motion estimation and motion compensation.
In accordance with the present invention, a method comprising the features of claim 1 and/or an apparatus comprising the features of claim 16 and/or a computer program product comprising the features of claim 22 and/or a data carrier comprising the features of claim 23 is/are provided.
Accordingly, it is provided:
A method for line-based motion estimation and motion compensation in video image data, especially for motion estimated and compensated field rate up-conversion in consecutive frames of a motion picture, comprising the steps of: providing a video signal comprising video image data of a line or part of the line of the picture; buffering the video image data in at least one line memory having the size of one video line or at least of the incoming or actually processing video image data.
An apparatus for line-based motion estimation and motion compensation in video image data, especially for motion estimated and compensated field rate up-conversion in consecutive frames or fields of a motion picture, wherein the apparatus is configured to perform a method according to the present invention.
A computer program product comprising a code, said code being configured to implement a method according to the present invention.
A data carrier comprising a computer program product ac-cording to the present invention.
One basic idea of the present invention is based on the conclusion that for the main stream market the performance and therefore the search range can be limited to the occurrence of the most likely motion direction in natural captured scenes, since most of the motion in natural scenes has only one direction. The present invention describes a method for motion estimation and motion compensation which operates only in one direction and therefore performs the motion estimation and motion compensation operations using a single line buffer memory, the so-called line memory. This offers the possibility to reduce the chip embedded memory to one single line memory for the previous and one single line memory for the current field or frame. This advantageously enables significant silicon area reducing and cost saving implementations.
In a preferred embodiment of the present invention, the MEMC is limited to motion in the horizontal direction only, since most of the motion in natural scenes has this direction.
In video signal processing line memories are often used in other applications which already have access to the previous and current motion portrayal, e.g. like so-called de-interlacer applications or temporal noise reduction applications. In a preferred embodiment these already existing line memories of the video application are now additionally used also for MEMC operations. By using existing line memories of the video signal processing system, no further memory bandwidth has to be added to the memory bus and the memory bandwidth is kept uninfluenced.
Thus, this solution offers the possibility to accomplish the MEMC operations by adding a minimal or in the optimal case no additional memory to the video processing system.
The present invention allows a variable performance of matching of accumulation profiles, where beside a comparison of the whole profile also (shift like) matching of subparts of the accumulation profiles is enabled. The later way represents a kind of fine grinned matching increasing accuracy of the matching process and thus of the derived vertical motion vector.
Additionally, a set of vector candidates derived from the previous accumulation profile can be used in the matching. Thus, the efficiency of performance can be increased, as only a set of vector candidates has to be checked.
A further advantage is that several error reduction methods can be employed to reduce the visibility of artefacts.
Advantages, embodiments and further developments of the present invention can be found in the further subclaims and in the following description, referring to the drawings.
In a preferred embodiment the method comprises the steps of: performing the motion estimation by detecting and analysing the video image data and by deriving a motion vector depending on the detected motion, wherein the motion vector contains only motion data of a motion of an object in one direction and especially in the horizontal direction; using the motion vector for motion compensation to interpolate a picture.
In a preferred embodiment the image data of the previous frame is derived from a first line memory and image data of the current frame is derived from a second line memory.
In a preferred embodiment the first line memory and/or the second line memory are further used in a de-interlacer application and/or a temporal noise reduction application.
In a preferred embodiment for the motion estimation a matching process is employed, wherein the luminance profile and/or chrominance profile is/are used as matching parameter.
In a preferred embodiment the matching process comprises the steps of: providing a first series of pixels within a line of a previous frame or field; providing several adjacent second series of pixels within a line of a current frame or field around a given centre pixel for which the motion shall be determined; successively comparing luminance and/or chrominance values of the second series of pixels with the first series of pixels; assigning a quality degree and/or a failure degree to each second series of pixels; selecting that second series of pixels having the highest quality-degrees and/or the lowest failure degrees to be the most probable second series of pixels; computing the motion vector in the horizontal direction on the basis of the selected second series of pixels.
In a preferred embodiment a SAD-based method and/or an ADRC-based method are used for the comparison of the luminance and/or chrominance values.
In a preferred embodiment pre-selected samples of motion vectors are used for the motion estimation.
In a preferred embodiment a variation of at least one selected sample of the motion vectors is performed in order to set up the process of motion estimation and/or to follow the deviation from the constant motion.
In a preferred embodiment the selection of the selected motion vectors is treated differently for the first line of a frame or a field compared to the other lines of the same frame or field.
In a preferred embodiment a damping value which depends on the selected motion vector is used to control those motion vectors with similar failure values and/or to bring the selection process of the motion vector into a desired direction.
In a preferred embodiment a motion vector histogram is used to provide a motion vector ranking on the basis of most and less used motion vectors in a current frame or field.
In a preferred embodiment for detecting unreliable motion vectors the method comprises the steps of: Comparing a luminance pixel value of a motion vector compensated previous frame or field with the luminance pixel value of a motion vector compensated current frame or field; Selecting the current motion vector as an unreliable vector if the evaluated difference in the comparing step exceeds a given threshold value; Marking the selected unreliable vector and/or replacing the unreliable vector by a reliable vector.
In a preferred embodiment artefacts generated by remaining and not replaced unreliable vectors are covered by an intentional blurring of those regions which comprises this artefact.
In a preferred embodiment an additional vertical motion estimation and motion compensation process is performed independently of the horizontal motion estimation and motion compensation process and/or in combination with the horizontal motion estimation and motion compensation process.
In a preferred embodiment the apparatus comprises a first line memory for storing image data of the previous frame and a second line memory for storing image data of the current frame.
In a preferred embodiment the first line memory and/or the current line memory are configured to be further used in a de-interlacer device and/or a temporal noise reduction device.
In a preferred embodiment the apparatus further comprises: an input terminal for providing a video signal comprising video image data; a buffer for buffering the video image data, wherein the buffer comprises at least one line memory having at least the size of one video line or at least of the incoming or actually processing video image data; a motion estimation and compensation device for performing the line-based motion estimation and motion compensation; an output terminal for providing a motion compensated video output signal.
In a preferred embodiment a motion vector memory, especially a third line memory, is provided to store at least one motion vector. However, the motion vectors may also be stored in external memory devices.
In a preferred embodiment the motion estimation and compensation device further comprises: A motion estimation device which generates a motion vector out of the image data stored in the first and second line memories; A motion compensation device which performs a motion compensation using the image data stored in the first and second line memories and employing the vector data.
For a more complete understanding of the present invention and advantages thereof, reference is now made to the following description taken in conjunction with the accompanying drawings. The invention is explained in more detail below using exemplary embodiments which are schematically specified in the figures of the drawings, in which:
In all figures of the drawings elements, features and signals which are the same or at least have the same functionality have been provided with the same reference symbols, descriptions and abbreviations unless explicitly stated otherwise.
In the following description of the present invention first of all a short overview of the motion estimation and motion compensation is presented.
The MEMC method consists mainly of two sections, the motion estimation and the motion compensation method. The motion estimation performs the measurement of the motion and derives the velocity of the displayed regions in pixel per picture (i.e. field or frame). Also the direction of the motion will be indicated by a positive or negative sign. These measured motion information is described in the form of a motion vector. The motion vector is used for the motion compensation to interpolate the picture at the temporal accurate position and to avoid so-called judder effects and/or so-called motion blurring effects.
A line memory 23, 24 as used in the present patent application indicates an embedded memory of a size of one video line of a frame or a field or at least less of the incoming video signal stream or actually processing video signal stream. A field denotes a video image or picture which comprises either odd or even lines. A frame denotes a video image comprising of the complete video information for one picture, i.e. of a field for the odd lines and the corresponding field for the even lines. A line denotes a full horizontal row within a field of one video picture or at least a part of this row.
Both of the line memories 23, 24 are coupled—on their output sides—to the motion estimation device 25 and to the motion compensation device 26. This enables the image data X1, X1′ stored in the line memories 23, 24 to be transferred to the motion estimation device 25 and to the motion compensation device 26, respectively. In
The motion estimation device 25 generates a motion vector signal X4 out of the image data X2, X2′ stored in the line memories 23, 24 by employing a matching process. This vector signal X4 is transferred to the motion compensation device 26. The motion compensation device 26 performs a motion compensation using the image data X3, X3′ stored in the line memories 23, 24 and applying the vector data X4 to this image data X3, X3′. At the output terminal 27, the motion compensation device 27 provides a video signal X5 which comprises information for a motion compensated picture. This video signal X5 is transferred via the output terminal 27 to a display 29, such as a LCD-panel 29 or the like.
With regard to
For the motion estimation a matching process is employed to select a corresponding series of pixels 32 which fits best to a given amount of pixels 30. For this selection a given amount of pixels 30 of a line of a current frame around the centre pixel 31 for which the motion shall be determined is taken from a line memory 24 of the current frame 32. Hereinafter this given amount of pixels 30 is denoted to as series of pixels 30. In the present embodiment a series of pixels 30 comprises 9 single pixels 33. It is self-understood that a series can also comprise a greater or a smaller amount of pixels 33.
For the selection the luminance profile of the pixels 33 is used as the matching parameter. Luminance is a photometric measure of the density of luminous intensity in a given direction. It describes the amount of light that passes through or is emitted from a particular area, and falls within a given solid angle. Thus, luminance is the photometric measure of the brightness in a frame of a motion picture. If the luminance is high, the picture is bright and if it is low the picture is dark. Thus, luminance is the black and white part of the picture.
This luminance profile is used to find out that series of nine pixels 34 out of the previous frame 35 which fits best with the series of nine pixels 30 of the current frame 32. In the embodiment of
A typical value for the search range is, e.g. 64 pixels (+31 . . . −32). However, it may also be possible to use less than 64 pixels, however, then the quality of the result of this comparison is increasingly going down. On the other hand it is also possible to use more than 64 pixels. Then the quality of the selection result is going up, however, this needs more computational effort. Therefore, typically a trade-off which provides an optimization between best quality of the selection result and simultaneously a minimum computation effort is employed.
In a preferred embodiment for each selected motion vector 37 a single matching process is performed in the way described above. This matching process is performed by assigning a quality degree and/or a failure degree for each series of pixels 30. Then, a quality-degree and/or a failure degree are assigned to each one of those series of pixels 30 which are undergoing the matching process. Those series of pixels 30 having the highest quality-degrees and/or the lowest failure degrees are selected as most probable series of pixels. These series of pixels 30 are then used for computing the motion vectors for the horizontal motion. Typically, but not necessarily a SAD method (SAD=sum of absolute difference) and/or ADRC (Adaptive Dynamic Range Coding) method is used for the comparison of the luminance and/or chrominance values.
Using Pre-Selected Motion Vector Samples for the Motion Estimation:
Assuming the motion of an object in the scene will be constant from field/frame to field/frame and the object is larger than a series of pixels (e.g. the above mentioned 9 pixels), then the matching process can then be performed more efficiently if a set 38 of pre-selected motion vectors 37—the so-called motion vector samples 37—are checked for a matching of the luminance profile (see
The zero-vector which indicates no motion of the object is typically one of the most used motion vector samples. This zero-vector is used in order to more efficiently detect regions within a picture showing no motion. In principle the amount of pre-selected motion vectors 37 which will taken into account depend strongly on what kind of motion vector quality is desired.
Variation of Selected Motion Vectors:
In order to set up the process of motion estimation and to follow the deviation from the constant motion, a variation of certain pre-selected motion vectors is required for test operation purposes. That means that for pre-selected motion vector samples a certain amount of motion will be added or subtracted. This can be done by applying a variance with different amount of motion speed to these motion vectors. The tested implementation checks between odd pixels and even pixels alternating an update of +/−1 pixels and +/−4 pixels on the previously determined motion vector. The selection of the variance is adjustable and variable as required or as the need arises and depends e.g. on the resolution of the incoming video signal.
For the line-based motion estimation it is very advantageous that the motion vector will converge quickly for the real motion in the scene. Therefore, the selection of the tested motion vectors is treated differently for the first line of a frame or field. For the first line of a frame or field testing is not possible in the normal way since a line above the first line which is needed for testing is not existing. In the first line of each field the selected motion vectors which normally test the motion vectors of the line above are loaded with vector values, which e.g. vary according to a triangle function from pixel to pixel. The triangle function oscillates between an adjustable minimum value and an adjustable maximum value. For that purpose also other regular oscillating functions e.g. a saw tooth function, a sinusoidal function, and the like may be employed for the determination of the motion vector of the first line.
The Matching Process:
In a preferred embodiment the matching process assigns a failure value to each tested motion vector. In another embodiment this value may be also a quality value. It might also be possible to evaluate as well a failure value and a quality value for the matching process. Preferably, the sum of the absolute difference (SAD) is used as the failure value or to at least derive the failure value. Ideally, to find the optimal motion vector a failure value of zero is needed. However, typically the failure value is different from zero. Therefore, the motion vector corresponding with the lowest failure value is then selected as the most probably motion vector representing the motion of an object in the local scene.
Attenuation of the Vector Selection, Vector Damping:
In a preferred embodiment a damping value is used which depends on the vector attenuation of the different motion vectors. This enables to control the motion vectors with equal failure values and/or to furnish the motion vector selection process with a certain direction.
Vector Memory:
The different motion vectors are advantageously stored in a vector memory. These motion vectors can be then—if required—fetched from the vector memory for further processing and/or for the motion estimation of the next pixels.
The motion estimation process forms a recursive process. Therefore, the size of this vector memory mainly depends on the desired quality level of the matching process. In one embodiment, the tested implementation comprises only one line of a vector memory. In this vector memory every second motion vector will be stored alternately, in order that an access of the motion vectors from the measured line above is possible.
Robustness Improvement by Providing a Vector Histogram:
In a preferred embodiment a motion vector histogram is calculated in order to create a highly reliable and homogeneous field of motion vectors. This vector histogram allows a vector majority ranking to derive most and less used motion vectors in an actual scene.
The provision of a motion vector histogram can be done either for the whole frame or field or only for parts of the frame or field. It is very efficient to split the picture into horizontal stripes and return a most often used vector for each stripe. In very a preferred embodiment news ticker information within a picture can be detected in that way very reliable.
Unreliable Vector Detection:
Under certain circumstances the motion estimation will not deliver reliable motion vectors. Especially for vertical movements as well as for occlusions and uncovered areas the matching process sometimes does not provide a reliable access of the current and previous line. Not reliable motion vectors, however, lead to relatively large, undesired differences for the compensation process.
Bad Vector Replacement:
The search process starts e.g. from the most outer border of the search range in order to find a reliable vector of the surrounding area. If the bad vector signal 60 indicates no bad vector 61 then this motion vector is used for the further motion estimation process 62. Otherwise, a search process 63 is started to select a motion vector from previous estimation processes. If this search process 63 indicates a bad vector 64 then this bad vector is replaced by a reliable vector 65 (see also steps 53-55 in
Adaptive Error Region Extension and Error Spike Suppression:
The remaining unreliable vectors typically lead to artefacts in the interpolated picture. In one embodiment—as already outlined above with regard to FIG. 8—these artefacts can be covered by an intentional blurring of this artefact comprising regions. In a preferred embodiment these artefact comprising regions—hereinafter also referred to as error regions—are cleaned up from single spikes in order to provide efficient artefact concealment. This process is shown with regard to
The artefact concealment according to the present invention may be performed on the basis of an error spike suppression and/or an error region extension method. In FIGS. 9(A)-(C) the left diagram side denotes the vector samples before the artefact concealment and the right diagram side denotes the vector samples after the artefact concealment. The reliable motion vectors are denoted by reference number 70 and the unreliable motion vectors which additionally comprise an arrow to illustrate the spike are denoted by reference numbers 71, 72, 73.
Single unreliable vectors 71—hereinafter also referred to as spikes—will be suppressed (
This artefact concealment operation ensures a proper behaviour of the blurring filter. In a preferred embodiment, the blurring filter is used only for blurring operations in the horizontal direction.
An error spike denotes the single occurence of one error surrounded by a mostly error-free region.
It is self understood that the above mentioned numerical data is merely illustrative and may be adapted to best provide an optimized blurring effect.
The motion estimation device 25 comprises a matching device 80, a cost/quality function device 81 and a vector selector device 82, which are arranged in series connection between the input side 83 of the motion estimation device 25 where the image data signals X1, X1′ stored in the both line memories 23, 24 are provided and the output side 84 of the motion estimation device 25 where the motion vector signal X4 in present. In the device elements 80-82 a matching process and a vector selection as described with regard to
The motion estimation device 25 further comprises a vector quality device 85 which is connected on the one hand to the input side 83 and on the other hand to the output side 84. The vector quality device 85 generates a quality signal X6 comprising an information of the vector quality out of the image data signals X1, X1′ and the motion vector signal X4.
The motion estimation device 25 further comprises a vector histogram device 86 and a vector majority device 87 which are arranged in series connection in a feedback path between the output side 84 and the matching device 80. Here, in the device elements 86, 87 a vector histogram is generated to provide a ranking of most and less used vectors in the actual scene as shown and described with regard to
The motion estimation device 25 may further comprise a further line memory 88 to store the motion vector data X4 and/or the data X6 for the vector quality.
The motion estimation device 25 further comprises a vector sample device 89.
This vector sample device 89 is also arranged in the feedback path and is connected at its input side with the line memory 88, the vector majority device 87 and advantageously with a further device 90. This further device 90 performs a variation of the motion vector samples by using a special signal having a certain magnitude, e.g. a sinusoidal signal, a saw tooth signal or the like. This certain signal is then used for a testing and/or matching process and/or an up-dating process of the first line of a frame or field. However, it might also be possible to randomly up-date different lines of the frame or field. On its output side, the vector sample device 89 is connected at its output side to the matching device 80.
The motion estimation device 25 further comprises a vertical motion estimation device 91. For vertical motions the above described one-dimensional motion estimation algorithm is not able to compensate fully motion in the vertical direction. However, the occurrence of vertical motions can be used to reduce the compensation in same regions of the picture by splitting the picture into different regions to derive vertical motion for each region. In this case the luminance values of the lines in the different region of the same picture will be summed up and stored individually for each line of this picture. This results in an accumulated vertical profile for different regions of the same picture. Then, the whole picture can be divided into smaller regions to derive a vertical motion for each of these regions. This vertical motion estimation process is performed in the vertical motion estimation device 91 which is connected to the input side 83 and which provides at its output side a sector based vertical motion index X7.
Thus, the vertical MEMC as sketched above can be performed independently of horizontal MEMC and also in combination with the horizontal MEMC, wherein the combination can be performed in dependence on a certain situation or the motions present, respectively. Further, such a methodology allows an implementation of vertical MEMC, which does not need large amounts of additional memory capacity to analyze data of consecutive frames being the case in the most methodologies of the prior art.
The motion estimation device 25 further comprises a vector damping device 92. In this damping device 92 a damping value as described above may be used to damp vector samples of the vector sample device 89 and to provide damped vector samples to the vector selector 82.
Hereinafter the motion compensation process which is performed in the motion compensation device 26 of
The motion compensation device 26 comprises a compensation device 100 which performs the temporal motion interpolation according to the motion vectors X4 estimated by the motion estimation device 25. In a preferred embodiment the compensation device 100 comprises a Median Filter which uses as input data the luminance values of the vector compensated previous line, the vector compensated current and the uncompensated previous line. Additionally, also the chrominance values can be compensated.
Depending on the vector quality a replacement vector indicated as reliable vector will be searched in the local area of the vector memory from the line above. If no reliable vector can be found the adaptive blurring or an other fallback methodology typically tries to cover this artefact.
The motion compensation device 26 further comprises a vertical motion control device 101 which provides a control signal X8 to the compensation device 100 in order to incorporate also information of a vertical motion to the compensation device 100.
The motion compensation device 26 further comprises a bad vector modification device 102. Based on information X4, X6 provided by the motion estimation device 25 the bad vector modification device 102 modifies bad vectors as shown and described above with regard to
The motion compensation device 26 further comprises an adaptive blurring device 103. Based on the motion compensated image data signal X10 and a blurring control signal generated by the bad vector modification device 102 this adaptive blurring device 103 performs an adaptive blurring e.g. such as described with regard to the
Unlike the first embodiment in
On-chip solutions for video processing which are memory-based have already existing internal line buffers 110-112—the so-called line memories 110-112—which carry video data from the previous and current field or frame. These line buffers 110-112 can be located e. g. within temporal noise reductions or de-interlacing units 113 which operate motion adaptive. With the proposed line-based MEMC these line buffers can be reused additionally for the motion estimation. For that purpose and in order to reduce motion judder artefacts from movie sources, a movie detector which indicates the current interpolated sequence of pull-down mode is used. A line buffer selector transfers the video signal data to the motion estimation device according to the previous and the current video input signal. This technique allows using already existing memory resources also for motion estimation which also prevents additional bandwidth for the temporal up-conversion process. Therefore, the chip area for the motion estimation and the motion compensation can be reduced to a minimum.
The de-interlacer device 113 uses three line memories 110, 111, 112 coupled on their input side to the memory bus 22 and providing at their output side line data. This line data provided by the line memories 110, 111, 112 is processed within the de-interlacer device and then provided to the motion compensation device 26. According to the present invention, these line memories 110, 111, 112 are additionally used also for the motion estimation device 25. For this purpose, the system 20 additionally comprises a selector device 114, where a movie sequence X0 is provided to this selector device 114. This movie sequence X0 may be then stored in an external memory 28 via the memory bus 22 and can be read out from this external memory 28 through the line memories 110, 111, 112. For an IMC operation, this data stored in the line memories 110, 111, 112 of the de-interlacer device 113 can be also used for MEMC. For this purpose the data stored in the line memories 110, 111, 112 is then provided as well to the motion estimation device 25 and the motion compensation 26 device.
Unlike the second embodiment in
GFI (Grey Frame Insertion) denotes an up-conversion method to double the incoming frame rate of a video sequence. MCSFI (Motion Controlled Smooth Frame
Insertion) denotes an up-conversion method which is motion detector controlled. Instead of these up-conversion method also a SFI-method (Smooth Frame Insertion) and/or a DFI-method (Dynamic Frame Insertion) may be applied, whereas SFI stands for an up-conversion method to double the incoming frame rate of a video sequence and DFI stands for an up-conversion method to double the incoming frame rate of a video sequence.
Typically for those up-conversion methods the high contrast regions are detected per unit and excluded from the compensation processing due to the incapability of those methods to compensate within these regions. Therefore a pixel gain detector controls a fader, which normally fades over from the processed video signal to the unprocessed video signal in order to reduce the introduction of artefacts. The present invention of the line-based MEMC is capable to perform the up-conversion also in these high contrast regions. Therefore, the signal of the line-based MEMC is applied to the unprocessed fader input of the non-vector-based up-conversion methods.
While embodiments and applications of this invention have been shown and described above, it should be apparent to those skilled in the art, that many more modifications (than mentioned above) are possible without departing from the inventive concept described herein. The invention, therefore, is not restricted except in the spirit of the appending claims. It is therefore intended that the foregoing detailed description is to be regarded as illustrative rather than limiting and that it is understood that it is the following claims including all equivalents described in these claims that are intended to define the spirit and the scope of this invention. Nor is anything in the foregoing description intended to disavow the scope of the invention as claimed or any equivalents thereof.
It is also noted that the above mentioned embodiments, examples and numerical data should be understood to be only exemplary. That means that additional system arrangements and functional units and operation methods and standards may be implemented within the MEMC-system.
| Number | Date | Country | Kind |
|---|---|---|---|
| 07017665.6 | Sep 2007 | EP | regional |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/IB2008/053127 | 8/5/2008 | WO | 00 | 5/24/2010 |
