Conversion unit and method and image processing apparatus

Abstract

A conversion unit (200) for converting a series of input image (110-114) into a series of output image (116-126). The time intervals (128,130) between consecutive input images (110-114) might be unequal. The time intervals (e.g. 132) between consecutive output images are substantially mutual equal (116-126). The output images (116-126) are calculated by means of motion compensated interpolation between input images (110-114).

Description

[0001] The invention relates to an image processing apparatus comprising:

[0002] a processor based graphics pipeline for generating a first series of input images comprising successively a first input image and a second input image;

[0003] a conversion unit comprising:

[0004] a means for receiving the first series of input images; and

[0005] a means for calculating a series of output images based on the first series of input images, with an output image being spaced a predetermined time interval from a predecessor if any; and

[0006] a display device for displaying the series of output images.

[0007] The invention further relates to a conversion unit comprising:

[0008] a means for receiving a first series of input images comprising successively a first input image and a second input image; and

[0009] a means for calculating a series of output images based on the first series of input images, with an output image being spaced a predetermined time interval from a predecessor, if any.

[0010] The invention further relates to a conversion method comprising:

[0011] a step of receiving a first series of input images comprising successively a first input image and a second input image; and

[0012] a step of calculating a series of output images based on the first series of input images, with an output image being spaced a predetermined time interval from a predecessor, if any.

[0013] An image processing apparatus of the kind described in the opening paragraph is known from the book “Computer Graphics Principles and Practice” by J. D. Foley, A. van Dam, S. K. Feiner, J. F Hughes, from Addison-Wesley, Reading etc, 1996. In chapter 4 of this book it is described that traditionally image data resulting from a graphics pipeline are stored in a frame buffer. A video controller accesses that frame buffer to transmit data to the display device.

[0014] The frequency of displaying images on the display device is called the display update rate. The frame buffer update, i.e. writing image data into the frame buffer, is usually variable and not properly fitted to the display update rate. To prevent visible artifacts in the images on the display, double buffering can be applied. This means that one of the frame buffers can be accessed by the video controller while the other is updated by the graphics pipeline. The combination of frame buffer(s) and video controller is a conversion unit.

[0015] However, due to e.g. insufficient system resources for the graphics pipeline, the frame buffer update rate may be slow compared to the display frame rate. This badly affects the quality of the images to be seen on the display. Especially movement can be shown shaky. The resources of the graphics pipeline may comprise a processor, a memory device and a data bus.

[0016] It is a first object of the invention to provide an image processing apparatus of the kind described in the opening paragraph which is able to visualize motion in images relatively smoothly.

[0017] It is a second object of the invention to provide a conversion unit of the kind described in the opening paragraph which is able to generate a series of output images showing a relatively smooth motion.

[0018] It is a third object of the invention to provide a conversion method of the kind described in the opening paragraph of generating a series of output images showing a relatively smooth motion.

[0019] The first object of the invention is achieved in that the conversion unit is designed to determine a first time interval between a first time of generation of the first input image and a second time of generation of the second input image and to calculate a particular output image by interpolating between the first input image and the second input image on the basis of the first time interval. The major advantage of this conversion unit is that it generates new images by means of temporal interpolation of the images of the input series. An image apparatus according to the prior art only “duplicates” images: it displays the original images several times. With a conversion unit according to the invention it is possible that the series of output images comprises copies of the images of the input series, besides the new images. The conversion unit requires information about the time of generation of the various input images, in order to control the interpolation. The result is that the conversion unit works even if the first time interval is substantially different from a second time interval between the second time of generation of the second input image and a third time of generation of a third input image of the input series. The first time interval and the second time interval might be different, e.g. because of a variable load on the resources of the graphics pipeline. Temporal interpolation is known in up-conversion in Philips television systems. However in that case the time intervals between consecutive images of the input series are mutually equal. Hence it is not required to determine the time intervals in order to control the interpolation in that case.

[0020] An embodiment of the image processing apparatus according to the invention further comprises a motion estimator for controlling the means for calculating the series of output images, in order to perform motion compensated interpolation. This might be e.g. a motion estimator as described by G. de Haan, et al in “True motion estimation with 3-D recursive search block-matching,” in Proceedings IEEE Transactions on Circuits and Systems for Video Technology, 1994, vol. 3, pp. 249-256. The advantage of motion compensated interpolation is an improved image quality. For the motion estimation it is possible to use more than two consecutive images. It is also possible to use more than two consecutive images for the interpolation.

[0021] An embodiment of the image processing apparatus according to the invention further comprises a second means for receiving a second series of input images and that the conversion unit is designed to calculate the series of output images based on images of the first series of input images and the second series of input images. An example of this embodiment is a TV being able to up-convert a sequence of video images together with the graphics as generated by the graphics pipeline, representing the settings of the TV, i.e. onscreen display. Some parts of the conversion unit can be shared to save costs. Two types of sharing can be distinguished:

[0022] Temporal: parts of the conversion unit are used alternatingly for processing images of the first input series and images of the second input series. Optionally intermediate results are stored temporarily.

[0023] Spatial: use is made of the fact that portions of images of the first series have to be merged with portions of the images of the second input series. It is possible to share e.g. memory locations that correspond with these portions.

[0024] In an embodiment of the image processing apparatus according to the invention the conversion unit is designed:

[0025] to calculate a first intermediate motion compensated interpolated image by interpolating between the first input image and the second input image of the first series of input images;

[0026] to calculate a second intermediate motion compensated interpolated image by interpolating between images of the second series of input images; and

[0027] to merge the first intermediate image with the second intermediate image in order to create a combined output image.

[0028] It is possible to change the order of interpolation and merging. However the order as chosen for this embodiment results in superb image quality of the output images.

[0029] The second object of the invention is achieved in that the conversion unit is designed to determine a first time interval between a first time of generation of the first input image and a second time of generation of the second input image and to calculate a particular output image by interpolating between the first input image and the second input image on the basis of the first time interval.

[0030] These and other aspects of the image processing apparatus and of the conversion unit according to the invention will become apparent from and will be elucidated with reference with respect to the implementations and embodiments described hereinafter and with reference to the accompanying drawings, wherein:

[0031] FIG. 1A schematically shows an embodiment of the image processing apparatus;

[0032] FIG. 1B schematically shows an embodiment of the image processing apparatus with a motion estimator;

[0033] FIG. 1C schematically shows an embodiment of the image processing apparatus which is designed to process two series of input images simultaneously;

[0034] FIG. 2A schematically shows an embodiment of the conversion unit;

[0035] FIG. 2B schematically shows an embodiment of the conversion unit with a motion estimator;

[0036] FIG. 2C schematically shows an embodiment of the conversion unit which is designed to process two series of input images simultaneously; and

[0037] FIG. 3 schematically shows that input images of two series are interpolated in the temporal domain and merged into one series of output images.

[0038] FIG. 1A schematically shows an embodiment of the image processing apparatus 100 which comprises:

[0039] a graphics pipeline 101 with a processor 108;

[0040] a conversion unit 105 with:

[0041] a means for receiving 102 the first series of input images 110-114, generated by the graphics pipeline 101; and

[0042] a means for calculating 104 a series of output images 116-126; and

[0043] a display device 106 for displaying the output images. This display device is optional. It is also possible that the output images are displayed on a display device external to the image processing apparatus 100, e.g. in the case of a Set Top Box.

[0044] The first input image 110, the second input image 112 and the third input image 114 are generated on time T1, T2 respectively T3. The conversion unit 105 is designed to determine the time interval between consecutive input images, e.g. a first time interval 128 between T1 and T2 or a second time interval 130 between T2 and T3. In order to determine a time interval 128,130 the conversion unit 105 requires the information about the time of generation of input images. This information is provided by the processor 108 of the graphics pipeline 101. The various time intervals 128,130 between generation of input images might be unequal. The output images 116-126 are generated on TA, TB, TC, TE respectively TF. The time interval 132 between two output images e.g. 116,118 is substantially equal to a predetermined value. The output images 116-126 are calculated by means of interpolating between images of the series of input images. The table below gives an example of the relation between input and output images.

Output image Is based on input images
116          110 and 112
118          110 and 112
120          112 and 114
122          112 and 114
124          112 and 114
126          112 and 114

[0045] FIG. 1B schematically shows an embodiment of the image processing apparatus 103 with a conversion unit 107 that comprises a motion estimator 109. This might be e.g. a motion estimator as described by G. de Haan, et al in “True motion estimation with 3-D recursive search block-matching,” in Proceedings IEEE International Conference on Circuits and Systems for Video Technology, 1994, vol. 3, pp. 249-256. The motion estimator 109 provides motion vector fields to the means for calculating 104 a series of output images 116-126. The vectors of these motion vector fields are used to shift blocks of pixels, i.e. portions of the input images 110-114, in order to calculate motion compensated output images 116-126. The means for calculating 104 a series of output images requires three types of input:

[0046] input images, being two dimensional arrays of pixel values;

[0047] motion vector fields, being two dimensional arrays of vectors; and

[0048] timing, scalar values indicating the generation time of the various images.

[0049] FIG. 1C schematically shows an embodiment of the image processing apparatus 111 which is designed to process two series of input images simultaneously. Besides a first series of input images 308,310 a second series of input images 302-306 is provided. The input images 302-306 of the second series are generated external to the image processing apparatus 111. The input images 302-306 of the second series are provided to the processing apparatus 111 via the input connector 106. These images might come from e.g. a broadcast or from a local storage device. The conversion unit 113 comprises a second means for receiving 115 and to buffer the second series of input images 302-306. The moments of time T1, T3 and T4 of creation of input images 302, 304 respectively 306 are equidistant in time space. The size of the images of the first series and of the second series might be different. The means for calculating 104 a series of output images is designed to merge images even if they differ in size. This can be achieved by means of zooming. It is also possible that an entire image of one of the series is merged with a portion of the other series in the case that the latter image is substantially bigger than the former image.

[0050] FIG. 2A schematically shows an embodiment of the conversion unit 200. The conversion unit 200 requires a series of input images 110-114 on its input connector 206. Input images 110-114 are buffered in the memory device of the receiving means 202. The memory device of the receiving means is designed to store at least two input images. The conversion unit 200 provides a series of output images 116-125 on its input connector 208. The conversion unit 200 requires for each of the input images 110-114 information about the time of creation T1-T3 on its control connector 201. The output images 116-126 are generated equidistant in time space: After each predetermined time interval 132 a new output image is generated by the means for calculating 204 based on the input images as stored in the memory device of the receiving means 202. The interpolation as applied by the means for calculating 204 is controlled by the relation between the times T1-T3 of creation of the input images and of the time of creation TA, TB, TC, TD, TE, TF of a particular output image.

[0051] FIG. 2B schematically shows an embodiment of the conversion unit 203 that comprises a motion estimator 205. The motion estimator 205 provides motion vector fields to the means for calculating 204 a series of output images. The vectors of these motion vector fields are used to shift blocks of pixels, i.e. portions of the input images 110-114, in order to calculate motion compensated output images 116-126.

[0052] FIG. 2C schematically shows an embodiment of the conversion unit 207 which is designed to process two series of input images simultaneously. The first series of input images is provided to the conversion unit 207 at the first input connector 206. The second series of input images is provided to the conversion unit 207 at the second input connector 211. A subset of the first series is stored in the memory device of the first means for receiving 202. A subset of the second series is stored in the memory device of the second means for receiving 209. The motion estimator 205 switches between calculating a motion vector field corresponding to images of the first series and calculating a motion vector field corresponding to images of the second series. These motion vector fields are input for the means for calculating 204. Optionally motion vector fields are stored temporarily. The means for calculating 204 calculates a first intermediate image by motion compensated interpolation of input images of one of the input series. After having calculated a second intermediate image based on input images of the other series, the first and the second intermediate image are merged and provided at the output connector 208.

[0053] FIG. 3 schematically shows that input images 302-310 of two series are interpolated in the temporal domain and merged into one series of output images 316-324. The first series of input images comprises the images 308,310 that are generated at T2 respectively T5. In the lower left corner of image 308 an arrow 312 is present with a vertical direction. In the lower left corner of image 310 an arrow 314 is present with a horizontal. The two series of images are provided to a conversion unit 207 as described in FIG. 2C. The output of the conversion unit 207 is a series of motion compensated interpolated output images 316-324. In each of these latter images an arrow 326-334 is present which depicts some states between vertical and horizontal. Because of the separate interpolations of images of the first input series and of images of the second input series, no visible artifacts are introduced in the direct neighborhood of the pixels corresponding to the arrows 326-334 as present in the output images 316-324.

[0054] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be constructed as limiting the claim. The word ‘comprising’ does not exclude the presence of elements or steps not listed in a claim. The word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitable programmed computer. In the unit claims enumerating several means, several of these means can be embodied by one and the same item of hardware.

Claims

1. An image processing apparatus (100) comprising: a processor (108) based graphics pipeline (101) for generating a first series of input images (110-114) comprising successively a first input image (110) and a second input image (112); a conversion unit (105) comprising: a means for receiving (102) the first series of input images (110-114); and a means for calculating (104) a series of output images (116-126) based on the first series of input images, with an output image (118) being spaced a predetermined time interval (132) from a predecessor (116) if any; and a display device (106) for displaying the series of output images (116-126), characterized in that the conversion unit (100) is designed to determine a first time interval (128) between a first time of generation (T1) of the first input image (110) and a second time of generation (T2) of the second input image (112) and to calculate a particular output image (122) by interpolating between the first input image (110) and the second input image (112) on the basis of the first time interval (128).

2. An image processing apparatus (103) as claimed in claim 1, characterized in that the conversion unit (107) further comprises a motion estimator (109) for controlling the means for calculating (104) the series of output images (116-126), in order to perform motion compensated interpolation.

3. An image processing apparatus as (111) claimed in claim 2, characterized in that the conversion unit (113) further comprises a second means for receiving (115) a second series of input images (302-306) and that the conversion unit (113) is designed to calculate the series of output images (316-324) based on images of the first series of input images (308,310) and the second series of input images (302-306).

4. An image processing apparatus as (111) claimed in claim 3, characterized in that the conversion unit (113) is designed: to calculate a first intermediate motion compensated interpolated image by interpolating between the first input image (110) and the second input image (112) of the first series of input images; to calculate a second intermediate motion compensated interpolated image by interpolating between images of the second series of input images; and to merge the first intermediate image with the second intermediate image in order to create a combined output image.

5. A conversion unit (200) comprising: a means for receiving (202) a first series of input images comprising successively a first input image (110) and a second input image (122); and a means for calculating (104) a series of output images (116-126) based on the first series of input images, with an output image (118) being spaced a predetermined time interval (132) from a predecessor (116), if any, characterized in being designed to determine a first time interval (128) between a first time of generation (T1) of the first input image (110) and a second time of generation (T2) of the second input image (112) and to calculate a particular output image (122) by interpolating between the first input image (110) and the second input image (112) on the basis of the first time interval (128).

6. A conversion unit (203) as claimed in claim 5, characterized in further comprising a motion estimator (205) for controlling the means for calculating (204) the series of output images (116-126), in order to perform motion compensated interpolation.

7. A conversion unit (207) as claimed in claim 5, characterized in further comprising a second means for receiving (209) a second series of input images (308,310) and that the conversion unit (207) is designed to calculate the series of output images (316-326) based on images of the first series of input images (308,310) and the second series of input images (302,306).

8. A conversion unit (207) as claimed in claim 6, characterized in further comprising a second means for receiving (209) a second series of input images (308,310) and that the conversion unit (207) is designed to calculate the series of output images (316-326) based on images of the first series of input images (308,310) and the second series of input images (302,306).

9. A conversion unit (207) as claimed in claim 7, characterized in being designed: to calculate a first intermediate motion compensated interpolated image by interpolating between the first input image (110) and the second input image (112) of the first series of input images; to calculate a second intermediate motion compensated interpolated image by interpolating between images of the second series of input images; and to merge the first intermediate image with the second intermediate image in order to create a combined output image.

10. A conversion method comprising: a step of receiving a first series of input images comprising successively a first input image (110) and a second input image (122); and a step of calculating a series of output images based on the first series of input images, with an output image (118) being spaced a predetermined time interval (132) from a predecessor (116), if any, characterized in that the method comprises a timing step to determine a first time interval (128) between a first time of generation (T1) of the first input image (110) and a second time of generation (T2) of the second input image (112) and that in the step of calculating the series of output images (116-126) a particular output images is calculated by interpolating between the first input image (110) and the second input image (112) on the basis of the first time interval (128).