Patent Application
20090073170
The invention relates to a method of computing an output disparity map, comprising output elements having output values corresponding to shifts to be applied to respective pixels of a first image to compute a second image, the computing on basis of an input disparity map comprising respective input elements having input values.
The invention further relates to a unit for computing an output disparity map, comprising output elements having output values corresponding to shifts to be applied to respective pixels of a first image to compute a second image, the computing on basis of an input disparity map comprising respective input elements having input values.
The invention further relates to an image processing apparatus comprising such a unit for computing an output disparity map.
The invention further relates to a computer program product to be loaded by a computer arrangement, comprising instructions to compute an output disparity map, comprising output elements having output values corresponding to shifts to be applied to respective pixels of a first image to compute a second image, the computing on basis of an input disparity map comprising respective input elements having input values.
Since the introduction of display devices, a realistic 3-D display device has been a dream for many. Many principles that should lead to such a display device have been investigated. Some principles try to create a realistic 3-D object in a certain volume. For instance, in the display device as disclosed in the article “Solid-state Multi-planar Volumetric Display”, by A. Sullivan in proceedings of SID'03, 1531-1533, 2003, visual data is displaced at an array of planes by means of a fast projector. Each plane is a switchable diffuser. If the number of planes is sufficiently high the human brain integrates the picture and observes a realistic 3-D object. This principles allows a viewer to look around the object within some extend. In this display device all objects are (semi-)transparent.
Many others try to create a 3-D display device based on binocular disparity only. In these systems the left and right eye of the viewer perceives another image and consequently, the viewer perceives a 3-D image. An overview off these concepts can be found in the book “Stereo Computer Graphics and Other True 3-D Technologies”, by D. F. McAllister (Ed.), Princeton University Press, 1993. A first principle uses shutter glasses in combination with for instance a CRT. If the odd frame is displayed, light is blocked for the left eye and if the even frame is displayed light is blocked for the right eye.
Display devices that show 3-D without the need for additional appliances are called auto-stereoscopic display devices.
A first glasses-free display device comprises a barrier to create cones of light aimed at the left and right eye of the viewer. The cones correspond for instance to the odd and even sub-pixel columns. By addressing these columns with the appropriate information, the viewer obtains different images in his left and right eye if he is positioned at the correct spot, and is able to perceive a 3-D picture.
A second glasses-free display device comprises an array of lenses to image the light of odd and even sub-pixel columns to the viewer's left and right eye.
The disadvantage of the above mentioned glasses-free display devices is that the viewer has to remain at a fixed position. To guide the viewer, indicators have been proposed to show the viewer that he is at the right position. See for instance U.S. Pat. No. 5,986,804 where a barrier plate is combined with a red and green led. In case the viewer is well positioned he sees a green light, and a red light otherwise.
To relieve the viewer of sitting at a fixed position, multi-view auto-stereoscopic display devices have been proposed. See for instance U.S. Pat. No. 60,064,424 and U.S. Pat. No. 20,000,912. In the display devices as disclosed in U.S. Pat. No. 60,064,424 and U.S. Pat. No. 20,000,912 a slanted lenticular is used, whereby the width of the lenticular is larger than two sub-pixels. In this way there are several images next to each other and the viewer has some freedom to move to the left and right.
In order to generate a 3D impression on a multi-view display device, images from different virtual view points have to be rendered. This requires either multiple input views or some 3D or depth information to be present. This depth information can be either recorded, generated from multiview camera systems or generated from conventional 2D video material. For generating depth information from 2D video several types of depth cues can be applied: such as structure from motion, focus information, geometric shapes and dynamic occlusion. The aim is to generate a dense depth map, i.e. per pixel a depth value. This depth map is subsequently used in rendering a multi-view image to give the viewer a depth impression. In the article “Synthesis of multi viewpoint images at non-intermediate positions” by P. A. Redert, E. A. Hendriks, and J. Biemond, in Proceedings of International Conference on Acoustics, Speech, and Signal Processing, Vol. IV, ISBN 0-8186-7919-0, pages 2749-2752, IEEE Computer Society, Los Alamitos, Calif., 1997 a method of extracting depth information and of rendering a multi-view image on basis of the input image and the depth map are disclosed. The multi-view image is a set of images, to be displayed by a multi-view display device to create a 3D impression. Typically, the images of the set are created on basis of an input image. Creating one of these images is done by shifting the pixels of the input image with respective amounts of shift. These amounts of shifts are called disparities. So, typically for each pixel there is a corresponding disparity value, together forming a disparity map. Disparity values and depth values are typically inversely related, i.e.:
with S being disparity, C a constant value and D being depth.
Suppose there is a image processing apparatus, e.g. a multi-view display device which is arranged to receive a signal representing a sequence of 2D images and corresponding depth maps and is arranged to convert that input into a sequence of multi-view images. Suppose that the multi-view display device is arranged to display 9 images simultaneously, i.e. each multi-view images comprises a set of 9 different images. The depth map is converted to a disparity map by a linear operation as specified in Equation 1 and applied to compute the multi-view images on basis of the capabilities of the multi-view display device and optionally on basis of user preferences. The conversion means that the range of depth values is mapped to a range of disparity values. For instance the depth values are in the range [50, 200] meter and the disparity values are in the range of [0,5] pixels. Typically this mapping corresponds to a linear scaling whereby the lowest value of the depth range is directly mapped to the lowest value of the disparity range and the highest value of the depth range is directly mapped to the highest value of the disparity map. Unfortunately this does not result in the best image quality of the multi-view image.
It is an object of the invention to provide method of the kind described in the opening paragraph resulting in a relatively high quality multi-view image when the output disparity map is used for computing the multi-view image, to be displayed on a multi-view display device.
This object of the invention is achieved in that the method comprises:
determining a particular input value on basis of the input disparity map on basis of a predetermined criterion;
determining a mapping function on basis of the particular input value, the mapping function such that the particular input value is mapped to a predetermined output value which is substantially equal to zero; and
mapping the input values of the input elements to respective output values of the output elements on basis of the mapping function.
Multiview display devices are typically driven in such a way that a portion of the scene seems to be located in front of the plane of the screen of the display device and an other portion to be located behind the plane of the screen. This results in an optimal use of the depth range of the display device. Due to crosstalk between the different images shown by the display device, objects that lie far from the plane of the screen, either in front or behind, appear blurry, because shifted versions of such parts of the scene are seen mixed. This effect is known as ghosting. However, objects that are located relatively close to the plane of the screen appear relatively sharp. Typically, important objects in a scene have to be visualized relatively sharp while less important objects may be visualized less sharp. The invention is based on this observation. That means that on basis of a predetermined criterion the input value of a particular input element is determined, whereby the predetermined criterion typically satisfies the condition that the particular input element corresponds to a relatively important object. The mapping of the input values of the input elements to respective output values of the output elements is such that the particular input element is mapped to a predetermined output value which is substantially equal to zero. The effect of this is that for rendering the different images of the multiview image no or substantially no pixel shifts have to be applied for the respective pixels corresponding to the particular input element. Applying no or substantially no pixel shifts results in being visible relatively close to the plane of the screen. As described above, that means that a relatively sharp representation is achieved. As a consequence, objects which have depth values, i.e. disparity values which differ relatively much from the value of the particular input element will be visualized more blurred.
In an embodiment of the method according to the invention, the mapping function corresponds to adding or subtracting a constant value from the input values of the input elements, the constant value based on computing a difference between the input value of the particular input element and the predetermined output value. Typically, the difference between the input value and the predetermined output value corresponds to an offset which is used as a constant value to be added or subtracted depending on the sign of the offset.
In an embodiment of the method according to the invention, the predetermined criterion is based on the first image. That means that on basis of analyzing the luminance and/or color values of the first image the particular input value is established.
In an embodiment of the method according to the invention whereby the predetermined criterion is based on the first image, the predetermined criterion is that the particular input value belongs to a particular input element corresponding to a particular pixel of the first image having a relatively high sharpness value compared to other sharpness values of respective other pixels of the first image. Preferably, the sharpness value of the particular pixel is determined by computing a difference of the luminance and/or color value of the particular pixel and the luminance and/or color value of a neighboring pixel of the particular pixel. Typical images as acquired by a camera are such that a portion of the scene which was in focus of the image acquisition optics is relatively sharp, while other parts of the scene are relatively unsharp, i.e. blurred. An advantage of this embodiment of the method according to the invention is that the portion of the scene which was in focus during acquisition of the image is mapped to the plane of the screen of the display device. Other portions of the scene which were out of focus, will appear to be in front or behind the plane of the screen of the display device.
In an embodiment of the method according to the invention, the predetermined criterion is that the particular input value belongs to a particular input element corresponding to a particular pixel of the first image which is located at a predetermined position. Typically, relatively important objects appear on predetermined positions in the image. E.g. because the person making the image likes such a composition. For instance, the predetermined position is close to the center of the first image. Think for instance about an image with one person in the middle. Alternatively, the predetermined position is close to the border of the first image. Think for instance about two persons talking to each other and located at respective borders of the image. An advantage of this embodiment of the method according to the invention is its simplicity. Notice that it is relatively easy to determine the input value of a particular input element on basis of this criterion, i.e. coordinates.
In an embodiment of the method according to the invention, the predetermined criterion is based on a motion vector field being computed on basis of the first image and a third image, the first image and the third image belonging to a sequence of temporarily succeeding images. Analysis of the motion vector field also provides information about relative importance of objects in the image. In the case of a stationary background a relatively fast moving object in the foreground is expected to be important. So, the predetermined criterion may be that the particular input value belongs to a particular input element corresponding to a particular pixel of the first image having a relatively large motion vector. Alternatively, the predetermined criterion is that the particular input element corresponds to a particular pixel of the first image having a relatively small motion vector. This corresponds to the case that the camera is panning in order to follow a moving object.
In an embodiment of the method according to the invention, the predetermined criterion is that the particular input value is equal to the value of the range of possible input values, which has a relatively high frequency of occurrence in a part of the input disparity map. An advantage of this embodiment according to the invention is that it is relatively robust. Typically, the spectrum of frequency of occurrence for temporally succeeding disparity maps is relatively constant. Consecutive multiview images will be rendered such that the same object remains on the same depth relative to the plane of the screen of the display device.
In an embodiment of the method according to the invention, the particular input value is based on a combination of two further input values, whereby the two further input values are determined on basis of the predetermined criterion as described above. It may be that there are multiple objects in the scene which are both relevant but which have a difference disparity. Selecting one of these objects in or to render it relatively sharp on the multiview display device may result in rendering the other object much to blurred. Going for a compromise, i.e. both objects moderately sharp, is advantageous in such a situation. Typically, the combination is a weighted average of the two further input values.
In an embodiment of the method according to the invention, the mapping function is based on a sharpness map, comprising elements of sharpness values which are based on difference between corresponding pixels of the first image and their respective neighboring pixels. Typically not only an offset, as described above, is needed to define the mapping function, but also a scaling factor. It is advantageous to limit the output range and thus the scaling factor on basis of the sharpness of the input image. On basis of the intrinsic sharpness of the first image and corresponding disparity values an optimal scaling factor is determined according to this embodiment of the method according to the invention.
It is a further object of the invention to provide unit of the kind described in the opening paragraph resulting in a relatively high quality multi-view image when the output disparity map is used for computing the multi-view image, to be displayed on a multi-view display device.
This object of the invention is achieved in that the unit comprises:
first determining means for determining a particular input value on basis of the input disparity map on basis of a predetermined criterion;
second determining means for determining a mapping function on basis of the particular input value, the mapping function such that the particular input value is mapped to a predetermined output value which is substantially equal to zero; and
mapping means for mapping the input values of the input elements to respective output values of the output elements on basis of the mapping function.
It is a further object of the invention to provide an image processing apparatus of the kind described in the opening paragraph resulting in a relatively high quality multi-view image when the output disparity map is used for computing the multi-view image, to be displayed on the image processing apparatus.
This object of the invention is achieved in that the image processing apparatus comprises the unit for computing an output disparity map, as claimed in claim 16.
It is a further object of the invention to provide a computer program product of the kind described in the opening paragraph resulting in a relatively high quality multi-view image when the output disparity map is used for computing the multi-view image, to be displayed on a multi-view display device.
This object of the invention is achieved in that the computer program product, after being loaded, provides said processing means with the capability to carry out:
determining a particular input value on basis of the input disparity map on basis of a predetermined criterion;
determining a mapping function on basis of the particular input value, the mapping function such that the particular input value is mapped to a predetermined output value which is substantially equal to zero; and
mapping the input values of the input elements to respective output values of the output elements on basis of the mapping function.
Modifications of the unit for computing an output disparity map and variations thereof may correspond to modifications and variations thereof of the image processing apparatus, the method and the computer program product, being described.
These and other aspects of the unit for computing an output disparity map, of the image processing apparatus, of the method and of the computer program product, according to the invention will become apparent from and will be elucidated with respect to the implementations and embodiments described hereinafter and with reference to the accompanying drawings, wherein:
Same reference numerals are used to denote similar parts throughout the figures.
The output disparity map is provided at the output connector 112. Optionally, a set of output disparity maps is provided, whereby the disparity maps of the set correspond to respective images of a multiview image.
The unit 100 for computing an output disparity map comprises:
first determining means 102 for determining the input value of a particular input element of the input disparity map on basis of a predetermined criterion;
second determining means 104 for determining a mapping function on basis of the input value of the particular input element, the mapping being function such that the input value of the particular input element is mapped to a predetermined output value which is substantially equal to zero; and
mapping means 106 for mapping the input values of the input elements to respective output values of the output elements on basis of the mapping function.
The first determining means 102, the second determining means 104 and the mapping means 106 may be implemented using one processor. Normally, these functions are performed under control of a software program product. During execution, normally the software program product is loaded into a memory, like a RAM, and executed from there. The program may be loaded from a background memory, like a ROM, hard disk, or magnetical and/or optical storage, or may be loaded via a network like Internet. Optionally an application specific integrated circuit provides the disclosed functionality.
Optionally, the unit 100 for computing an output disparity map comprises a second input connector 110 for providing additional information, e.g. video data or motion vector fields as disclosed in
The mapping function may be any kind of operation for mapping the input elements to respective output elements. Typically, the input elements have values in an input range and the output elements have values in an output range, whereby the input range and the output range are different.
first: the input value 206 of a particular element of the input disparity map is determined on basis of a predetermined criterion. In this case it appears that the input value 206 of the particular element equals 1.5.
second: a mapping function is determined on basis of the input value 206 of the particular input element. The mapping function is such that the input value 208 of the particular input element is mapped to a predetermined output value 208 which is substantially equal to zero. In this case the mapping function is relatively easy. For that an offset is computed by computing the difference between the input value 206 and the output value 208. The offset is 1.5.
third: the input values of the input elements are mapped to respective output values of the output elements on basis of the mapping function. That means that from each input element the offset being 1.5 is subtracted to compute the corresponding output element. The result is the first output disparity map.
Optionally, the first output disparity map 202 is scaled to a second output disparity map in order to map the range of values to a range 204 of values which are suitable for the display device or which corresponds to a user preference. It should be noted that the scaling should be such that no shift of the range takes place i.e. the particular value 210 of the second output disparity map corresponds to the output value 208 of the first output disparity map.
Preferably, the mapping function comprises a scaling which is based on a sharpness map, comprising elements of sharpness values which are based on differences between corresponding pixels of the first image and their respective neighboring pixels. If something is already unsharp then it is not a problem to give it a relatively large disparity, i.e. displaying it relatively remote from the plane of the screen of the display device. The additional unsharpness because of the relatively large disparity is hardly visible. However, objects which are relatively sharp should be rendered with a relatively small amount of disparity to prevent blurring.
To determine a scaling factor G which can be applied to scale an input disparity map into an output disparity map the following strategy is preferred. Suppose there is a sharpness map comprising respective sharpness values for the pixels of an image and there is an input disparity map comprising respective disparity values for the pixels of the image. The sharpness value for a pixel with coordinates (x,y) is indicated with E(x,y) and the input disparity value for a pixel with coordinates (x,y) is indicated with Sin(x,y). Assume that for each sharpness value E(i)) there is an allowable maximum disparity value Smax(i). That means that there is a function mapping a sharpness value to a maximum disparity value.
f(E(i))=Smax(i) (2)
Typically the function is such that a relatively low sharpness value is mapped to a relatively high maximum disparity value. The function may be a linear function, e.g. as specified in equation 3.
f(E(i))=C1E(i)+C2
The relation between the allowable maximum disparity and the input disparity of a particular pixel is determined by the maximum allowable scale factor g(x,y) for the particular pixel.
In order to determine a scaling factor G for the total input disparity map, one has to determine the minimum of the maximum allowable scale factors g(x,y) of the input disparity map. Notice that a particular scale factor which is larger than the maximum scale factor for the particular pixel results in an output disparity value which is higher than the allowable maximum disparity value for that particular pixel. So, to find the scaling factor G for the input disparity map the minimum g(x,y) has to be established.
G=min(x,y)(g(x,y)) (4)
The first embodiment of the unit 100 for computing an output disparity map is arranged to determine the mapping function on basis of the input disparity map. Preferably this is done by making a histogram of disparity values, i.e. counting the frequency of occurrence of the different disparity values. A next step is determining a particular value with a relatively high frequency of occurrence. This particular value should be mapped to the predetermined output value which is substantially equal to zero. With substantially equal to zero is meant the range of values corresponding to no or a limited shift to be applied for rendering such that the selected region of the images appears to be in the plane of the screen of the display device.
Instead of determining a particular value with a relatively high frequency of occurrence a limited set of such values could be selected, e.g. two or three values. Subsequently an average of this set of values is computed and used as the value to be mapped to the predetermined value, i.e. zero. Optionally, the respective determined frequency of occurrences are used for weighting the values in order to compute a weighted average.
Alternative to or in combination with determining a histogram of values, the first embodiment of the unit 100 for computing an output disparity map is arranged to determine the particular value to be mapped to the predetermined value on basis of an other predetermined criterion. The criterion may be selecting the particular value on basis of its coordinates, e.g. being located in the center or at the border of the image and thus input in the center or at the border disparity map, respectively.
In order to avoid certain changes in the consecutive disparity maps and resulting forward and backward movement of the scene, the consecutive input or output disparity maps are low-passed filtered, for example which an IIR filter with a half time of several video frames.
Alternatively, the sequence of images corresponds to a static background in front of which an object moves. In that case a relatively high motion vector corresponds to a moving object. Then, a particular element of the input disparity map corresponding to one of the pixels for which such a relatively high motion vector has been determined is selected as the particular element of which the value has to be mapped to the predetermined output value.
receiving means 602 for receiving a signal representing video images;
a combination of processing units 604, comprising the unit 100 for computing an output disparity map and the rendering unit 302 as disclosed in any of the
The signal may be a broadcast signal received via an antenna or cable but may also be a signal from a storage device like a VCR (Video Cassette Recorder) or Digital Versatile Disk (DVD). The signal is provided at the input connector 610. The image processing apparatus 600 might e.g. be a TV. Alternatively the image processing apparatus 600 does not comprise the optional display device but provides the output images to an apparatus that does comprise a display device 606. Then the image processing apparatus 600 might be e.g. a set top box, a satellite-tuner, a VCR player, a DVD player or recorder. Optionally the image processing apparatus 600 comprises storage means, like a hard-disk or means for storage on removable media, e.g. optical disks. The image processing apparatus 600 might also be a system being applied by a film-studio or broadcaster.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be constructed as limiting the claim. The word ‘comprising’ does not exclude the presence of elements or steps not listed in a claim. The word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitable programmed computer. In the unit claims enumerating several means, several of these means can be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words are to be interpreted as names.
| Number | Date | Country | Kind |
|---|---|---|---|
| 04105305.9 | Oct 2004 | EP | regional |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/IB05/53452 | 10/21/2005 | WO | 00 | 4/23/2007 |
