The present invention relates to an image processor generating a three-dimensional image, an image display apparatus including the image processor, and an image taking apparatus including the image processor.
A device is proposed that takes images of the same object (hereinafter simply referred to as an identical object) at the same time with two cameras for left and right eyes arranged side-by-side so as to generate a three-dimensional image (also referred to as a stereoscopic image) through the execution of various image processes to two image data (hereinafter, simply referred to as images) generated by the two cameras (see Patent Document 1).
A gap between images generated by overlapping the two images is referred to as a disparity. This disparity varies depending on a distance from cameras to an object.
For example, it is assumed that two cameras are disposed on a straight line with the optical axes of the cameras arranged in parallel to take images of an identical object. In this case, if the object is distant, the disparity of the object is substantially zero. As a distance between the cameras and the object is reduced, the disparity of the object increases. Therefore, if a three-dimensional image is generated and output for display based on two images including an object with a larger disparity, a projection amount of the object increases (appears closer) and a three-dimensional effect is enhanced.
However, if the disparity is increased greater than a certain level, an image sensed by the left eye is not fused with an image sensed by the right eye and a double image appears and makes the stereoscopic viewing impossible (fusional limitation). A double image due to an excessive disparity is known to cause visual fatigue. Therefore, to comfortably view a three-dimensional image, a depth (projection, depression) range of an object must be limited when the image is displayed. This limitation is disclosed in Non-Patent Literature 1.
For example, Patent Document 1 discloses a technique of generating a three-dimensional image without generating a double image due to an excessive disparity. In the technique disclosed in Patent Document 1, a depth range is nonlinearly controlled such that a three-dimensional image is displayed within a three-dimensionally viewable range of an observer, so as to generate an image easily three-dimensionally viewed by the observer.
If a main object desired to have an enhanced three-dimensional effect has no disparity close to the middle between the maximum and minimum photographable disparities, it is problematic that the three-dimensional effect of the main object is not sufficiently acquired. This problem will specifically be described with reference to
As described above, various image processes are executed for images acquired by shooting the main objects 1600, 1601, and the background 1602 by two cameras disposed on a straight line to generate a three-dimensional image, which is output for display on a three-dimensional image display apparatus (hereinafter, simply referred to as a display apparatus).
Since a distance from the cameras is increased in the order of the main objects 1600, 1601, and the background 1602, when γ denotes a disparity of the main object 1600 and β denotes a disparity of the main object 1601, γ>β is satisfied. It is assumed that the disparity γ of the main object 1600 and the disparity β of the main object 1601 are not disparities close to the middle between the maximum and minimum photographable disparities. The background 1602 is located furthest from the cameras and, therefore, the background 1602 has the smallest disparity α.
The main object 1600 is perceived in the vicinity of a position 1700; the main object 1601 is perceived in the vicinity of a position 1701; and the background 1602 is perceived in the vicinity of a position 1702.
When the three-dimensional image of the main objects 1600, 1601, and the background 1602 depicted in
Particularly when a disparity of a main object is substantially identical to a disparity of a background or a foreground, the three-dimensional effect of the main object is significantly deteriorated.
It is an object of the present invention to solve the problem and to provide an image processor generating a three-dimensional image easily three-dimensionally viewed by, and hardly causing fatigue of, an observer, and easily adjusting a three-dimensional effect of an arbitrary portion in the three-dimensional image.
A first technical means of the present invention is an image processor using a captured image from one view point and a disparity image corresponding to the captured image as inputs to generate a three-dimensional image based on the captured image and the disparity image, comprising: a disparity correction portion for correcting a disparity of the disparity image within a predetermined range; and a main object determination portion for determining at least one main object of the captured image, wherein the image processor converts a disparity of the disparity image such that a disparity corresponding to an image of at least one main object determined by the main object determination portion reaches to a predetermined value, and generates a three dimensional image based on a disparity image depending on the converted disparity and the captured image.
A second technical means of the present invention is an image processor using two captured images acquired by shooting an identical object from two viewing positions as inputs to generate a three-dimensional image based on the captured images, comprising: a disparity calculation portion for calculating a disparity from the two images; a disparity correction portion for correcting a disparity calculated by the disparity calculation portion within a predetermined range; and a main object determination portion for determining at least one main object of the captured image, wherein the image processor converts a disparity calculated by the disparity calculation portion such that a disparity corresponding to at least one main object determined by the main object determination portion reaches a predetermined value, and generates a three dimensional image based on the converted disparity and the captured image.
A third technical means is the image processor of the first technical means, wherein the disparity of the disparity image is converted into the predetermined value based on a linear function prescribing a relationship between an input disparity and an output disparity, and a graph showing the relationship between the input disparity and the output disparity is made by connecting a plurality of line segments having different slopes.
A fourth technical means is the image processor of the first technical means, wherein the predetermined value is zero or a value within a predetermined range based on zero.
A fifth technical means is the image processor of the first technical means, comprising a blurring process portion for executing a blurring process for a pixel value of the captured image corresponding to an object having a disparity equal to or less than a first predetermined value or equal to or greater than a second predetermined value based on the disparity image and the captured image.
A sixth technical means is the image processor of the fifth technical means, wherein the blurring process portion increases a blurring amount corresponding to a pixel value of a main object in proportion to an absolute value of disparity of the main object in the captured image.
A seventh technical means is an image display apparatus comprising: the image processor of the first technical means.
An eighth technical means is an image taking apparatus comprising: the image processor of the first technical means.
The image processor of the present invention generates a three-dimensional image easily three-dimensionally viewed by, and hardly causing fatigue of, an observer, and can easily adjust a three-dimensional effect of an arbitrary portion in the three-dimensional image through disparity conversion, and can easily enhance a three-dimensional effect and change a sense of depth of a main object.
(First Embodiment)
The present invention will now be described in detail with reference to the drawings. Configurations in the drawings are exaggeratingly depicted to facilitate understanding and have intervals and sizes different from actual configurations.
The image processor 100 is made up of a disparity correction portion 101, a main object determination portion 102, a disparity conversion portion 103, and an image generation portion 104 and uses a captured image from one view point and a disparity image corresponding to the captured image as inputs to generate a three-dimensional image based on the captured image and the disparity image.
A disparity image input from an external device (not depicted) etc., is input to the disparity correction portion 101. A captured image input from an external device etc., is input to the main object determination portion 102 and the image generation portion 104.
The disparity correction portion 101 corrects a disparity of the input disparity image within a predetermined range and outputs a disparity image storing the corrected disparity to the disparity conversion portion 103. The disparity image will be described later.
The main object determination portion 102 determines a main object of the input captured image and outputs the positional information of the main object to the disparity conversion portion 103.
The disparity conversion portion 103 extracts the disparity (corrected) corresponding to the image of the main object from the disparity image input from the disparity correction portion 101 based on the positional information of the main object input from the main object determination portion 102. The disparity conversion portion 103 converts the extracted disparity based on a predetermined conversion equation and outputs a disparity image storing the converted disparity to the image generation portion 104. Specifically, the disparity conversion portion 103 converts the disparity of the disparity image such that a disparity corresponding to the image of the main object determined by the main object determination portion 102 reaches a predetermined value. The predetermined value is a value causing the main object to be displayed on or near a display screen when the main object is displayed on a three-dimensional display apparatus. Specifically, the predetermined value is, for example, zero, or a value within a predetermined range based on zero. The predetermined value is adjustable.
As a result, the disparity is converted such that the three-dimensional effect of the main object is enhanced.
The image generation portion 104 generates an image for the left eye (left image) and an image for the right eye (right image) based on the input disparity image and the input captured image and outputs the images to a display apparatus (not depicted).
Therefore, the image generation portion 104 generates a three-dimensional image based on the disparity image using the disparity converted by the disparity conversion portion 103, and the captured image.
The details of the functional blocks will hereinafter be described in detail. First, the disparity image input to the disparity correction portion 101 will be described with reference to
As depicted in
The disparity image means an image storing a disparity indicative of a gap of an identical object in two captured images of the identical object taken from two viewing positions correspondingly to the pixels of the object for each pixel. In the example described above, the disparity of the object is stored correspondingly to the object of the left image 204 for each pixel of the disparity image.
Another pixel 206b of the disparity image 206 also stores a disparity of the object of the left image 204 corresponding to the pixel 206b.
The disparity may be on the basis of a pixel or a distance.
The disparity correction portion 101 corrects a disparity of a disparity image so as to suppress an excessive disparity. The correction of disparity can be performed in a well-known method described later.
Disparity correcting methods include two representative methods. One method is to normalize a disparity and the other method is to clip a disparity. The normalizing of disparity is characterized in that an overall depth relationship is preserved without saturating the disparity and the clipping of disparity is characterized in that a depth amount of an object having a disparity within a three-dimensionally viewable range is retained.
The horizontal axes indicate an input disparity Din and the vertical axes indicate an output disparity Dout. The input disparity means a disparity of a disparity image input to the disparity correction portion 101 (uncorrected disparity) and the output disparity means a disparity of a disparity image output from the disparity correction portion 101 (corrected disparity). In the equations, din corresponds to the input disparity Din, and dout in the equations corresponds to the output disparity Dout.
Din
Dout
A line 300 represents a relationship between the input disparity and the output disparity before executing the disparity correction and the input disparity and the output disparity have the same value before executing the disparity correction. A line 301 represents a relationship between the input disparity Din before the disparity correction and the output disparity Dout after the disparity correction.
In the method of
In the method of
The range of the three-dimensionally viewable binocular disparity depends on a screen size of a display apparatus and a viewing distance from an observer and Non-Patent Literature 1 describes that the range of the three-dimensionally viewable binocular disparity (fusional limitation) is up to about two degrees while a range of the binocular disparity allowing comfortable viewing of a three-dimensional image is equal to or less than one degree.
When Dout
A disparity image storing the disparity corrected by the disparity correction portion 101 is output to the disparity conversion portion 103.
The main object determination portion 102 determines a main object from objects of an input captured image and outputs positional information of the main object, for example, coordinate information indicative of a pixel position of the main object, to the disparity conversion portion 103. The determination of the main object can be implemented by a well-known method. For example, the method may be a method of selecting an object in focus from a captured image to determine the object as the main object, a method of executing a pattern recognition process to a captured image to determine an extracted object as the main object, or a method in which a user determines the main object by himself/herself. Alternatively, a facial recognition technique may be used for determining a certain person as the main object.
The disparity conversion portion 103 performs the conversion such that a disparity corresponding to an image of the main object determined by the main object determination portion 102 reaches a predetermined value. Specifically, the disparity is converted such that the three-dimensional effect of the object is enhanced. Assuming that the object 201 is the main object in the example of
Description will be made of how objects are perceived by an observer when the objects are displayed with disparities set to zero and increased/decreased to positive/negative values on a display apparatus with reference to
Specifically, when an object is displayed on a display apparatus, the object appears more projected when the disparity of the object is increased to a positive value while the object appears more depressed when the disparity is increased to a negative value. Since an object with zero disparity has no gap between left and right images, when the object is displayed on the display apparatus, the perceived position is on a display screen, making no difference from a depth position perceived two-dimensionally. Therefore, even if the image with zero disparity is displayed on the display apparatus, an observer does not perceive the presence of a depth and perceives the image as being two-dimensionally displayed at the position same as the actual display position. Displaying the object with zero disparity on a display apparatus means that an object is displayed on a display screen. The disparity of zero or a value within a predetermined range based on zero corresponds to the predetermined value.
It is assumed that Din (input disparity) is a disparity of a front side surface of an object A depicted in
If Dout is set to a positive value, the object A is perceived as being projected on the front side of the image display surface as indicated by a position 601. If Dout is set to a negative value, the object A is perceived as being depressed on the rear side of the image display surface as indicated by a position 602.
The conversion of the disparity can be executed with disparity conversion equations depicted in
The disparity conversion equation of
The disparity conversion equation of
In this case, the input disparity Din denotes a disparity before the disparity conversion, and Din
The output disparity Dout denotes a disparity after the disparity conversion, and Dout
A line 302 represents a relationship between the input disparity Din before the disparity conversion and the output disparity Dout after the disparity conversion. Din0 and Dout0 may be set to any values.
In the equations, din corresponds to the input disparity Din before the disparity conversion and dout in the equations corresponds to the output disparity Dout after the disparity conversion.
The graphs of
In the graph of
While the object A described with reference to
While the object A described with reference to
As depicted in
Description will hereinafter be made of how the main object is perceived by an observer when the disparity of the main object depicted in
A disparity γ of a main object 1600 of
By increasing the disparities, the main object 1600 is perceived in the vicinity of a position 1000 and the main object 1601 is perceived in the vicinity of a position 1001. In this way, depth amounts can be enlarged to enhance the three-dimensional effect.
As compared to
Such conversion of a disparity of a main object can enhances the three-dimensional effect of the main object and a surrounding portion of the main object. If the main object has a thickness, an average value of the maximum and minimum disparities of the main object is defined as the disparity of the main object. If a plurality of main objects exists, average values of the maximum and minimum disparities of a plurality of the main objects are defined as the disparities of the main objects. This is preferred since the disparities (display positions) of the main objects can be converted inconsideration of the thicknesses of the main objects. It is preferred to calculate an average of disparity in consideration of an area of a region since a display position can be converted in consideration of the center of gravity of a main object.
As described above, the disparity conversion portion 103 extracts a disparity (after correction) for each pixel of a main object determined by the main object determination portion 102 from the disparity image input from the disparity correction portion 101, converts the extracted disparity of each pixel based on the predetermined conversion equation depicted in
The image generation portion 104 generates a three-dimensional image based on a captured image input from an external apparatus etc., and the disparity image input from the disparity conversion portion 103.
A three-dimensional image generating method will hereinafter be described. A captured image, a disparity image, and an output image are denoted by I, D, and O, respectively. A pixel value at coordinates (x, y) in the images is represented by I (x, y) and D (x, y). An image horizontally moved by a disparity from the captured image I to the output image O is created by using (Eq. 11).
At this point, if pixels overlap with each other, a pixel with a larger disparity (an object located at a position closer to a camera) is used and, if a pixel is not filled, the pixel is interpolated by using upper, lower, left, and right pixels. The captured image I and the output image O are output as the left image and the right image, respectively.
With the configuration described above, even if a value of disparity between the main object and the foreground or background is small, the disparity can be converted to enlarge the difference of disparity between the main object and the foreground or background.
As a result, a depth amount between the main object and the foreground or background can be converted to generate an image with the three-dimensional effect enhanced within a three-dimensionally viewable range for an observer.
Although linear conversion is used for the correction and conversion of disparity in the disparity correction portion 101 and the disparity conversion portion 103 in this embodiment, nonlinear conversion can be used to acquire the same effect.
Although one set of input and output disparities Din0 and Dout0 is used in this embodiment, two or more sets can be used for converting the disparity to acquire the same effect.
Second Embodiment
A second embodiment of the present invention will hereinafter be described in detail with reference to the drawings. However, the portions having the same functions as the first embodiment are denoted by the same reference numerals.
The image processor 1100 is formed by adding a disparity calculation portion 1101 to the image processor 100 described with reference to
The two captured images acquired by shooting an identical object from at least two viewing positions are input to the disparity calculation portion 1101. Any one of the two captured images is input to the main object determination portion 102 and the image generation portion 104. One captured image of the two captured images is a captured image shot by a camera on the left side facing the object and this image is referred to as a left image. The other captured image is a captured image shot by a camera on the right side facing the object and this image is referred to as a right image.
The left and right images are input to the disparity calculation portion 1101 and the left image is input to the main object determination portion 102 and the image generation portion 104.
The disparity calculation portion 1101 calculates a disparity based on the left and right images. The disparity calculation portion 1101 generates a disparity image described in the first embodiment based on the calculated disparity. The disparity can be calculated by using a well-known technique and, for example, a block matching method can be used. The block matching method is a method of calculating a disparity by checking the left and right images against each other on the basis of an appropriately determined block to define the most similar blocks between images as corresponding blocks.
The disparity calculation portion 1101 outputs the generated disparity image to the disparity correction portion 101.
The disparity correction portion 101 corrects a disparity of the disparity image input from the disparity calculation portion 1101 within a predetermined range. In other words, the disparity calculated by the disparity correction portion 101 is corrected within a predetermined range.
At the time of this correction, the disparity can be adjusted such that an interval between the camera taking the left image and the camera taking the right image (a base-line length) is exceeded. When a three-dimensional image is taken, the base-line length is often set to 65 mm. This is because a width between human eyes is about 65 mm.
However, in some cases, an image must be taken with a base-line length equal to or less than 65 mm because of a relationship of arrangement positions of cameras. When the same object is shot in the same composition, the disparity is reduced as the base-line length decreases and, therefore, a sense of depth is reduced. The disparity correction portion 101 prevents the reduction of the sense of depth by performing the correction such that the minimum value and the maximum value of the input disparity are expanded. Therefore, an image taken with an arbitrary base-line length can be generated from an image taken with a fixed base-line length.
When the disparity is corrected by using (Eq. 1), Din
The disparity correction portion 101 outputs a disparity image storing a corrected disparity to the disparity conversion portion 103.
As described above, the disparity conversion portion 103 converts the disparity of the main object through the disparity conversion depicted in
Specifically, the disparity conversion portion 103 converts the disparity calculated by the disparity calculation portion 1101 (disparity of the disparity image created by the disparity calculation portion 1101) such that a disparity corresponding to the main object determined by the main object determination portion 102 reaches a predetermined value. The predetermined value is, for example, zero, or a value within a predetermined range based on zero. The predetermined value is adjustable.
The image generation portion 104 generates a three-dimensional image based on the left image input from an external apparatus etc., and the disparity image input from the disparity conversion portion 103. The generation of the three-dimensional image has been described in the first embodiment and therefore will not be described.
With the configuration described above, an image with the three-dimensional effect enhanced within a three-dimensionally viewable range for an observer can easily be generated. The disparity of a disparity image generated based on left and right images taken with a shorter base-line length can be corrected to generate an image as if the image is taken with a longer base-line length. This is particularly effective when a three-dimensional image is generated based on the left and right images taken in a situation where a base-line length cannot be made longer, for example, the left and right images of endoscopic cameras or cameras built into a small device, and the three-dimensional effect of an object is enhanced.
(Third Embodiment)
A third embodiment of the present invention will hereinafter be described in detail with reference to the drawings. However, the portions having the same functions as the first embodiment are denoted by the same reference numerals.
The image processor 1400 of
A disparity image input from an external apparatus etc., is input to the disparity correction portion 101 and the blurring process portion 1401. A captured image input from an external apparatus etc., is input to the main object determination portion 102 and the blurring process portion 1401.
The blurring process portion 1401 executes a blurring process for a pixel value of the captured image corresponding to an object having a disparity equal to or less than a first predetermined value or equal to or greater than a second predetermined value based on the input disparity and captured images. The blurring process portion 1401 outputs the captured image subjected to the blurring process to the image generation portion 104. The first predetermined value<the second predetermined value is satisfied. In this case, the blurring process portion 1401 increases a blurring amount corresponding to a pixel value of a main object in proportion to an absolute value of disparity of the main object in the captured image.
In the example of
The first predetermined value and the second predetermined value will be described with reference to
If the object to be blurred by the blurring process portion 1401 is an object having a disparity equal to or less than the first predetermined value (in the range indicated by reference numeral 1500), i.e., an object having a disparity equal to or less than Dout
This process generates an image with blur in the area perceived as being depressed as compared to the range of the three-dimensionally viewable disparity. The image subjected to this blurring process is displayed as if a depth of field is reduced from that of the actual shooting. In an image with a depth of field reduced, a sense of depth can easily be sensed because a background is blurred depending on a distance between an object and a camera.
If the object to be blurred by the blurring process portion 1401 is an object having a disparity equal to or greater than the second predetermined value (in the range indicated by reference numeral 1501), i.e., an object having a disparity equal to or less greater Dout
This processing can blur a range which is difficult for three-dimensional viewing and, therefore, an easily viewable three-dimensional image can be acquired. If an object having a disparity equal to or less than the first predetermined value and an object having a disparity equal to or greater than the second predetermined value are blurred at the same time, a three-dimensional image can be acquired with a three-dimensionally viewable range having higher visual attractiveness.
The blurring process can be implemented with a well-known method. Representative methods of the blurring process include a smoothing filter, a Gaussian filter, etc. The smoothing filter is a technique of averaging pixel values by using pixel values around a pixel of interest to acquire a pixel value of a processed image. For a pixel of interest of the object to be blurred, eight neighboring pixel values in 3×3 pixels can be used for the averaging, or 24 neighboring pixel values in 5×5 pixels can be used for the averaging. As the number of peripheral pixel values used for the averaging is increased, a blurring amount of a processed pixel value becomes greater.
Therefore, if an object having a disparity equal to or less than the first predetermined value (equal to or less than Dout
Therefore, the blurring process portion 1401 increases a blurring amount corresponding to a pixel value of a main object in proportion to an absolute value of disparity of the main object in the captured image.
By executing this blurring process, a depth relationship can be retained even if a disparity included in a range difficult for three-dimensional viewing is clipped. Since the blurring process is executed depending on a disparity of each pixel of the captured image, a level of blur of each pixel is varied depending on a disparity and the same blurring effect as an image taken by a camera is acquired rather than uniform blur.
The blurring process portion 1401 outputs the captured image after the execution of the blurring process to the image generation portion 104.
The disparity correction portion 101 corrects a disparity with the clipping method as described with referenced to
The image generation portion 104 generates a three-dimensional image based on the captured image input from the blurring process portion 1401 and the disparity image input from the disparity conversion portion 103.
By executing the disparity conversion with the configuration described above, an image with the three-dimensional effect enhanced within a three-dimensionally viewable range for an observer can easily be generated. By executing the blurring process for a captured image based on a disparity image before executing the disparity correction, a blurring amount does not become constant even in a range with a saturated disparity and a three-dimensional image can be generated with an enhanced three-dimensional effect.
The image processor of the present invention can easily generate a three-dimensional video with a three-dimensional (depth) effect enhanced between an object and a background (or foreground) within a three-dimensionally viewable range.
The embodiments described above are also applicable to an integrated circuit/chipset mounted on the image processor.
The image processor of the present invention is applicable to an image display apparatus capable of displaying a three-dimensional image. An image can be displayed with an enhanced three-dimensional effect by including the image processor of the present invention. The image processor of the present invention is applicable to various image display apparatuses having display surfaces different in size and resolution.
The image processor of the present invention is also applicable to an image taking apparatus capable of taking a three-dimensional image and this is preferred since an image can be taken while a result of taking a three-dimensional image is previewed.
Explanations of Letters or Numerals
100, 1100, 1400 . . . image processor; 101 . . . disparity correction portion; 102 . . . main object determination portion; 103 . . . disparity conversion portion; 104 . . . image generation portion; 1101 . . . disparity calculation portion; 1401 . . . blurring process portion; 201 . . . object; 202, 203 . . . camera; 204 . . . left image; 205 . . . right image; 206 . . . disparity image; and 206a, 206b . . . pixel.
Number | Date | Country | Kind |
---|---|---|---|
2010-120216 | May 2010 | JP | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/JP2011/061821 | 5/24/2011 | WO | 00 | 11/7/2012 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2011/148921 | 12/1/2011 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
5748199 | Palm | May 1998 | A |
6163336 | Richards | Dec 2000 | A |
8320641 | Steinberg et al. | Nov 2012 | B2 |
8405708 | Robinson et al. | Mar 2013 | B2 |
Number | Date | Country |
---|---|---|
6-194602 | Jul 1994 | JP |
10-191397 | Jul 1998 | JP |
2003-209858 | Jul 2003 | JP |
2004-221700 | Aug 2004 | JP |
2005-91508 | Apr 2005 | JP |
Entry |
---|
“3DC Safety Guidelines for Dissemination of Human-friendly 3D”, Safety/Guidelines Section, revised on Apr. 20, 2010, pp. 1-41. |
Number | Date | Country | |
---|---|---|---|
20130051660 A1 | Feb 2013 | US |