Patent Application
20120314061
The present invention relates to an imaging apparatus for measuring a depth of a scene using a plurality of images captured from a single viewpoint.
Conventionally, various methods have been suggested for measuring, without contact, a depth of a three-dimensional scene, that is, a distance from the imaging apparatus to each object (hereinafter, referred to as an “object distance”). Such methods can be largely classified into an active technique and a passive technique. The active technique is to irradiate the object with infrared rays, ultrasound, or laser, and calculate the object distance based on a length of time until a wave which is reflected returns or an angle of the reflected wave. The passive technique is to calculate the object distance based on an image of the object. In particular, in the case of using an imaging apparatus such as a camera to measure the object distance, the passive technique which does not require an apparatus for emitting infrared rays and so on is widely used.
Various passive techniques have been suggested, one of which is referred to as Depth from Defocus (hereinafter, referred to as “DFD”). The DFD is a distance measuring technique based on a blur changing its size and form depending on the object distance. The DFD has features such as not requiring a plurality of cameras, allowing distance measurement using a small number of images, and so on.
Hereinafter, a principle of the DFD is briefly described.
A captured image including a blur (hereinafter, referred to as a “blurred image”) is an image obtained by convoluting a Point Spread Function (PSF), which is a function of the object distance, into an all-in-focus image which represents a state without a lens-derived blur. Since the PSF is the function of the object distance, the object distance can be determined with the DFD by detecting the blur in the blurred image. However, at this time, the all-in-focus image and the object distance are unknown. One mathematical expression concerning the blurred image, the all-in-focus image, and the object distance is established for a single blurred image, and therefore a new mathematical expression is obtained when a new blurred image having a different focal position is captured. That is, a plurality of the above expressions are obtained corresponding to the respective blurred images each having a different focal position. The object distance is calculated by solving the expressions obtained as described above. Various methods for obtaining and solving the expressions have been suggested for the DFD, for example, Patent Literature (PL) 1 and Non-Patent Literature (NPL) 1.
The DFD is a technique to determine the object distance by using the PSF for the blur included in the blurred image. However, in the DFD, the forms of the PSFs are similar between positions in front of and behind the image point corresponding to the object distance. Therefore, the DFD has a problem in that it is ambiguous whether a PSF is one that is behind the image point or one that is in front of the image point, due to the influence of a noise included in the image, thereby making the distinction between those PSFs difficult.
To solve this problem, for example, increasing the number of images each having a different focal position can improve the accuracy rate of the estimation of the object distance. In addition, as in NPL 1, using apertures having a non point-symmetry shape as a whole can avoid the ambiguous distinction of the forms of the PSFs between positions in front of and behind the image point corresponding to the object distance.
Exemplary solutions to the problems concerning the ambiguous distinction of the forms of the PSFs between positions in front of and behind the image point corresponding to the object distance are given above, and the exemplary solutions are: the method of increasing the number of the images each having a different focal position; and the method of using apertures having a non point-symmetry shape as a whole. However, the former method has a problem of increasing the capturing time due to the increase in number of images. In addition, the latter method has a problem in that the light is shielded by a part of the apertures and thus the amount of light decreases, thereby degrading the estimation accuracy of the object distance.
The present invention is conceived in view of the above problems, and an object of the present invention is to provide an imaging apparatus which eliminates the ambiguous distinction of the forms of the PSFs between positions in front of and behind the image point corresponding to the object distance without decreasing the amount of light to be exposed, and estimates the object distance from a small number of captured images.
To achieve the object described above, an imaging apparatus according to an aspect of the present invention includes: an imaging device which captures an image; an optical system for forming an image of an object on the imaging device; an optical element having a birefringence effect; and a distance measurement unit which measures a distance from the imaging device to the object, using the captured image and a point spread function having a form changed by the optical element between positions in front of and behind an image point corresponding to the distance to the object.
This configuration makes it possible to change the form of the point spread function (PSF) between the positions in front of and behind the image point corresponding to the object distance by using the optical element having the birefringence effect. This configuration allows elimination of the ambiguous distinction of the PSFs between positions in front of and behind the image point corresponding to the object distance, thereby allowing estimation of the object distance from a small number of captured images. Moreover, the decrease in the amount of light can be prevented in the case of using the birefringent substance because shielding light is unnecessary, compared to the method of using non point-symmetric apertures.
The optical element having the birefringence effect only affects astigmatism in general (especially when the birefringent substance is a parallel plate and the optical system is a telecentric optical system), unlike other optical elements. Therefore, even when the form of the PSF is changed between the positions in front of and behind the image point corresponding to the object distance, a small effect is given on other aberrations. Therefore, the optical system does not need to be re-designed. That is, the present invention can be achieved by only adding the optical element to a currently available apparatus which calculates the PSF.
Here, it is preferable that the optical element has an optic axis of which a direction is not parallel to a light axis of the optical system.
Here, it is preferable that the optical element is placed between the imaging device and the optical system on the light axis of the optical system, and a plane of the optical element which intersects with the light axis of the optical system is perpendicular to the light axis of the optical system.
Here, it is preferable to further include an optical element moving unit which turns on or off the birefringence effect on the light axis of the optical system by inserting or retracting the optical element with respect to the light axis of the optical system, in which the distance measurement unit measures the distance from the imaging device to the object, using an image captured by the imaging device without the birefringence effect of the optical element and using an image captured with the optical element placed on the light axis of the optical system.
Here, it is preferable that the optical element turns on or off the birefringence effect electrically or magnetically, and the distance measurement unit measures the distance to the object, using the image captured by the imaging device without the birefringence effect of the optical element and using an image captured with the optical element placed on the light axis of the optical system.
Here, it is preferable to further include a reference image generation unit which generates a reference image from the image captured by the imaging device without the birefringence effect of the optical element, in which the distance measurement unit estimates the point spread function and measure the distance to the object, using the reference image and the image captured through the optical element.
Here, it is preferable that the reference image generation unit generates an all-in-focus image as the reference image from the image captured by the imaging device without the birefringence effect of the optical element.
Here, it is preferable that the optical system has an optical property of image-space telecentricity.
Here, a light beam splitting unit which splits a light beam into light beams in a plurality of optical paths may further be included. The imaging device may include a plurality of imaging devices, and each of the imaging devices may capture the object according to a corresponding one of the optical paths resulting from the splitting by the light beam splitting unit, and the optical element may be placed on at least one of the optical paths resulting from the splitting by the light beam splitting unit.
Note that the present invention can be implemented not only as such an imaging apparatus, but also an imaging method including steps of operations of the characteristic elements included in the imaging apparatus. In addition, the present invention can be implemented as a program for causing a computer to execute the imaging method. Such a program can be distributed via a recording medium such as a CD-ROM, or a transmission medium such as the Internet. Furthermore, the present invention can be implemented as an integrated circuit which performs processing of each processing unit.
The imaging apparatus according to the present invention calculates a form of a PSF included in an image by using at least two images, one of which is the image including the PSF, thereby allowing stable and highly-accurate determination of an object distance.
In
Hereinafter, embodiments of the present invention are described with reference to the drawings. Note that the embodiments described as follows show preferred exemplary embodiments of the present invention. Elements, arrangement positions and connection conditions of those elements, and orders of operations described in the embodiments below are merely examples, and they are not intended to limit the present invention. The present invention is only defined by the scope of the claims. Therefore, the elements in the below-described embodiments but not described in the independent claims showing the most generic concepts are not always necessary to achieve the problems of the present invention, but they are described as elements configuring more preferred embodiments.
An imaging apparatus 10 includes: an optical system 11, a birefringent substance 12, an actuator 13, a focal range control unit 14, an imaging device 15, an image obtainment unit 16, a reference image generation unit 17, and a distance measurement unit 18.
In
Next, described is a method of changing, by the birefringent substance 12, a form of a PSF between positions in front of and behind the image point corresponding to the object distance.
The birefringent substance 12 is a substance having optical anisotropy, and has a property of separating a light beam into an ordinary ray and an extraordinary ray according to a polarization direction of the light beam entering the substance. The ordinary ray and the extraordinary ray are determined according to a direction of an optic axis (optic axis of a crystal) specific to the birefringent substance 12. The ordinary ray is a ray having an electric field oscillating perpendicular to a plane formed by the optic axis and the incident light beam. The extraordinary ray is a ray having an electric field oscillating within the plane. Note that the direction and the number of the optic axis changes depending on types of the substance. When the substance has one optic axis, it is called uniaxial, and when the substance has two optic axes, it is called biaxial. In Embodiment 1, calcite which is a uniaxial crystal is used as the birefringent substance 12.
The difference between the ordinary ray and the extraordinary ray is that when those rays pass through the birefringent substance 12, the speed of the ordinary ray is constant regardless of the direction of propagation of the ordinary ray. On the other hand, the speed of the extraordinary ray varies depending on the direction of propagation of the extraordinary ray. In addition, a refractive index “no” for the ordinary ray and a refractive index “ne” for the extraordinary ray are different. Because of the difference between the refractive index “no” for the ordinary ray and the refractive index “ne” for the extraordinary ray, and the property of varying the speed of the extraordinary ray depending on the direction of propagation of the ray, when the light beam enters the birefringent substance 12, the travelling direction differs between the ordinary ray and the extraordinary ray as shown in
The present invention uses especially the extraordinary ray to change the form of the PSF between the positions in front of and behind the image point corresponding to the object distance.
The positional relationship among the optical system 11, the birefringent substance 12, and the imaging device 15 is shown in
Use of the configuration in
Difference in form of the PSF between the positions in front of and behind the image point corresponding to the object distance eliminates the ambiguity in determination of the distance, thereby allowing unambiguous estimation of the object distance. Hereinafter, a description is given based on
When the birefringent substance 12 is used as in
In practice, the ordinary ray and the extraordinary ray are detected simultaneously in the configuration in
Next, a method of obtaining a reference image is described.
The imaging apparatus 10 according to Embodiment 1 uses a reference image (all-in-focus image) having no blur derived from the optical system 11. The image having no blur derived from the optical system 11 may also be referred to as an image having a deep depth of field. The depth of field can be easily deepened by narrowing the aperture of the optical system. However, the amount of light to be received by the imaging device 15 is decreased with this method. To solve this problem, techniques have been suggested for deepening the depth of field without narrowing the aperture. One of the techniques is called Extended Depth of Field (hereinafter, referred to as EDoF). Hereinafter, specific techniques of EDoF are described.
The simplest technique of EDoF is to capture a plurality of images while the focal position is gradually shifted during the capture, extract the focused parts from the obtained images, and combine them.
In contrast, Non-patent Literature (NPL) 2 discloses a technique of changing the focal position during exposure, and generating an image having no blur.
Specifically, moving the imaging device or the lens in the direction of the light axis during the exposure allows the PSFs to be almost constant regardless of the object distance, and a uniformly blurred image can be obtained. Performing deconvolution by using the PSF which is constant and free from the influence of the object distance makes it possible to obtain an image entirely having no blur.
On the other hand, a technique for EDoF using a special optical element is also suggested. One example is a method of using an optical element called a cubic phase mask. One example of the form of the cubic phase mask is shown in
Note that a method of changing the focal position during the exposure time is used in the following description as a technique for obtaining a reference image by extending the depth of field.
Next, a flow of the process of calculating the object distance is described.
First, an image I which is an image of the object captured through the birefringent substance, and a reference image I′ are obtained (Step S101 and Step S102). Note that the order of Step S101 and Step S102 may be reversed. Here, the reference image obtained here is an image of the object captured without the birefringent substance 12.
Here, the relationship expressed in Math 1 shown below is established between the image I and the reference image I′.
[Math. 1]
I(x,y)=I′(x,y)*h(x,y,d(x,y)) Expression 1
Here, h represents the PSF at a position (x, y) in the image, and d(x, y) represents the object distance at the position (x, y). In addition, * in the expression represents the convolution operation. The PSF differs depending on the object distance. Therefore, when objects are at different distances, the PSFs each having a different object distance at its corresponding position in the image is convoluted into an image having no blur to obtain the image I.
Next, an initial value 1 is substituted into a counter i (Step 103), and an error function C(x, y, di) for the object distance at an i-th stage is calculated for each pixel in the image (Step S104). The error function is expressed in Math 2 shown below.
[Math. 2]
C(x,y,di)=|(x,y)−I′(x,y)*h(x,y,di)| (i=1,2, . . . , n) Expression 2
Here, h(x, y, di) represents the PSF corresponding to the object distance di. The PSF corresponding to the object distance di (i=1 to n, where n is a natural number of 2 or more) is stored in advance in the memory in the imaging apparatus 10, for example. Expression 2 corresponds to calculating difference between the actually-captured image I and the image obtained by convoluting the PSF h(x, y, di) corresponding to the object distance di at the i-th stage into the reference image I′ having no blur. When the captured object is actually present at the i-th stage, the value of the error function C(x, y, di) is minimum, which is the difference between the image I and the reference image I.
The error function C(x, y, di) in Expression 2 is an absolute value of the difference between the actually-captured image I and the image obtained by convoluting, for each pixel, the PSF h(x, y, di) corresponding to the object distance di at the i-th stage into the image having no blur. The error function may also be obtained based on any form representing the distance, such as the L2 norm.
When the error function is calculated, it is determined whether the value of the counter i reaches n (Step S105). When the value of the counter i does not reach n, the value of the counter i is incremented by 1 (Step S106), and this process is repeated until the value of the counter i reaches n.
When all of the error functions from the first stage to the stage n are calculated, the object distance is calculated (Step S107). The object distance d(x, y) at the position (x, y) is expressed in Expression 3 shown below.
In practice, to reduce influence of a noise included in the captured image I, for example, the image is divided into blocks and the sum of the error functions of the blocks is obtained, and the object distance having the minimum error function is set to the distance of the object captured in the blocks as a whole. This process enables more stable distance measurement.
According to the above configuration, since the form of the PSF differs between the positions in front of and behind the image point corresponding to the object distance, the object distance can be estimated unambiguously.
In this embodiment, the direction of the optic axis of the birefringent substance 12 is upward in
Moreover, even though it is described that calcite, a uniaxial crystal, is used as the birefringent substance 12 in this embodiment, other materials having the birefringence effect may be used. Besides the direction of the optic axis, the number of the optic axis may also be used as a factor of controlling the form of the PSF. The effect of the present invention may also be obtained by using a biaxial birefringent subtance, besides the uniaxial birefringent substance. The range of the variation may be expanded by placing a plurality of uniaxial or biaxial birefringent substances, or both of the birefringent substances. Thickness and type of the birefringent substance also changes the form of the PSF to be obtained between positions in front of and behind the image point corresponding to the object distance.
Note that in this embodiment, it is described that the image of the object captured through the birefringent substance 12 and the image of the object captured without the birefringent substance 12 may be obtained by moving the birefringent substance by the actuator. However, other methods may be used. For example, in general, the images may be obtained by (i) moving the birefringent substance into or out of the optical path by the movement of the birefringent substance itself caused by physically driving the substance, or (ii) using an optical element capable of controlling the effect of the birefringence.
In the former way (i), for example, the birefringent substance is moved lineally by the actuator or the plate of the birefringent substance is rotated while the birefringent substance held perpendicular to the light axis, thereby creating a situation where the birefringent substance is on or out of the optical path. For the latter way (ii), examples of the element include an element capable of performing electric control such as the electrooptic effect, and an element capable of performing magnetic control. In these cases, the presence or absence of the effect of the birefringence may be controlled by applying and not applying the voltage or magnetic field. In addition, instead of the birefringent substance, a material such as a liquid crystal may be used, which is a material capable of controlling the effect of the birefringent substance electrically and magnetically.
The position of the birefringent substance is not limited to the position shown in
Note that the optical system 11 is desirable to be an optical system in which forms of PSFs are the same at all of the image heights. In particular, the optical system 11 is desirable to be an image-space telecentric optical system. The image-space telecentric optical system is an optical system in which the principle light beam and the light axis are parallel to each other at all of the image angles in the image space. In the configuration in
An imaging apparatus 19 according to Embodiment 2 of the present invention has a configuration of splitting the ordinary ray and the extraordinary ray, and obtaining an image including only the ordinary ray and an image including only the extraordinary ray.
In
More specifically, the imaging apparatus has a configuration as shown in
Examples of the light beam splitting unit 20 used for splitting the light beam include an unpolarized beam splitter and a polarized beam splitter. When the unpolarized beam splitter is used, the image I to be obtained includes both of the extraordinary ray and the ordinary ray as in Embodiment 1. When the polarized beam splitter is used, the image I only including the extraordinary ray can be obtained by controlling the optic axis of the birefringent substance and the direction of the polarization. Limiting the light beam included in the image Ito the extraordinary ray allows an image to be captured without a noise caused by the ordinary ray. Therefore, a more accurate image for deriving the object distance can be obtained. In addition, when the polarized beam splitter is used, the birefringent substance may also be placed between the polarized beam splitter and the optical system. In this case, the polarization direction needs to be selected such that only the ordinary ray reaches the imaging device B22.
Note that the image only including the extraordinary ray can also be obtained by only transmitting the extraordinary ray by using the optical element that only transmits a specific polarization such as a polarizer, although the amount of light in such an image decreases.
According to the above configuration, the image I and the reference image I′ can be obtained at the same time, and no difference other than the blur is generated in both of the images. Therefore, the object distance can be obtained more accurately. In the configuration in Embodiment 1, the image I and the reference image I′ are not obtained at the same time. Thus the relative position of the object with respect to the imaging apparatus may change due to the motion of the object and the imaging apparatus itself, and differences other than the blur are generated in both of the images and the accuracy of the distance measurement is likely to be degraded. However, if the capturing time for one image is the same, the amount of light entering one imaging device becomes larger in Embodiment 1 because the light beam is not split, and thus a signal-to-noise ratio (S/N ratio) is larger in Embodiment 1.
The image I and the reference image I′ are obtained in Embodiment 1 by temporally separating the functions of the imaging apparatus, while the image I and the reference image I′ are obtained in Embodiment 2 by spatially separating the functions of the imaging apparatus. Since the light beam is split in Embodiment 2, the amount of light for each of the image I and the reference image I′ is decreased, but the total amount of the light of both of the images is not decreased, and thus the amount of light is not wasted. If the time required for obtaining both of the images are the same in Embodiment 1 and Embodiment 2, the total amount of light is the same in Embodiment 1 and Embodiment 2.
Note that in Embodiment 1 and Embodiment 2, an all-in-focus image is used as the reference image to obtain the object distance, but it is not limited to this. An image having a uniform blur may be used as the reference image to derive the object distance.
Note that an LSI which is an integrated circuit may be usually used for a control unit of the actuator 13 which is a birefringence effect providing unit, the image obtainment unit 16 which is an imaging unit, and the distance measurement unit 18 among functional blocks in the block diagrams (such as
Although the integrated circuit is here referred to as an LSI, it may be referred to as an IC, a system LSI, a super LSI or an ultra LSI, depending on the degree of integration.
The method of forming integrated circuitry is not limited to use of LSIs. Dedicated circuitry or a general-purpose processor may be used instead of LSIs. Also applicable is a field programmable gate array (FPGA), which allows post-manufacture programming, or a reconfigurable processor LSI, which allows post-manufacture reconfiguration of connection and setting of circuit cells therein.
Furthermore, in the event that an advance in or derivation from semiconductor technology brings about an integrated circuitry technology whereby an LSI is replaced, the functional blocks may be obviously integrated by using such new technology. The adaptation of biotechnology or the like is possible.
Among the functional blocks, only a unit for storing data to be processed may be excluded from integration into a single chip and configured otherwise.
The imaging apparatus according to the present invention can measure a distance based on the image captured from a single viewpoint, and thus the present invention can be applied to general imaging apparatuses.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2010-260859 | Nov 2010 | JP | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/JP2011/006420 | 11/18/2011 | WO | 00 | 7/19/2012 |
