Combined stereoscopic and phase detection depth mapping in a dual aperture camera

Abstract

In an imaging system having a first camera with a first field of view (FOV) and a second camera with a second FOV smaller than the first FOV, wherein the first and second FOVs overlap over an overlap region, a method for calculating a calibrated phase detection depth map over the entire first FOV comprises calculating a stereoscopic depth map in the overlap region using image information provided by the first and second cameras, obtaining a first camera phase detection (PD) disparity map in the entire first FOV, and using the stereoscopic depth map in the overlap region to provide a calibrated 2PD depth map in the entire first FOV.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a 371 National Phase application from international application PCT/IB2017/055380 filed Sep. 6, 2017.

FIELD

Embodiments disclosed herein relate in general to digital cameras and in particular to dual-aperture digital cameras.

BACKGROUND

Digital camera modules are currently being incorporated into a variety of host devices. Such host devices include cellular telephones (e.g. smartphones), personal data assistants (PDAs), computers, and so forth. Some of these host devices include two or more digital camera modules (also referred to as optical imaging sub-systems or “sub-cameras”). When two such modules are used for example as “back” cameras in a smartphone, the back cameras provide a dual-aperture imaging system, also referred to a “dual-aperture camera”. A number of smartphone manufacturers already include dual-aperture cameras in their products.

Dual-aperture cameras disclosed by at least some of the present inventors may be found for example in U.S. Pat. Nos. 9,185,291, 9,392,188 and 9,413,972. In a dual-aperture digital camera, each sub-camera includes one or more lenses and/or other optical elements which define an aperture such that received electro-magnetic radiation is imaged by the optical sub-system and a resulting image is directed towards a two-dimensional (2D) pixelated image sensor region. The image sensor (or simply “sensor”) region is configured to receive the image and to generate a set of image data based on the image. The digital camera may be aligned to receive electromagnetic radiation associated with scenery having a given set of one or more objects. The set of image data may be represented as digital image data, as well known in the art. Hereinafter in this description, “image” “image data” and “digital image data” may be used interchangeably. Also, “object” and “scene” may be used interchangeably. As used herein, the term “object” is an entity in the real world imaged to a point or pixel in the image.

A sensor and its associated lens form a lens/sensor combination. The two lenses of a dual-aperture camera have different focal lengths. Thus, even though each lens/sensor combination is aligned to look in the same direction, each captures an image of the same subject but with two different fields of view (FOVs). In such cases, one camera and its lens and sensor are commonly called “Wide” and the other camera and its sensor and lens are commonly called “Tele”. Each sensor provides a separate image, referred to respectively as “Wide” (or “W”) and “Tele” (or “T”) images. A Wide image reflects a wider FOV (marked FOV_W) than a Tele image (where the FOV is marked FOV_T). The Wide image also has lower resolution than the Tele image.

Depth maps and associated methods to obtain such maps using multi-cameras (and in particular dual-aperture cameras) are known. A depth map is a rendition of depth values for all the pixels in an image. If one can calculate the depth value of each pixel, then in essence one gets a depth map. The depth map can be extracted or calculated from a disparity map (a rendition of disparity for each pixel) plus from additional information discussed below.

A depth map obtained with a dual-aperture camera is referred to as “stereoscopic” depth map. In some dual-aperture cameras, one (“reference”) camera is equipped with a Wide lens and with a Phase-Detection (PD) sensor. The second camera is equipped with a Tele lens, so that the overlapping field of view of the two cameras is partial relative to the FOV reference camera. The region of the Tele FOV that overlaps the Wide FOV is referred to as “overlap region”. All regions in the Wide FOV that are not overlapped by the Tele FOV are referred to as “non-overlap regions”. Alternatively, in some embodiments both cameras may be equipped with a 2PD sensor, i.e. a sensor in which each sensor pixel is divided into 2 sub-pixels and supports depth estimation via calculation of disparity between the image produced by all the right sub-pixels and that produced by all left sub-pixels. PD sensors take advantage of multiple micro-lenses (or partially covered micro-lenses) to detect pixels in and out of focus. Micro-lenses are calibrated so that objects in focus are projected onto the sensor plane at the same location relative to the lens, see FIG. 1.

FIG. 1 shows a point object 102 in focus, with a micro-lens projecting the light from the object onto the center of two sub-pixels, causing zero-disparity. FIG. 2 shows light-rays from a point object 202 out of focus. The left micro-lens projects the light from the object onto the center of the left sub-pixel. The right micro-lens projects the same object onto the right sub-pixel, causing a positive disparity value of 2, which is not directly related to the true distance of the object from the sensor. Objects before/after the focal plane are projected to different locations relative to each lens, creating a positive/negative disparity between the projections. As stated above, this disparity is zero for the focal plane and increases in magnitude as the object moves further away from that plane. The 2PD disparity information can be used to create a “2PD depth map”. Note that this 2PD depth map is both crude (due to a very small baseline) and relative to the focal plane. That is, zero-disparity is detected for objects in focus, rather than for objects at infinity.

All known methods to obtain depth maps using dual or multi-aperture cameras suffer from the problem that while the depth map is accurate in an overlap region, it is inaccurate in the non-overlap region. For camera arrays where the FOV of the modules is different (e.g. dual- or multi-aperture cameras with Wide and Tele lenses), a fine and absolute depth map can be extracted only for the overlap region using mainly the stereoscopic information. No absolute depth map can be obtained for the non-overlap regions. There is therefore a need for and it would be advantageous to have systems and methods to extend the absolute depth information to the non-overlap regions as well. Further, it would be advantageous to enhance the accuracy of the depth map in the overlap region, by relying on additional information from 2PD sensor(s).

SUMMARY

In an exemplary embodiment there is provided a method comprising providing an imaging system having a first camera with a first FOV and a second camera with a second FOV smaller than the first FOV, wherein the first FOV and the second FOV overlap over an overlap region, calculating a stereoscopic depth map in the overlap region using respective image information provided by the first and second cameras, obtaining a first camera 2 sub-pixel phase detection (2PD) disparity map in the entire first FOV, and improving the stereoscopic depth map or the 2PD depth map in at least the overlap region using the stereoscopic depth map in the overlap region and/or the first camera 2PD disparity map in the entire first FOV.

In an exemplary embodiment, the improving the stereoscopic depth map or the 2PD depth map in at least the overlap region includes using the stereoscopic depth map in the overlap region and/or the first camera 2PD disparity map in the entire first FOV to provide a calibrated 2PD depth map in the entire first FOV.

In an exemplary embodiment, the calculating a stereoscopic depth map in the overlap region includes calculating an absolute stereoscopic depth map in the overlap region.

In an exemplary embodiment, the calculating an absolute stereoscopic depth map in the overlap region includes cropping the image information provided by the first camera to match the second FOV so that disparity at infinity is zero.

In an exemplary embodiment, the using the stereoscopic depth map in the overlap region and the first camera 2PD disparity map in the entire first FOV to provide a calibrated PD depth map in the entire first FOV includes converting disparities in the first camera 2PD disparity map in the entire first FOV from pixel units into calibrated physical units based on the calibrated result of a stereo disparity.

In an exemplary embodiment, the improving the stereoscopic depth map or the 2PD depth map in at least the overlap region includes using the 2PD depth map to improve the stereoscopic depth map in the overlap region.

In an exemplary embodiment, the obtaining a 2PD depth map in the entire first FOV includes obtaining the 2PD depth map using a first camera image sensor.

In an exemplary embodiment, the obtaining a 2PD depth map in the entire first FOV includes obtaining the 2PD depth map using a first camera image sensor and a second camera image sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting examples of embodiments disclosed herein are described below with reference to figures attached hereto that are listed following this paragraph. The drawings and descriptions are meant to illuminate and clarify embodiments disclosed herein, and should not be considered limiting in any way. Like elements in different drawings may be indicated by like numerals. Elements in the drawings are not necessarily drawn to scale.

FIG. 2 shows an object in focus, with a micro-lens projecting the light from the object onto the center of two sub-pixels, causing zero-disparity;

FIG. 3 shows an object out of focus, with a left micro-lens projecting the light from the object onto the center of the left sub-pixel and a right micro-lens projecting the same object onto the center of the right sub-pixel, causing a positive disparity value of 2, which has no physical meaning of the true object distance;

FIG. 3 shows an exemplary baseline (d₁) between two cameras used for stereoscopic depth calculation;

FIG. 4 shows on the left an exemplary disparity map obtained from the 2PD sensor and on the right the disparity map for the same image generated using a stereoscopic method;

FIG. 5 shows an exemplary baseline (d₂) used for 2PD depth calculation;

FIG. 6 shows in a flow chart an exemplary embodiment of a method disclosed herein;

FIG. 7 shows in (a) an exemplary disparity map for an image generated using a 2PD sensor and on the right, in (b) the disparity map for the same image generated using a stereoscopic method, and in (c) a calibrated result with physical distance units as produced by the method described in the flow chart of FIG. 6.

DETAILED DESCRIPTION

An exemplary embodiment of a method to extend the absolute depth information obtained by stereoscopic vision in the overlap region to the non-overlap region as well is described next, with reference to FIG. 6. In the exemplary embodiment, the method includes:

In a step 602, provide Wide and Tele images having respective Wide and Tele field of view (FOVs)

In a step 604, crop the Wide image so that disparity at infinity is zero to provide a cropped Wide image. This prepares the Wide camera image for depth calculation.

In a step 606, calculate stereoscopic disparity (in absolute physical units) in the overlap region using the cropped Wide image and the Tele image to output a disparity map in absolute physical units in the overlap region. Such a disparity map has zero-disparity at infinity and in general follows equation 1:

$\begin{array}{cc}Z=\sqrt{\frac{F_{\mathrm{wide}}*d_1}{D_1}}& \left(1\right)\end{array}$ where Z is the reference (in this case Wide) camera-to-object distance in physical units (e.g. in mm), F_wide the focal length of the Wide camera, d₁ the distance between the centers of the main lenses (baseline) and D₁ is the disparity in pixels (see FIG. 3 for more details). For example, d₁=10 mm. Equation 1 provides stereoscopic conversion from disparity in pixel units to distance in physical units. Note that this equation is true for optimal conditions, which can be obtained by applying a known calibration process.

• • In a step 608, calculate a 2PD disparity map with disparities in pixel units in the entire Wide FOV using a 2PD sensor. FIG. 4 shows on the left an exemplary 2PD disparity map obtained from a 2PD sensor: the central region (marked with dots) is in focus and thus has zero disparity (pixel units). Other pixels have different disparity values, i.e. are out of focus, but their camera-to-object distance is unknown. On the right, FIG. 4 shows a disparity map for the same image generated using a known stereoscopic method the disparity in pixels is zero for objects at infinity and increases as the object distance from the camera decreases. Thus, the 2PD disparity values can be converted to the physical camera-to-object distances. Disparities in the stereoscopic map are much larger, as the baseline is larger (d₂<d₁).
  • The 2PD disparities may be converted from pixel units to distance in physical units using Equation 2.

$\begin{array}{cc}Z=\sqrt{\frac{F_{\mathrm{wide}}*d_2}{D_{\mathrm{infinity}}-D_2}}& \left(2\right)\end{array}$ where Z is the camera-to-object distance in physical units (e.g. mm), F_wide the focal length of the Wide camera, d₂ is approximately equal to 0.5×m where m is the diameter of the wide camera lens aperture and, D_infinity is the disparity of objects at infinity and D₂ the disparity in pixels, dependent on focal position (see FIG. 5 for more details). For example, m=2 mm. Note that D_infinity depends on the focus position.

• • In a step 610, compare the absolute stereoscopic physical distances obtained in step 606 with the 2PD disparities obtained in step 608 in the overlap region to create an absolute physical distance map for the entire Wide FOV For a given pixel in the overlap region, its camera-to-object distance in physical units (Z), should be identical for both (stereoscopic and 2PD) methods Based on equation 1 and equation 2, D_infinity is found using equation 3:

$\begin{array}{cc}{D}_{\mathrm{infinity}}=\frac{d_2*D_1}{d_1}+D_2& \left(3\right)\end{array}$ Using D_infinity, the disparities in the non-overlap region can now be converted into calibrated physical units, by applying equation 2.

FIG. 7 shows in (a) an exemplary disparity map for an image generated using a 2PD sensor and in (b) the disparity map for the same image generated using a stereoscopic method. Both (a) and (b) show disparities (offsets) in units of pixel. FIG. 7 shows in (c) a calibrated depth map obtained over the entire Wide FOV with a method disclosed herein, with physical distance units (e.g. cm). The dotted areas in (a), (b) and (c) represent the overlap region.

Alternatively or in addition to the extension of the absolute depth information obtained by stereoscopic vision in the overlap region to the non-overlap region, one may use the 2PD disparity map from step 608 above to enhance the result of stereoscopic disparity, step 612. The 2PD disparities may be obtained from the Wide camera alone, of from both the Wide and Tele cameras. The 2PD disparity map can be used to define a local search range for the stereoscopic algorithm. 2PD disparity can be calculated along a vertical (e.g. Y) axis, while stereoscopic disparity can be calculated along a horizontal (e.g. X) axis, or vice-versa (depending on hardware assembly). Objects lying along a single axis will be better detected by one calculation than by the other (i.e. by 2PD disparity vs. stereoscopic disparity or vice versa). Such objects are detected and greater reliability is assigned to the appropriate choice.

The level of disagreement between the depth calculated by the 2PD disparity and the depth calculated by stereoscopic disparity algorithms can be used as a reliability measure per pixel. For example, after the calibration of the 2PD disparity map (using steps 600-610 above for the overlap region only), one may compare the depth calculated by both methods. In case of significant disagreement (for example, if the stereoscopic disparity method can reach an accuracy of ±1 pixel, “significant disagreement” may be defined as more than 2 pixels), this depth value can be considered unreliable and marked as an outlier.

In conclusion, using either steps 600-610 or steps 600-608 plus the enhancement of stereoscopic disparity described above, the entire FOV of the Wide camera will have absolute disparity values (i.e. true physical distance to an object), with the overlap region obtaining these absolute values from the stereoscopic+2PD values and the non-overlap region obtaining these absolute values based on Equation 3.

While this disclosure has been described in terms of certain embodiments and generally associated methods, alterations and permutations of the embodiments and methods will be apparent to those skilled in the art. For example, while the usage of 2 cameras for depth calculation is described in some detail, depth information may be extracted from multiple (>2) cameras as well. The disclosure is to be understood as not limited by the specific embodiments described herein.

All references mentioned in this application are hereby incorporated by reference in their entirety for all purposes set forth herein. It is emphasized that citation or identification of any reference in this application shall not be construed as an admission that such a reference is available or admitted as prior art.

Claims

1. A method, comprising: obtaining respective image information by applying an imaging system having a first camera with a first field of view (FOV) and a second camera with a second FOV smaller than the first FOV, wherein the first FOV and the second FOV overlap over an overlap region;calculating a stereoscopic depth map in the overlap region using the respective image information provided by the first and second cameras;obtaining a first camera 2 sub-pixel phase detection (2PD) disparity map in the entire first FOV; andgenerating a calibrated 2PD depth map in the entire first FOV to improve the stereoscopic depth map or a 2PD depth map in at least the overlap region based on the stereoscopic depth map in the overlap region and the first camera 2PD disparity map in the entire first FOV.

2. The method of claim 1, wherein the generating the calibrated 2PD depth map in the entire first FOV to improve the stereoscopic depth map or the 2PD depth map in at least the overlap region includes using the stereoscopic depth map in the overlap region ands the first camera 2PD disparity map in the entire first FOV.

3. The method of claim 2, wherein the calculating a stereoscopic depth map in the overlap region includes calculating an absolute stereoscopic depth map in the overlap region.

4. The method of claim 3, wherein the calculating an absolute stereoscopic depth map in the overlap region includes cropping the image information provided by the first camera to match the second FOV so that disparity at infinity is zero.

5. The method of claim 4, wherein the using the stereoscopic depth map in the overlap region and the first camera 2PD disparity map in the entire first FOV includes converting disparities in the first camera 2PD disparity map in the entire first FOV from pixel units into calibrated physical units based on the calibrated result of a stereo disparity.

6. The method of claim 1, wherein the generating the calibrated 2PD depth map in the entire first FOV to improve the stereoscopic depth map or the 2PD depth map in at least the overlap region includes using the 2PD depth map to improve the stereoscopic depth map in the overlap region.

7. The method of claim 6, wherein the obtaining the 2PD depth map in the entire first FOV includes obtaining the 2PD depth map using a first camera image sensor.

8. The method of claim 6, wherein the obtaining the 2PD depth map in the entire first FOV includes obtaining the 2PD depth map using a first camera image sensor and a second camera image sensor.

9. A method comprising: obtaining respective image information by applying an imaging system having a first camera with a first field of view (FOV) and a second camera with a second FOV smaller than the first FOV, wherein the first FOV and the second FOV overlap over an overlap region;calculating a stereoscopic depth map in the overlap region using the respective image information provided by the first and second cameras;obtaining a first camera 2 sub-pixel phase detection (2PD) disparity map in the entire first FOV; andgenerating a calibrated 2PD depth map in the entire first FOV to improve a 2PD depth map in at least the overlap region based on the stereoscopic depth map in the overlap region and the first camera 2PD disparity map in the entire first FOV.