3D CALIBRATION METHOD AND APPARATUS FOR MULTI-VIEW PHASE SHIFT PROFILOMETRY

Abstract

Disclosed is a three-dimensional (3D) calibration method and device for multi-view phase shift profilometry. A 3D calibration method for multi-view phase shift profilometry, performed by a computer device, according to an example embodiment may include acquiring a phase that is data required for 3D reconstruction from parameters of at least one camera and a projector based on a phase measuring profilometry (PMP) method; defining a phase-to-depth function for each camera-and-projector combination and performing calibration of optimizing the parameters of the camera and the projector; and acquiring a point cloud that includes depth information using the optimized parameters of the camera and the projector.

Description

BACKGROUND

1. Field of the Invention

The following example embodiments relate to a three-dimensional (3D) calibration method and apparatus for multi-view phase shift profilometry, and more particularly, to a 3D calibration method and apparatus for multi-view phase shift profilometry that may improve quality of a 3D reconstruction task in a multi-view system.

2. Description of the Related Art

Acquiring geometric data of objects present in a three-dimensional (3D) space, that is, 3D scanning refers to technology with a great value. In particular, 3D scanning may be technology with infinite utility, such as equipment to acquire and inspect information on a 3D shape of an object, such as a semiconductor.

Representative 3D scanning technology includes a phase measuring profilometry (PMP) method, which is a method of acquiring 3D information of an object by projecting a predefined structured light image through a projector and capturing the same through a camera using the camera and the projector. However, when performing 3D scanning of an object with this method, there are restrictions that the camera needs to be able to view a point to be acquired and light emitted from the projector needs to reach the point. For example, when it is assumed that the camera is present in a downward direction from the top of a system, side data of an object that may be acquired as a result of 3D scanning may be very insufficient. Therefore, to perform 3D scanning of the object in all directions, the projector needs to emit light from as many different directions as possible and it is necessary to perform capturing from cameras present at various positions.

In addition, to acquire high-quality results, it is necessary to combine 3D data, to discard unnecessary points, and to process data based on more reliable points, rather than simply acquiring the 3D data with a combination of a single camera-and-projector pair.

SUMMARY

Example embodiments describe a three-dimensional (3D) calibration method and apparatus for multi-view phase shift profilometry, and more particularly, provide technology for optimizing various parameters including position information of a projector and a camera in a system with a plurality of top and side cameras.

Example embodiments provide a 3D calibration method and apparatus for multi-view phase shift profilometry that may perform calibration of a phase measuring profilometry (PMP) system for 3D reconstruction with only one target by performing shooting for a calibration target and then adding a depth map of a side camera to a depth map of a main camera and by acquiring a final 3D point cloud in which a depth map fusion is performed and may perform calibration and 3D reconstruction with only one scene without performing additional shooting while moving a corresponding target.

According to an example embodiment, there is provided a 3D calibration method for multi-view phase shift profilometry, performed by a computer device, the 3D calibration method including acquiring a phase that is data required for 3D reconstruction from parameters of at least one camera and a projector based on a phase measuring profilometry (PMP) method; defining a phase-to-depth function for each camera-and-projector combination and performing calibration of optimizing the parameters of the camera and the projector; and acquiring a point cloud that includes depth information using the optimized parameters of the camera and the projector.

The acquiring of the phase may include projecting a series of structured light images using the projector; detecting projected light that is reflected from an object using a sensor of the at least one camera; and acquiring the phase required for the 3D reconstruction using a single camera-and-projector pair.

The acquiring of the phase may include configuring a main camera located at the top and a side camera located on at least one side surface, and acquiring the phase based on the PMP method using the projector located between the main camera and the side camera.

The 3D calibration method may further include performing transformation from world coordinates to camera coordinates for all points in the point cloud, storing a z value of the camera coordinates, and acquiring a depth map for each camera-and-projector pair; and computing pixel-wise confidences of all the depth maps and performing the 3D reconstruction by acquiring a final point cloud through a confidence-based depth map fusion that integrates the respective depth maps into a single depth map based on the confidences.

The performing of the 3D reconstruction by acquiring the final point cloud may include performing a depth map fusion through a weighted sum with a depth map of the main camera along with a corresponding confidence for points visible to the main camera, with respect to points in a 3D space acquirable through a depth of each pixel when computing the pixel-wise confidences of the depth maps; and performing the depth map fusion through a weighted sum in a depth map of a remaining side camera for points not visible to the main camera.

According to another example embodiment, there is provided a 3D calibration apparatus for multi-view phase shift profilometry, the 3D calibration apparatus including a phase acquisition unit configured to acquire a phase that is data required for 3D reconstruction from parameters of at least one camera and a projector based on a PMP method; a calibration unit configured to define a phase-to-depth function for each camera-and-projector combination and to perform calibration of optimizing the parameters of the camera and the projector; and a point cloud acquisition unit configured to acquire a point cloud that includes depth information using optimized parameters of the camera and the projector.

The phase acquisition unit may be configured to acquire the phase required for the 3D reconstruction using a single camera-and-projector pair in such a manner that the projector projects a series of structured light images and projected light is reflected from an object and detected at a sensor of the at least one camera.

The phase acquisition unit may be configured to configure a main camera located at the top and a side camera located on at least one side surface, and to acquire the phase based on the PMP method using the projector located between the main camera and the side camera.

The 3D calibration apparatus may further include a depth map acquisition unit configured to perform transformation from world coordinates to camera coordinates for all points in the point cloud, to store a z value of the camera coordinates, and to acquire a depth map for each camera-and-projector pair; and a depth map fusion unit configured to compute pixel-wise confidences of all the depth maps and to perform the 3D reconstruction by acquiring a final point cloud through a confidence-based depth map fusion that integrates the respective depth maps into a single depth map based on the confidences.

The depth map fusion unit may be configured to perform a depth map fusion through a weighted sum with a depth map of the main camera along with a corresponding confidence for points visible to the main camera, with respect to points in a 3D space acquirable through a depth of each pixel when computing the pixel-wise confidences of the depth maps, and to perform the depth map fusion through a weighted sum in a depth map of a remaining side camera for points not visible to the main camera.

According to example embodiments, it is possible to provide a 3D calibration method and apparatus for multi-view phase shift profilometry that may optimize various parameters including position information of a projector and a camera and may improve quality of a 3D reconstruction task in a system with a plurality of top and side cameras.

Also, according to some example embodiments, it is possible to provide a 3D calibration method and apparatus for multi-view phase shift profilometry that may perform calibration of a PMP system for 3D reconstruction with only one target by performing shooting for a calibration target and then adding a depth map of a side camera to a depth map of a main camera and by acquiring a final 3D point cloud in which a depth map fusion is performed and may perform calibration and 3D reconstruction with only one scene without performing additional shooting while moving a corresponding target.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate a three-dimensional (3D) model of a calibration target and a two-dimensional (2D) view including Aruco markers according to an example embodiment.

FIGS. 2A and 2B illustrate an actual captured image of a calibration target according to an example embodiment.

FIGS. 3A and 3B illustrate Aruco marker detection results from an image and reprojection results of a recognized marker using an optimized parameter according to an example embodiment.

FIG. 4 illustrates an example (vertical axis) of an image to be projected from a projector according to an example embodiment.

FIGS. 5A and 5B illustrate decoding results of a structured light pattern image captured by a camera according to an example embodiment.

FIGS. 6A and 6B illustrate a phase unwrapping input value and output value according to an example embodiment.

FIG. 7 illustrates acquisition of 3D data through camera pixel coordinates and phase according to an example embodiment.

FIGS. 8A and 8B illustrate a confidence computation process of a depth map according to an example embodiment.

FIG. 9 illustrates a target image, a depth map, and confidence map according to an example embodiment.

FIG. 10 is a flowchart illustrating a 3D calibration method for multi-view phase shift profilometry according to an example embodiment.

FIG. 11 is a block diagram illustrating a 3D calibration apparatus for multi-view phase shift profilometry according to an example embodiment.

FIG. 12 is a diagram for explaining a depth map fusion according to an example embodiment.

FIGS. 13A and 13B illustrate a final 3D point cloud according to an example embodiment.

DETAILED DESCRIPTION TO CARRY OUT THE INVENTION

Hereinafter, example embodiments will be described with reference to the accompanying drawings. However, the example embodiments may be modified in various other forms and the scope of the present invention is not limited to the example embodiments described below. Also, various example embodiments are provided to more fully explain the present invention to one of ordinary skill in the art. Shapes and sizes of elements in the drawings may be exaggerated for clear explanation.

The example embodiments aim to perform three-dimensional (3D) reconstruction of an object using a main camera located at the top of a system, a side camera located at the side of the system, and a projector located therebetween. To this end, 3D data of the object is to be acquired using a commonly available phase measuring profilometry (PMP) method.

First, in the PMP method, a projector projects a series of structured light images and projected light is reflected from an object and sensed by a sensor of a camera. In this case, data required for 3D reconstruction, that is, a phase may be acquired using a single camera-and-projector pair. A process of generating the structured light image is called encoding, and a process of transforming an image captured from the camera to a phase map is called decoding. Here, a final goal of the PMP method is to define and optimize a phase-to-depth function for each camera.

The phase-to-depth function is to transform phase to depth and may be expressed in various types of models. In the example embodiment, a stereo model is adopted. A stereo triangulation model models the projector with a single pinhole camera and acquires 3D data in the same manner as the existing stereo matching. In the stereo triangulation model, the phase-to-depth function may be expressed as a function of an intrinsic parameter and an extrinsic parameter of the camera and the projector. Therefore, a main goal of the example embodiment may be to establish a calibration process of optimizing parameters of the camera and the projector.

The phase-to-depth function may be defined for every camera-and-projector combination at the same time of performing calibration of the camera and the projector.

However, through the above process, each camera-and-projector combination may be acquired and 3D data may be acquired in a form of an independent point cloud or depth map. Here, an error may occur therebetween. Therefore, 3D reconstruction is finally completed through a confidence-based depth map fusion that integrates the respective depth maps into a single depth map. In this confidence-based depth map fusion process, a confidence is computed for each pixel of each depth map. Here, points in a 3D space that may be acquired through depth of each pixel may be divided into two cases: a case in which a point is visible to a main camera located at the top and a case in which a point is not visible.

For points visible to the main camera, a depth map fusion is performed through a weighted sum with a depth map of the main camera along with a corresponding confidence. For points not visible to the main camera, the depth map fusion is performed through a weighted sum in a depth map of a remaining side camera.

Hereinafter, a 3D calibration method and device for multi-view phase shift profilometry according to an example embodiment will be described in more detail.

Table 1 shows a notation.

TABLE 1
Notation        Description
[x y z], [x y]  3D or 2D coordinates having x, y, and
                z axis or x and y axis coordinate values
[u v]           Pixel coordinates having horizontal axis
                and vertical axis coordinate values
Xw = [xw yw zw] World coordinates of 3D point
Xc = [xc yc zc] Camera coordinates of 3D point
R, t            Rotation and translation of camera,
                projector, and target plane
p = [u v]       Camera pixel coordinates
xn              Point on normalized plane of camera
                and projector
φf              Wrapped phase in image of frequency f
ψx · ψy         Unwrapped phase in direction of horizontal
                axis and vertical axis
n, c            Normal and plane equation constant of plane
d(u, v)         depth at (u, v) pixel of depth map
C(u, v)         Confidence at (u, v) pixel of depth map
D(u, v)         Depth at (u, v) pixel after depth map fusion

Camera Parameter

Camera parameters may be divided into an extrinsic parameter, such as R and t, and an intrinsic parameter including K and lens distortion coefficients (k₁, k₂, k₃). The extrinsic parameter refers to a parameter for a relationship between a world coordinate system and a camera coordinate system. The following relationship is established between a world coordinate X_w of a point present in a 3D space and a camera coordinate X_c.

$\begin{array}{cc}{X}_c={\left[\begin{array}{ccc}{x}_c& y_c& z_c\end{array}\right]}^{’}={\mathrm{RX}}_w+t.& \left[\mathrm{Equation}1\right]\end{array}$

Here, R and t are 3×3 and 3×1 matrices, respectively, each expressing a rigid-body transform in the 3D space and may be expressed as follows.

$\begin{array}{cc}\begin{array}{c}T=\left[\begin{array}{cc}R& t\\ 0& 1\end{array}\right]=\stackrel{\_}{\exp }\left(q\right),q{\in }^{~6},T{\in }^{~4\times 4},\exp :\mathrm{se}\left(3\right)\text{⟶}\mathrm{SE}\left(3\right)⟶\stackrel{\_}{\exp }{:}^{~6}{\text{⟶}}^{~4\times 4}\\ q=\left[\begin{array}{cccccc}{t}_x& t_y& t_z& q_0& q_1& q_2\end{array}\right]\end{array}& \left[\mathrm{Equation}2\right]\end{array}$

In [Equation 2], vector q includes a total of six elements, three translations and three quaternions.

In a pinhole camera model, p_c is projected onto the normalized image plane, that is, the plane of z=1 in the camera coordinate system.

$\begin{array}{cc}{\hat{x}}_n={\left[\begin{array}{ccc}{x}_n& y_n& 1\end{array}\right]}^{’}={\left[\begin{array}{ccc}{x}_c/z_c& y_c/z_c& 1\end{array}\right]}^{’}& \left[\mathrm{Equation}3\right]\end{array}$

In [Equation 3], {circumflex over (x)}_c denotes a homogeneous coordinate of x_c. Meanwhile, a camera using a telecentric lens, such as the main camera at the top of the system, is projected in a slightly different manner.

$\begin{array}{cc}{\hat{x}}_n={\left[\begin{array}{ccc}{x}_n& y_n& 1\end{array}\right]}^{’}={\left[\begin{array}{ccc}{x}_c& y_c& 1\end{array}\right]}^{’}& \left[\mathrm{Equation}4\right]\end{array}$

An intrinsic parameter of the camera specifies a relationship between x_n and a pixel coordinate p. First, radial distortion correction by a camera lens is performed at normalized image coordinates.

$\begin{array}{cc}\begin{array}{c}{x}_n=x_n^{\prime }\left(1+k_1{r}_n^2+k_2{r}_n^4+k_3{r}_n^6\right)\\ y_n=y_n^{\prime }\left(1+k_1{r}_n^2+k_2{r}_n^4+k_3{r}_n^6\right)\\ r_n^2=x_n^{\prime 2}+y_n^{\prime 2}\end{array}& \left[\mathrm{Equation}5\right]\end{array}$

x_c′ and y_c′ denote distorted normalized coordinates distorted in the x axis and the y axis, respectively. Then, the camera matrix K denotes a relationship between a distorted normalized camera coordinate x_n′ and the pixel coordinate p.

$\begin{array}{cc}p=K{\hat{x}}_n^{\prime }& \left[\mathrm{Equation}6\right]\end{array}$

Through the above process, the pixel coordinate or vice versa may be computed from an arbitrary 3D point X_w.

Calibration Target Configuration

When it is assumed that there is no distortion by the camera lens, a relationship between the 3D point X_w and the corresponding pixel p in the pinhole camera model is expressed as follows.

$\begin{array}{cc}\begin{array}{cc}s·\hat{p}=K[R❘t]{\hat{p}}_w=K[R❘t]\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right],& K=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\end{array}& \left[\mathrm{Equation}7\right]\end{array}$

s denotes a scale factor of a homogeneous coordinate, {circumflex over (p)}, each of ƒ_x, ƒ_y denotes a focal length, and each of u₀ and v₀ denotes a principal point. Meanwhile, a case of a telecentric model is as follows.

$\begin{array}{cc}\hat{p}=\left[\begin{array}{cccc}{f}_x& 0& 0& u_0\\ 0& f_y& 0& v_0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cc}R& t\\ 0& 1\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]& \left[\mathrm{Equation}8\right]\end{array}$

Here, R and t denote transformation matrices from a checkerboard coordinate system to the camera coordinate system. Here, a relationship between a checkerboard present on one plane and a corresponding pixel may be explained by adding a condition of z_w=0.

$\begin{array}{cc}s·\hat{p}=K\left[R❘t\right]\left[\begin{array}{c}{x}_w\\ y_w\\ 0\\ 1\end{array}\right]=K\left[\begin{array}{ccc}{r}^1& r^2& t\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ 1\end{array}\right]=H\left[\begin{array}{c}{x}_w\\ y_w\\ 1\end{array}\right],R=\left[\begin{array}{ccc}{r}^1& r^2& r^3\end{array}\right].& \left[\mathrm{Equation}9\right]\end{array}$

Likewise, in the case of a telecentric lens camera, it may be expressed as follows.

$\begin{array}{cc}\hat{p}=\left[\begin{array}{cccc}{f}_x& 0& 0& u_0\\ 0& f_y& 0& v_0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cc}R& t\\ 0& 1\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ 0\\ 1\end{array}\right]=\left[\begin{array}{cccc}{f}_x& 0& 0& u_0\\ 0& f_y& 0& v_0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}{r}^1& r^2& t\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ 1\end{array}\right]=H\left[\begin{array}{c}{x}_w\\ y_w\\ 1\end{array}\right].& \left[\mathrm{Equation}10\right]\end{array}$

Here, homography H may acquire a solution in a closed form through coordinates of a predefined checkerboard corner and values of pixel coordinates recognized from an image. The homography includes product of K and extrinsic parameters R and t. However, to inversely compute parameters from H, a plurality of items of homography need to be secured by capturing checkerboard images of various orientations. Theoretically, at least three items of homography are required to compute solutions of all parameters.

Therefore, a camera calibration process generally starts with acquiring images by variously changing an angel of a plane on which a checkerboard pattern is engraved. However, a novel calibration target object is proposed to take advantage of a multi-view system and to perform a projector calibration, which is described below.

FIG. 1 illustrates a 3D model of a calibration target and a 2D view including Aruco markers according to an example embodiment. In detail, (a) of FIG. 1 illustrates the 3D model of the calibration target according to an example embodiment and (b) of FIG. 1 illustrates the 2D view including Aruco markers according to an example embodiment.

FIG. 2 illustrates an actual captured image of a calibration target according to an example embodiment. In detail, (a) of FIG. 2 illustrates an image captured by a main camera located at the top according to an example embodiment, and (b) of FIG. 2 illustrates an image captured by a side camera 01 (west) according to an example embodiment.

The proposed calibration target is designed to meet the following conditions while performing the example embodiments.

• • One camera needs to be able to view as many planes as possible.
  • At least two cameras capable of viewing one plane need to be present.
  • For more accurate homography computation, at least three Aruco markers need to be arranged on one plane.
  • Aruco markers with the same ID are not used several times.

An ID of each Aruco marker of a target does not overlap. Therefore, when a single marker is recognized by a camera, on which plane a corner of the marker is present may be known. Also, each world coordinate system may be defined based on a bottom plane and a coordinate system on a remaining plane may be transformed to a world coordinate system through R and t. For example, if transformation from a local coordinate system to a world coordinate system on an i^th plane is R_plane,ƒ, t_plane,ƒ, it may be expressed as follows.

$\begin{array}{cc}{X}_w=R_{\mathrm{plane},i}{X}_{\mathrm{plane},i}+t_{\mathrm{plane},i}=R_{\mathrm{plane},i}\left[\begin{array}{c}{x}_{\mathrm{plane},i}\\ y_{\mathrm{plane},i}\\ 0\end{array}\right]+t_{\mathrm{plane},i}=\left[r_{\mathrm{plane},i}^1{r}_{\mathrm{plane},i}^2\right]\left[\begin{array}{c}{x}_{\mathrm{plane},i}\\ y_{\mathrm{plane},i}\end{array}\right]+t_{\mathrm{plane},i}& \left[\mathrm{Equation}11\right]\end{array}$

Camera Calibration

FIG. 3 illustrates Aruco marker detection results from an image and reprojection results of a recognized marker using an optimized parameter according to an example embodiment. In detail, (a) of FIG. 3 illustrates Aruco marker detection results from the image according to an example embodiment, and (b) of FIG. 3 illustrates reprojection results of the recognized marker using the optimized parameter according to an example embodiment. Camera distortion correction is not applied to the image in (a), while the camera distortion correction is applied to the image in (b).

In a typical calibration method of capturing a checkerboard pattern once for each image, one homography is computed for each image and a checkerboard pattern at this time is captured over wide pixels. On the other hand, in a method according to an example embodiment of capturing a plurality of checkerboard planes in one image, a single checkerboard pattern is effectively applied in a narrow area within the image. In the case of an actual camera, a distortion phenomenon by a lens is present and the distortion phenomenon relates to not a linear issue but a nonlinear issue. Therefore, it is difficult to optimize K, R, and t through a plurality of items of homography and it is easy to fall into local minima.

In the case of falling into local minima, it indicates that not only camera parameters but also a coordinate system transformation of each plane in a calibration target may be erroneously optimized. However, in the case of using the aspect that one plane is visible to a plurality of cameras, a camera parameter may be computed without falling into local optima by taking the advantage of a multi-view system.

Initially, since which ID marker is present on each plane is known and local coordinates of each marker corner are predefined, world coordinates of each marker corner may be expressed as a function that uses R_plane,ƒ, t_plane,ƒ as parameters. Then, p predicted through [Equation 1], [Equation 3], [Equation 4], [Equation 5], and [Equation 6] may be expressed as a function that uses world coordinates of the marker corner and an intrinsic parameter and an extrinsic parameter of the camera as parameters and may be compared to actual p^gt (undistorted). Accordingly, by optimizing a parameter that simultaneously minimizes a sum of marker corners recognized by all cameras, not only each parameter of all cameras but also R_plane,ƒ, t_plane,ƒ for all planes of the calibration target may also be optimized. Since a total of five cameras capture a target including 18 planes, the system used in the example embodiment may be expressed as follows.

$\begin{array}{cc}\begin{array}{c}\min \text{?}\sum _i\sum _j\sum _p{p\left(X_{\mathrm{plane},i}\right)-p^{\mathrm{gi}}}_2^2,\\ i=0,1,\dots ,17,j=0,1,2,3,4\end{array}& \left[\mathrm{Equation}12\right]\end{array}$ $\text{?}\text{indicates text missing or illegible when filed}$

Phase Decoding and Unwrapping

FIG. 4 illustrates an example (vertical axis) of an image to be projected from a projector according to an example embodiment. Referring to FIG. 4, four images may be generated for each of frequencies ƒ=4⁰, 4¹, 4², 4³ and a total of 16 images may be generated and, in the same manner, 16 horizontal axis images may be generated and projected from the projector.

Pixel intensity of an image projected from the projector is in a cosine-wave form of various frequencies in a horizontal axis direction or a vertical axis direction. Frequencies actually used in the example embodiment are ƒ=4⁰, 4¹, 4², 4³. total of 4×4=16 images are projected in each of the vertical direction and the horizontal direction by shifting a phase by

$0,\frac{\pi }{2},\pi ,\frac{3}{2}\pi$

for waves generated at a total of four frequencies.

Light projected from the projector is reflected from an object and detected by a camera. When intensity of an i^th image projected with the frequency ƒ at a pixel (u, v) is I_ƒ,t(u,v), it may be expressed as follows.

$\begin{array}{cc}{I}_{f,i}\left(u,v\right)=A\left(u,v\right)+B\left(u,v\right)\cos \left[{\varphi }_f\left(u,v\right)-\frac{2\pi i}{N}\right],N=4.& \left[\mathrm{Equation}13\right]\end{array}$

Here, by combining all pixel intensities at i=0, 1, 2, and 3, phase ϕ_ƒ(u, v) at the pixel (u, v) when the frequency is ƒ may be acquired as follows.

$\begin{array}{cc}{\varphi }_f\left(u,v\right)=\mathrm{arc}\tan \left\{\frac{\sum _{i=0}^N{I}_{f,i}\left(u,v\right)\sin \left(\frac{2\pi i}{N}\right)}{\sum _{i=0}^N{I}_{f,i}\left(u,v\right)\cos \left(\frac{2\pi i}{N}\right)}\right\}& \left[\mathrm{Equation}14\right]\end{array}$

FIG. 5 illustrates decoding results of a structured light pattern image captured by a camera according to an example embodiment. Referring to (a) of FIG. 5, a wrapped phase on the right may be acquired through four images on the left (ƒ=4⁰). Referring to (b) of FIG. 5, a wrapped phase on the right may be acquired through four images on the left (ƒ=4¹).

However, ϕ_ƒ(u, v) is a wrapped phase, not an unwrapped phase that is actually used for 3D reconstruction. A wrapped phase at ƒ=4⁰ includes all areas of an object in a single wavelength, but may not contain detailed information. An unwrapped phase at ƒ=4³ includes locally detailed information, but does not include all areas in a single wavelength.

Therefore, after computing the wrapped phase for all frequencies ƒ, the unwrapped phase ψ(u, v) that encompasses all the wrapped phases from ƒ=4⁰ to ƒ=4³ needs to be computed.

FIG. 6 illustrates a phase unwrapping input value and output value according to an example embodiment.

(a) of FIG. 6 illustrates the phase unwrapping input value according to an example embodiment and (b) of FIG. 6 illustrates the phase unwrapping output value according to an example embodiment. A final unwrapped phase may be computed by integrating all the wrapped phases that are ƒ=4⁰, 4¹, 4², 4³.

When ψ_ƒ is an unwrapped phase that encompasses wrapped phases from ƒ=4⁰ to ƒ=4^j, it may be expressed as follows.

$\begin{array}{cc}{\psi }_0\left(u,v\right)={\varphi }_0\left(u,v\right).& \left[\mathrm{Equation}15\right]\end{array}$

Also, a process of computing ψ_j+1(u, v) using ϕ_j+1(u, v) and ψ_j(u, v) is as follows.

$\begin{array}{cc}\begin{array}{c}{n}_{j+1}\left(u,v\right)=\mathrm{round}\left(\frac{4{\psi }_j-{\varphi }_{j+1}}{2\pi }\right),\\ {\psi }_{j+1}\left(u,v\right)={\varphi }_{j+2}+2\pi n_{j+1}.\end{array}& \left[\mathrm{Equation}16\right]\end{array}$

A finally unwrapped phase ψ(u, v)=ψ_j(u, v) is used as an input value for the 3D reconstruction.

Projector Calibration

FIG. 7 illustrates acquisition of 3D data through camera pixel coordinates and phase according to an example embodiment. Referring to FIG. 7, in a calibration stage, loss is constructed and optimized using a distance value between an intersection point X_intersect between a camera 710 and a projector 720 and a target plane 730.

In the example embodiment, the projector uses a pinhole camera model, such as a side camera. However, unlike the side camera capable of directly acquiring pixel coordinates, the projector needs to indirectly use an unwrapped phase captured by the camera and thus, a slight change in an intrinsic parameter of the projector is required. Basically, the unwrapped phase has a linear relationship with pixel coordinates (u, v)_projector in the pinhole camera model.

$\begin{array}{cc}\begin{array}{c}{\psi }_x=w_{x}u+b_x,\\ {\psi }_y=w_{x}u+b_x,\end{array}& \left[\mathrm{Equation}17\right]\end{array}$

Substituting this into [Equation 6], it may be expressed as follows.

$\begin{array}{cc}\left[\begin{array}{c}{\psi }_x\\ {\psi }_y\\ 1\end{array}\right]=\left[\begin{array}{ccc}{w}_x& 0& b_x\\ 0& w_y& b_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_x& v_0\\ 0& 0& 1\end{array}\right]{\stackrel{*}{X}}_{n,p}& \left[\mathrm{Equation}18\right]\end{array}$

In [Equation 18], X_n,p denotes distorted coordinates present on a normalized projector plane in the projector. Also, assuming that all parameters of a camera and a projector are known in any one camera-and-projector pair, a 3D point acquired by using phase and pixel coordinates as input values needs to meet the following equation.

$\begin{array}{cc}\left\{\begin{array}{c}R\text{?}{x}_w+R\text{?}{y}_w+R\text{?}{z}_w+t\text{?}=x_n\left(R\text{?}{x}_N+R\text{?}{y}_w+R\text{?}{z}_w+t\text{?}\right)\\ R\text{?}{x}_w+R\text{?}{y}_w+R\text{?}{z}_w+t\text{?}=y_n\left(R\text{?}{x}_w+R\text{?}{y}_w+R\text{?}{z}_w+t\text{?}\right)\\ R\text{?}{x}_w+R\text{?}{y}_w+R\text{?}{z}_w+t\text{?}=x_{n,p}\left(R\text{?}{x}_N+R\text{?}{y}_w+R\text{?}{z}_w+t\text{?}\right)\end{array}\right\}& \left[\mathrm{Equation}19\right]\end{array}$ $\text{?}\text{indicates text missing or illegible when filed}$

When geometrically analyzed, first two equations of [Equation 19] represent a straight line that passes through a point [x_n, y_n, 1] present on the normalized camera plane and a camera origin, and the last equation represents a plane that includes a projector origin and a straight line x=x_n,p (or y=y_n,p) on a normalized projector plane. A point p_intersect=[x_w, y_w, z_w] reconstructed in the 3D space may be an intersection point of the straight line and the plane.

Arranging [Equation 19], p_intersect (X_intersect in FIG. 7) may be expressed as a function for a projector parameter.

$\begin{array}{cc}{P}_{\mathrm{intersect}}={\left[\begin{array}{ccc}R\text{?}-x_{n}R\text{?}& R\text{?}-x_{n}R\text{?}& R\text{?}-x_{n}R\text{?}\\ R\text{?}-y_{n}R\text{?}& R\text{?}-y_{n}R\text{?}& R\text{?}-y_{n}R\text{?}\\ R\text{?}-x\text{?}R\text{?}& R\text{?}-x\text{?}R\text{?}& R\text{?}-x\text{?}R\text{?}\end{array}\right]}^{-1}\left[\begin{array}{c}{x}_{n}t\text{?}-t\text{?}\\ y_{n}t\text{?}-t\text{?}\\ x\text{?}{t}_{p3}-t_{p1}\end{array}\right]& \left[\mathrm{Equation}20\right]\end{array}$ $\text{?}\text{indicates text missing or illegible when filed}$

Also, since an Aruco marker ID of a calibration target is unique, an area on an image on which plane is located may be extracted from the camera. A phase (ψ_x, ψ_y) corresponding to a pixel (u, v) within an area in which a plane i is captured may be transformed to x_o, y_g, x_p (or y_p) through [Equation 5], [Equation 6], and [Equation 18]. Through this, p_intersect may be computed and since p_intersect is present on the plane i, equation of the plane i needs to be satisfied.

$\begin{array}{cc}\begin{array}{c}{n}_i{p}_{\mathrm{intersect}}+c_i=0,\\ n_i=R_{\mathrm{plane},i}\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right],c_i=-n_i·t_{\mathrm{plane},i}\end{array}& \left[\mathrm{Equation}21\right]\end{array}$

Therefore, an optimal parameter of the projector may be acquired through an optimization process of using (n_ip_intersect+c_i)² as a loss and using an intrinsic parameter and an extrinsic parameter of the projector as parameters.

$\begin{array}{cc}\underset{R,t}{\mathrm{minimize}}\sum _i\sum _{n,y}{\left\{n_i{p}_{\mathrm{intersect}}\left(u,v,\psi \left(u,v\right);R,t\right)+c_i\right\}}^2.& \left[\mathrm{Equation}22\right]\end{array}$

Here, n_i denotes an i-plane normal vector.

3D Reconstruction

A point cloud may be acquired through [Equation 20] using parameters of the camera and the projector optimized through the above process. Also, through [Equation 1], transformation from world coordinates to camera coordinates may be performed for all points in the point cloud and a depth map may be acquired for each camera-and-projector pair by storing a z value of the camera coordinates. However, due to a measurement error in the camera, an error may occur in the depth map. Therefore, pixel-wise confidences of all depth maps need to be computed and a process of integrating the point cloud based on the same needs to be performed.

To acquire a depth map pixel-wise confidence of a corresponding camera by using an arbitrary camera as a reference, depth d_ref(u, v) corresponding to each pixel (u, v) may be acquired and world coordinates of a corresponding point in the 3D space may be acquired and world coordinates of the corresponding point in the 3D space may be acquired. Also, a remaining camera excluding a reference camera may transform world coordinates of a corresponding point p(u, v) to each camera coordinate through [Equation 1]. Here, a camera j(ĭref) acquires values of pixel coordinates u_j and v_j corresponding to p(u, v) and then performs a visibility test using a depth map of the camera j.

Visibility v_j(p(u, v)) is 1 when the camera j compares an actual depth z_j of p(u, v) to d_j(u_j, v_j) acquired through the depth map of the camera j and, as a result, d_j(u_j, v_j) is larger, and is zero when d_j(u_j, v_j) is smaller, and thereby p is determined to not be visible from the camera j. For every camera j with v_j(p)=1, confidence C_ref,j(u, v) may be computed as follows.

$\begin{array}{cc}{C}_{\mathrm{ref},j}\left(u,v\right)=\exp \left\{-{\left(z_j-d_j\left(u_j,v_j\right)\right)}^2\right\}& \left[\mathrm{Equation}23\right]\end{array}$

FIG. 8 illustrates a confidence computation process of a depth map according to an example embodiment. Referring to (a) and (b) of FIG. 8, the closer an actual depth d₂′ between a 3D point generated from a depth map of a reference camera and another camera j to a depth map value of the camera j, the higher the confidence. Here, 810 represents reference camera coordinates and 820 represents camera coordinates.

Confidence C_ref(u, v) of a pixel (u, v) of the reference camera may be represented as follows for every camera j with v_j(p(u, v))=1.

$\begin{array}{cc}{C}_{\mathrm{ref}}\left(u,v\right)=\prod _j{C}_{\mathrm{ref},j}\left(u,v\right).& \left[\mathrm{Equation}24\right]\end{array}$

By going through a process of computing confidences for all depth maps and then integrating the same into one through the aforementioned process, the 3D reconstruction is finally completed. For arbitrary camera j, a depth and a confidence are given for each pixel of a depth map. Here, there may be a case in which a 3D point p_j acquired through the depth map is visible to the main camera and a case in which it is not visible. When it is visible to the main camera, depth and confidence (d, C) of p_j may be stored by the camera in an array L_main(u, v) for a pixel (u, v) corresponding to p_j. When this process is performed for depth maps of all cameras, a 3D point array may be given for each pixel (u, v) of the main camera. Accordingly, the depth map of the main camera is computed in a form of a weighted sum as follows.

$\begin{array}{cc}{D}_{\min }=\left(u,v\right)=\frac{\sum _{\left(d,C\right)\in L_{\mathrm{main}}\left(u,y\right)}C\times d}{\sum _{\left(d,C\right)\in L_{\mathrm{main}}\left(u,v\right)}C}& \left[\mathrm{Equation}25\right]\end{array}$

When p_j is not visible to the main camera, the side camera performs the aforementioned process and a confidence-based depth map fusion is finally completed.

FIG. 9 illustrates a target image, a depth map, and a confidence map according to an example embodiment.

Referring to FIG. 9, a view on the left represents a captured target image, a view in the middle represents a depth image from a corresponding camera, and a view on the right represents a confidence map. Starting from a first row, a main camera, a side camera 00 (east), a side camera 01 (west), a side camera 02 (south), and a side camera 03 (north) are shown.

It can be seen that depth and confidence variously appear due to a plurality of variables, including view vectors of the projector and the camera and an angle of reflected light.

FIG. 10 is a flowchart illustrating a 3D calibration method for multi-view phase shift profilometry according to an example embodiment.

Referring to FIG. 10, the 3D calibration method for multi-view phase shift profilometry performed by a computer device according to an example embodiment may include operation S110 of acquiring a phase that is data required for 3D reconstruction from parameters of at least one camera and a projector based on a PMP method, operation S120 of defining a phase-to-depth function for each camera-and-projector combination and performing calibration of optimizing the parameters of the camera and the projector, and operation S130 of acquiring a point cloud that includes depth information using the optimized parameters of the camera and the projector.

Also, the 3D calibration method for multi-view phase shift profilometry may further include operation S140 of performing transformation from world coordinates to camera coordinates for all points in the point cloud, storing a z value of the camera coordinates, and acquiring a depth map for each camera-and-projector pair, and operation S150 of computing pixel-wise confidences of all the depth maps and performing the 3D reconstruction by acquiring a final point cloud through a confidence-based depth map fusion that integrates the respective depth maps into a single depth map based on the confidences.

According to example embodiments, camera calibration may be performed without moving a single target and calibration of a camera and a projector may be simultaneously performed with a single target. Also, a phase-to-depth function through an inverse camera model of the projector may be defined.

Hereinafter, the 3D calibration method for multi-view phase shift profilometry according to an example embodiment is further described.

The 3D calibration method for multi-view phase shift profilometry according to an example embodiment will be described using a 3D calibration apparatus for multi-view phase shift profilometry as an example.

FIG. 11 is a block diagram illustrating a 3D calibration apparatus for multi-view phase shift profilometry according to an example embodiment.

Referring to FIG. 11, a 3D calibration apparatus 1100 for multi-view phase shift profilometry according to an example embodiment may include a phase acquisition unit 1110, a calibration unit 1120, and a point cloud acquisition unit 1130. Also, depending on example embodiments, the 3D calibration apparatus 1100 for multi-view phase shift profilometry may further include a depth map acquisition unit 1140 and a depth map fusion unit 1150.

In operation S110, the phase acquisition unit 1110 may acquire a phase that is data required for 3D reconstruction from parameters of at least one camera and a projector based on a PMP method.

In detail, the phase acquisition unit 1110 may acquire the phase required for the 3D reconstruction from a single camera-and-projector pair in such a manner that the projector projects a series of structured light images and projected light is reflected from an object and detected at a sensor of the at least one camera. Here, the phase acquisition unit 1110 may configure a main camera located at the top and a side camera located on at least one side surface, and may acquire the phase based on the PMP method using the projector present between the main camera and the side camera.

In operation S120, the calibration unit 1120 may define a phase-to-depth function for each camera-and-projector combination and may perform calibration of optimizing the parameters of the camera and the projector. Here, the phase-to-depth function relates to transforming the phase to a depth or a depth map.

That is, the calibration unit 1120 may perform a calibration of forming a phase-to-depth map with a camera parameter using a fixed target. In the prior art, calibration may be performed by performing shooting while physically turning a flat target several times. In the case of using this dynamically moving object, the phase-to-depth map of the PMP method may not be calibrated. The example embodiments may simultaneously acquire camera parameters through phase-to-depth map calibration of the PMP method by intentionally making a target static and by using the same to project images of various phases onto the target.

In operation S130, the point cloud acquisition unit 1130 may acquire a point cloud that includes depth information using optimized parameters of the camera and the projector.

In operation S140, the depth map acquisition unit 1140 may perform transformation from world coordinates to camera coordinates for all points in the point cloud, may store a z value of the camera coordinates, and may acquire a depth map for each camera-and-projector pair.

In operation S150, the depth map fusion unit 1150 may compute pixel-wise confidences of all the depth maps and may perform the 3D reconstruction by acquiring a final point cloud through a confidence-based depth map fusion that integrates the respective depth maps into a single depth map based on the confidences.

FIG. 12 is a diagram for explaining a depth map fusion according to an example embodiment.

Referring to FIG. 12, the depth map fusion unit 1150 may divide points in a 3D space that may be acquired through a depth 1240 of each pixel when computing a confidence 1250 for each pixel of a depth map into two cases: a case in which a point is visible to a main camera 1201 at the top and a case in which a point is not visible. That is, it may be classified depending on presence or absence of visibility 1260 through the main camera 1201.

For points visible to the main camera 1201, a depth map fusion may be performed through a weighted sum 1220 with a depth map 1210 of the main camera along with the corresponding confidence 1250. For points not visible to the main camera 1201, the depth map fusion may be performed through a weighted sum 1270 in a depth map 1280 of a side camera 1202. Through this, a final 3D point cloud 1230 may be acquired.

FIGS. 13A and 13B illustrate a final 3D point cloud according to an example embodiment.

FIGS. 13A and 13B illustrate the final 3D point cloud acquired by performing shooting for a calibration target and then completing a depth map fusion through addition of a depth map of a side camera to a depth map of a main camera.

As a result, calibration of a PMP system for 3D reconstruction is successfully performed with only one target through the example embodiment and it can be seen that the calibration and the 3D reconstruction may be performed with only one scene without performing additional shooting while moving a corresponding target. Also, the phase-to-depth function may be configured using only parameters of the camera and projector without additional parameters.

The aforementioned apparatuses may be implemented through hardware components, software components, and/or combination of the hardware components and the software components. For example, the apparatuses and the components described in the example embodiments may be implemented using one or more general-purpose or special purpose computers, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of responding to and executing instructions in a defined manner.

Although the example embodiments are described with reference to some specific example embodiments and accompanying drawings, it will be apparent to one of ordinary skill in the art that various alterations and modifications in form and details may be made in these example embodiments without departing from the spirit and scope of the claims and their equivalents. For example, suitable results may be achieved if the described techniques are performed in different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents.

Therefore, other implementations, other example embodiments, and equivalents of the claims are to be construed as being included in the claims.

Claims

1. A three-dimensional (3D) calibration method for multi-view phase shift profilometry, performed by a computer device, the 3D calibration method comprising: acquiring a phase that is data required for 3D reconstruction from parameters of at least one camera and a projector based on a phase measuring profilometry (PMP) method;defining a phase-to-depth function for each camera-and-projector combination and performing calibration of optimizing the parameters of the camera and the projector; andacquiring a point cloud that includes depth information using the optimized parameters of the camera and the projector.

2. The 3D calibration method of claim 1, wherein the acquiring of the phase comprises: projecting a series of structured light images using the projector;detecting projected light that is reflected from an object using a sensor of the at least one camera; andacquiring the phase required for the 3D reconstruction using a single camera-and-projector pair.

3. The 3D calibration method of claim 1, wherein the acquiring of the phase comprises configuring a main camera located at the top and a side camera located on at least one side surface, and acquiring the phase based on the PMP method using the projector located between the main camera and the side camera.

4. The 3D calibration method of claim 1, further comprising: performing transformation from world coordinates to camera coordinates for all points in the point cloud, storing a z value of the camera coordinates, and acquiring a depth map for each camera-and-projector pair; andcomputing pixel-wise confidences of all the depth maps and performing the 3D reconstruction by acquiring a final point cloud through a confidence-based depth map fusion that integrates the respective depth maps into a single depth map based on the confidences.

5. The 3D calibration method of claim 4, wherein the performing of the 3D reconstruction by acquiring the final point cloud comprises: performing a depth map fusion through a weighted sum with a depth map of the main camera along with a corresponding confidence for points visible to the main camera, with respect to points in a 3D space acquirable through a depth of each pixel when computing the pixel-wise confidences of the depth maps; andperforming the depth map fusion through a weighted sum in a depth map of a remaining side camera for points not visible to the main camera.

6. A three-dimensional (3D) calibration apparatus for multi-view phase shift profilometry, the 3D calibration apparatus comprising: a phase acquisition unit configured to acquire a phase that is data required for 3D reconstruction from parameters of at least one camera and a projector based on a phase measuring profilometry (PMP) method;a calibration unit configured to define a phase-to-depth function for each camera-and-projector combination and to perform calibration of optimizing the parameters of the camera and the projector; anda point cloud acquisition unit configured to acquire a point cloud that includes depth information using optimized parameters of the camera and the projector.

7. The 3D calibration apparatus of claim 6, wherein the phase acquisition unit is configured to acquire the phase required for the 3D reconstruction using a single camera-and-projector pair in such a manner that the projector projects a series of structured light images and projected light is reflected from an object and detected at a sensor of the at least one camera.

8. The 3D calibration apparatus of claim 6, wherein the phase acquisition unit is configured to configure a main camera located at the top and a side camera located on at least one side surface, and to acquire the phase based on the PMP method using the projector located between the main camera and the side camera.

9. The 3D calibration apparatus of claim 6, further comprising: a depth map acquisition unit configured to perform transformation from world coordinates to camera coordinates for all points in the point cloud, to store a z value of the camera coordinates, and to acquire a depth map for each camera-and-projector pair; anda depth map fusion unit configured to compute pixel-wise confidences of all the depth maps and to perform the 3D reconstruction by acquiring a final point cloud through a confidence-based depth map fusion that integrates the respective depth maps into a single depth map based on the confidences.

10. The 3D calibration apparatus of claim 9, wherein the depth map fusion unit is confiured to perform a depth map fusion through a weighted sum with a depth map of the main camera along with a corresponding confidence for points visible to the main camera, with respect to points in a 3D space acquirable through a depth of each pixel when computing the pixel-wise confidences of the depth maps, and to perform the depth map fusion through a weighted sum in a depth map of a remaining side camera for points not visible to the main camera.