Patent Application
20220318948
The present invention is related generally to an apparatus and a method for stitching images of a large scene or surface captured by a camera with unknown perturbed poses.
Image stitching techniques aim to fuse a collection of overlapped images to achieve a broad view of an object or a scene of interest with satisfactory resolution in applications such as Google earth mapping, video stabilization, panoramic image construction, drone image stitching, and 3D structure reconstruction, etc. In order to achieve high stitching precision required by various applications, it is necessary to compensate geometric distortions of collected images, in which camera pose estimation for each image is a crucial task, as even a tiny error in camera pose estimation may lead to obvious distortion and visual mismatch of the stitched image, and consequently restrict subsequent applications. Therefore, it is of great importance and daunting challenge to accurately estimate camera poses given collected images.
In recent decades, pairwise image alignment and stitching methods have been well studied as a generalized perspective-n-point (PnP) problem using matching feature points. In particular, for each pair of images to be stitched, SIFT-feature-based matching points of the image pair are utilized to estimate associated camera poses using a PnP method such that images can be stitched one by one. Pairwise image stitching methods generally work well for a small number of images with explicit matching points. However, when the total number of images is large, or feature points are not well matched, pairwise stitching methods may fail, since any false matching points may cause abnormal camera pose error, and this error will propagate to subsequent camera pose estimates used for stitching, finally deteriorate the image stitching performance. To solve this problem, global alignment approaches such as bundle adjustment have been explored to improve the robustness of stitching. The bundle adjustment method considers matching feature points between a target image and a bundle of its overlapping reference images, instead of just one reference image, to estimate the camera pose of the target image. Therefore, the overall robustness is improved due to the increase of matching points. However, the performance is still not quite satisfactory, especially for high precision stitching since those false matching points still inevitably cause camera pose errors. Accordingly, there is a need to develop novel system and method of image stitching using robust camera pose estimation for improving the performance.
In our proposed framework, we proposed a robust camera pose estimation method which can accurately estimate camera poses even with abnormal pairwise camera pose errors. In particular, given a collection of indexed images, we first construct relative camera pose matrices of all pairs of images. Each entry corresponds to a target and a reference image pair indexed by the row and the column, respectively. When the pair of images are overlapped to each other, the corresponding entry is achieved by solving a perspective-n-point (PnP) problem using matching feature points of the overlapped area, otherwise the entry is non-observable. For 6-degrees-of-freedom (6 DoF) camera poses, we construct 6 matrices, each matrix corresponding to one of the six degrees of freedom. As a global alignment approach, our proposed method aims to estimate camera poses based on the partially observed matrices of relative poses by exploiting the underlying rank-2 matrices of relative poses and the joint sparsity of camera pose errors in estimation. Numerical simulations demonstrate that our proposed method outperforms existing pairwise and global alignment baselines in stitching images.
Low-rankness and sparsity have been widely studied in recent years in exploiting the low-dimensional structure of signals, such as image restoration, signal processing, video processing, machine learning, and Euclidean distance matrix reconstruction. A well-known method to solve this type of problems is the robust principal component analysis (RPCA), which recovers a low-rank matrix and a sparse error matrix from their superposition observations. Inspired by the RPCA but differently, here we exploit the low-rankness in an implicit way rather than using an explicit low-rank constraint such as matrix nuclear norm. We also promote the joint sparsity of relative pose errors given partially observed matrices.
Accordingly, some embodiments of the present disclosure can provide contributions that can be summarized as follows:
1. This disclosure proposes a robust image stitching method which exploits the underlying rank-two matrices of relative poses and the joint sparsity of camera pose errors. The proposed method exhibits robust performance even under abnormal camera poses.
2. The disclosure implicitly describes the low-rankness and joint sparsity given partially observed matrices. The proposed method is potentially applicable to various scenarios targeting a sequence of linearly related estimations.
3. The disclosure provides numerical experiments using synthetic images captured by a camera with unknown perturbed poses. Our experiments demonstrated significant improvement in performance over other baseline methods.
Further, some embodiments of the present invention are based on recognition that an image forming system is provided for constructing a whole image of an object. The image forming system may include an interface configured to sequentially acquire images of partial area of the object, captured by a camera, wherein two adjacent images of the partial areas include overlap portions, wherein the sequential images correspond to a three dimensional (3D) surface of the object, wherein the geometrical information (geometrical parameters) of the object is provided, wherein an initial pose of the camera is provided; a memory to store computer-executable programs including a pose estimation method and a stitching method; and an image stitching processor configured to perform steps of: estimating poses with respect to the sequential images by the pose estimation method; projecting the sequential images into a 3D surface based on the camera pose estimations by solving a perspective-n-point (PnP); constructing a two-dimensional (2D) image of the object based on the estimated camera poses; and outputting the constructed 2D image via the interface.
Yet further, another embodiment may provide an image forming method for constructing a whole image of an object. In this case, the method comprises steps of: sequentially acquiring images of partial areas of the object captured by a camera via an interface, wherein two adjacent images of partial areas include overlap portions, wherein the sequential images correspond to a three dimensional (3D) surface of the object, wherein the geometrical information of the object is provided, wherein an initial pose of the camera is provided; providing computer-executable programs including a pose estimation method and a stitching method from a memory; and performing steps, using an image stitching processor: estimating poses with respect to the sequential images by the pose estimation method; projecting the sequential images into a 3D surface based on the camera pose estimations by solving a perspective-n-point (PnP); constructing a two-dimensional (2D) image of the object based on the estimated camera poses; and outputting the constructed 2D image via the interface.
The accompanying drawings, which are included to provide a further understanding of the invention, illustrate embodiments of the invention and together with the description serve to explain the principle of the invention.
Various embodiments of the present invention are described hereafter with reference to the figures. It would be noted that the figures are not drawn to scale elements of similar structures or functions are represented by like reference numerals throughout the figures. It should be also noted that the figures are only intended to facilitate the description of specific embodiments of the invention. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, an aspect described in conjunction with a particular embodiment of the invention is not necessarily limited to that embodiment and can be practiced in any other embodiments of the invention.
The image stitching system 100 may include a network interface controller (interface) 110 configured to receive images from a camera/sensor 141 and display 142 images. The camera/sensor 141 is configured to take overlapped images of the surface of interest 140.
Further, the image stitching system 100 may include a memory/CPU unit 120 to store computer-executable programs in a storage 200. The computer-executable programs/algorithms may include overlapped images 205, an image stitching module 210, which includes a robust camera pose estimation unit. The computer-executable programs are configured to connect with the memory/CPU unit 120 that accesses the storage 200 to load the computer-executable programs.
Further, the Memory/CPU unit 120 is configured to receive images (data) from the camera/sensor 151 or an image data server 152 via a network 150 and perform the displacement measurement 100 discussed above.
Further, the image stitching system 100 may include at least one camera that is arranged to capture images of the surface of interest 140, and the at least one camera may transmit the captured mages to a display device 142 via the interface.
In some cases, the at least one camera may be a high resolution digital optical camera. In this case, the sequentially acquired images of partial areas of the object are captured by the camera, the high resolution digital optical camera.
|M| and then form a matrix of relative pose {tilde over (L)}=[{tilde over (l)}(1), {tilde over (l)}(2), . . . , {tilde over (L)}(K)]∈
|M|×K then estimate camera poses by solving an optimization problem that exploiting joint sparsity and low-rankness as
With the optimal solution {circumflex over (P)}, camera poses are recovered by {circumflex over (p)}n={circumflex over (P)}n,:T−{circumflex over (P)}1,:T+p1, for n=1, . . . N. All images captured by the camera are projected back to the large surface for stitching process using interpolation.
In this work, we target the problem of stitching images of a large 3D surface with known geometry. Without loss of generality, we consider a collection of N images {Xn}n=1N covering a large surface U, for instance a huge painting surface, as illustrated in
We consider a six-degrees-of-freedom (6-DoF) pin-hole camera model in image acquisition. Based on this model, each pixel (xc, yc) in image Xn is projected from a point (xu, yu, zu) on the 3D surface U according to
where R=Rx(θx)Ry(θy)Rz(θz)∈3×3 is the rotation matrix of three degrees of freedom determined by Euler angles θx, θy, and θz, with Rx(⋅), Ry(⋅), and Rz(⋅) representing the rotation matrix around x, y, and z axes, respectively; T=[Tx, Ty, Tz]T∈
3 is the translation vector of translations in x, y, and z directions; f is the focal length of the camera; and v is a pixel-dependent scaling factor to ensure that the projected pixel lies in the focal plane. We note that the pose of the 6-DoF camera is determined by a vector p=[θx, θy, θz, Tx, Ty, Tz]T∈
K with K=6 in this specific problem.
For image stitching, an essential task is to estimate camera poses {pn}n=1N for all images {Xn}n=1N in the collection such that the projective transformation of surface points shown in (1) can be properly compensated. To simplify notations, we concatenate camera pose vectors {pn}n=1N as row vectors into a matrix P, and define the k-th column (k-th pose parameter) vector of P as hk, i.e.,
P=[p1,p2, . . . ,pN]T=[h1,h2, . . . ,hK]∈N×K. (2)
For the k-th parameter of camera poses, we define a relative camera pose matrix L(k)∈N×N with each entry
L
(k)(i,j)=pk(k)−pj(k)=hk(i)−hk(j), for i,j=1,2, . . . ,N;k=1,2, . . . ,K. (3)
where pi(k) denotes the k-th parameter of pi, and L(k)(i, j), the i-th row and j-th column entry of L(k), represents the k-th parameter of the camera pose associated with image xi relative to that associated with image Xj. We can rewrite the matrix L(k) in the form of vectors
as
L
(k)
=h
k1T−1hkT∈N×N, (4)
where 1 is an N-dimensional vector with all entries equal to 1. It is straightforward to demonstrate that for any non-zero vector hk≠1
we have
rank(L(k))=rank(hk1T)+rank(1hkT)=2. (5)
Motivated by the aforementioned rank-2 relative camera pose matrices and recent research work of robust low-rank matrix recovery, we aim to recover camera poses from relative camera pose matrices. To this end, we first construct relative camera pose matrices {tilde over (L)}(k)∈N×N using a traditional PnP method given the collection of images.
For the (i, j)-th entry of {tilde over (L)}(k), we seek to calculate the relative camera pose of the i-th image relative to the j-th image using a PnP method based on SIFT feature matching points in the overlapped region. For those pairs that are not overlapped, meaning no matching points can be found, their corresponding entries in {tilde over (L)}(k) remain missing.
Let M be a binary matrix indicating the observability of {tilde over (L)}(k) entries. M (i, j)=1 if {tilde over (L)}(k)(i, j) is observable, and M(i, j)=0 otherwise. Considering camera pose errors caused by false matching points, we model the observed noisy relative pose matrix as
{tilde over (L)}
(k)
=S
(k)
+L
(k)
,k=1, . . . ,K, (6)
where S(k)∈N×N represents sparse pose estimation errors. To introduce our proposed approach, we vectorize all observed entries of {tilde over (L)}(k) as
{tilde over (l)}
(k)
=vec({{tilde over (L)}(k)(i,j)|M(i,j)=1})∈|M|, (7)
In the same element order, similarly we have
s
(k)
=vec({S(k)(i,j)|M(i,j)=1})∈|M|, (8)
and
l
(k)
=vec({L(k)(i,j)|M(i,j)=1})∈|M|, (9)
Recalling (4), we can express l(k) as
l
(k)
=Ah
k, (10)
where
A=[ . . . ,αi,j, . . . ]T∈|M|×N, (11)
where (i, j) is the index that M(i, j)=1 holds. Vector αi,j∈N has all-zero entries except that αi,j(i)=1 and αi,j(j)=−1. We denote
{tilde over (L)}=[{tilde over (l)}(1),{tilde over (l)}(2), . . . ,{tilde over (l)}(K)]∈|M|×K, (12)
S=[s(1),s(2), . . . ,s(K)]∈|M|×K, (13)
and
L=[l(1),l(2), . . . ,l(K)]=AP∈|M|×K. (14)
Considering that different parameters of the camera pose (such as rotation angle θx and transition Tx are of different magnitude ranges, we introduce a weight vector w=[w1, w2, . . . , wK]T where
to normalize the magnitude ranges. Consequently, the camera pose estimation problem is formulated as
where W=diag{w} is a diagonal matrix whose diagonal is w, denotes ∥⋅∥F denotes the Frobenius norm, ∥⋅∥2.1 denotes the 2.1 norm of a matrix defined as ∥Q∥2.1=Σi=1I√{square root over (Σj=1J [Q(i, j)]2)} for any matrix Q∈
I×J, and λ is the regularization parameter.
Note that (15) is similar to the well-known RPCA in the sense that it decomposes a noisy matrix into a low-rank matrix and a sparse matrix. Nevertheless, here we implicitly impose low-rankness rather than use an explicit low-rank constraint such as nuclear-norm minimization. In addition, we promote joint sparsity across parameters of each pose using the 2.1 norm. This is based on the observation
that when one parameter of a camera pose is not well estimated, all other parameters associated the camera exhibit large errors.
To solve (15), we utilize the alternating minimization method by alternately updating S and P until its stopping criteria is satisfied. The sub-problem with respect to P is a standard least-squares problem, which can be solved by
p
(i)
=A
†({tilde over (L)}−S(i-1)), (16)
where A† denotes the pseudo-inverse of the matrix A and superscripts (t) and (t−1) represent iteration numbers.
The subproblem of S can be solved by row-dependent soft-thresholding as
where the subscript (i, :) denotes the i-th row of the corresponding matrix respectively, {circle around (⋅)} stands for the element-wise product.
We observe that when λ=0 and S=0, Algorithm 1 is reduced to the least-squares method for camera pose estimation
{circumflex over (P)}
LS
=A
†
{tilde over (L)}. (18)
Note that any global rigid translation on camera poses leads to the same relative camera pose matrix L. To avoid ambiguity, we take the first camera pose as a noiseless reference, with other camera poses calculated from the output {circumflex over (P)} of
{circumflex over (p)}n={circumflex over (P)}n,:T−{circumflex over (P)}1,:T+p1, for n=1, . . . N. (19)
Once we obtained estimated camera poses {pn}n=1N, we project all N images to the 3D surface U according to (1) to construct the final image using interpolation.
To validate our proposed jointly sparsity and rank-2-based global stitching approach, we simulated a sequence of N=50 images {Xn}n=1N of size 500×600 captured by a camera at different poses using model shown in (1), as illustrated in
To estimate camera poses, we consider four different methods for comparison: (1) pairwise method, (2) bundle adjustment method, (3) least-squares method, and (4) our proposed method. For the pairwise method, we utilize a SIFT-feature-based perspective-n-point (PnP) method on matching points of the image pair to be stitched with prior knowledge of the 3D scene geometry.
For the bundle adjustment baseline, we expand SIFT-feature matching points by including all those between the target image and its overlapping neighbors. Specifically, when stitching the i-th image, we consider the set of neighbors i={g|g<i, |g−i|≤25} for matching points. For the least-squares method and our proposed method, the mask M is given according to the valid number of overlapping neighbors by
The hyper-parameter in (15) is tuned as λ=0.01 for good performance.
N×N as an example, where the white area in the square matrix corresponds missing values. For comparison, in
As an example, we show one estimated sparse error matrix Ŝ(4) in
To further quantitatively analyze the performance of our camera pose estimation, we compute the relative estimation error
of different methods, where l(k)∈|M|×1 is defined in (9), {circumflex over (l)}(k) is the corresponding reconstruction, and M∈
N×N.
We proposed a robust camera pose estimation method for stitching a large collection of images of a 3D surface with known geometry. To address the issue of accumulating camera pose error using traditional PnP methods, we constructed a partially observed relative pose matrix for each parameter of camera poses, and decompose it into a rank-2 matrix of relative camera poses and a sparse matrix of camera pose errors by exploiting the joint sparsity of camera pose errors in estimating camera poses. Numerical experiments with images captured by a camera with random poses demonstrate that when abnormal error occurs in camera pose estimation using traditional PnP methods, our proposed method is still capable of detecting and reducing the estimation error, yielding robust camera pose estimation and significantly better stitching results.
The above-described embodiments of the present invention can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component. Though, a processor may be implemented using circuitry in any suitable format.
Also, the embodiments of the invention may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.
Use of ordinal terms such as “first,” “second,” in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements.
Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention.
Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.
The above-described embodiments of the present invention can be implemented using hardware, software, or a combination of hardware and software.
Also, the embodiments of the invention may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.
Use of ordinal terms such as “first,” “second,” in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements.
