Patent Application
20190068876
The present invention relates to the stitching of images taken by more than one camera into a single high-quality image.
This section is intended to provide a background or context to the invention disclosed below. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived, implemented, or described. Therefore, unless otherwise explicitly indicated herein, what is described in this section is not prior art to the description in this application, and is not admitted to be prior art by inclusion in this section.
The present invention belongs to the category of “stitching” and streaming images for panorama-, and 3D-360°/Virtual Reality/Augmented Reality-based applications, where the content is captured from multiple cameras which have overlapping content. The present invention is related to the generation of high-quality stitching from multiple images. Optical flow from the overlap region of the cameras is computed and used in such a process.
The process of stitching 2D- and 3D-360° videos at reasonably high quality can be a computationally intensive and time-consuming process. The present invention proposes an optimization to the process of stitching 2D- and 3D-360° videos from multi-camera constellations with some overlap, and could benefit live-streaming use cases and offline-content workflows requiring a fast turnaround.
Multiple cameras in a certain capture configuration are used to capture 360-degree content, which is subsequently stitched to form a stereo panorama. Regular geometric-based stitching methods, such as affine or homography, generate stitching artifacts, when the scene has depth variations in an overlap region where images being stitched together have common subject matter. In order to create stitching of better quality, depth variations in the overlapped regions should be handled carefully. The following are two state-of-the-art best practices for generating high quality stitching.
The first is a fast method using blending, such as multi-band. This method gives rise to artifacts in the seam region, especially when the seam region includes content with varying depth.
The other method is a compute-intensive method, which involves computing optical flow/disparity vectors and performing a view synthesis of the overlap region from disparity and native camera-images. This method gives results superior to those obtained with the fast method.
The fast method is only good for mono-panorama; in the case of stereo-panorama, even though stitching seams are placed at the same position, while the objects in the scenes are shifted, artifacts for the most part remain high. The compute-intensive method generates the best quality stitching, but is not suitable for many applications, such as live broadcasting of virtual reality (VR), etc.
The present invention is intended to address these issues, and to provide an alternate method of aligning and stitching images without the disadvantages noted above.
This section contains examples of possible implementations and is not meant to be limiting.
In an exemplary embodiment, the present invention is a method which comprises obtaining a first image and a second image, wherein the first image and the second image overlap with one another in a seam region, and are to be stitched together at said seam region to form a combined image.
Optical flow for overlapping portions of the first image and the second image is computed, and the overlapping portions of said first image and said second image are divided into a plurality of segments.
A representative maxflow for each of the segments of the overlapping portions of said first image (maxflow_1) and said second image (maxflow_2) is computed.
Convergence limits, convLow and convHigh, of a final convergence (finalConv) are computed from maxflow_1, and flow convergence using maxflow_1 and maxflow_2 are also computed, the convergences being flowConv1 and flowConv2, respectively. The flow consistencies for maxflow_1 and maxflow_2, and the convergence consistencies for flowConv1 and flowConv2 are checked.
The average (flowConv) of flowConv1 and flowConv2 is calculated. When flow and convergence consistencies are satisfied, whether flowConv is greater than convLow is checked. When flowConv is greater than convLow, finalConv is set to be equal to flowConv, and finalConv is used for stitching the first image to the second image at said seam region to form the combined image.
On the other hand, when flow and convergence consistencies are not satisfied, or when flowConv is not greater than convLow, computing convergence (distConv) is computed using a distance-based approach. Then, whether distConv is greater than convLow and less than convHigh is determined.
When distConv is greater than convLow and less than convHigh, finalConv is set to be equal to distConv. Otherwise, when distConv is not greater than convLow and less than convHigh, finalConv is set to be equal to convLow.
In another exemplary embodiment, the present invention is an apparatus which comprises a first camera for obtaining a first image, and a second camera for obtaining a second image. The apparatus further includes at least one processor, and at least one memory including computer program code. The at least one memory and the computer program code are configured, with the at least one processor, to cause the apparatus to perform the method described in the preceding paragraphs.
In yet another exemplary embodiment, the present invention is an apparatus having means for performing the various steps in the method described in the preceding paragraphs.
In still another exemplary embodiment, the present invention is a computer program product comprising a non-transitory computer-readable storage medium bearing computer program code embodied therein for use with an apparatus. The computer program code comprising code for performing the various steps in the method described in the preceding paragraphs.
In the attached Drawing Figures:
In brief, the present invention provides an alignment method in which objects in the overlap regions of two images are aligned based on a convergence/scaling function of the images from their centers. Optimal convergence leads to better alignment of the objects at the overlap region, and the method for computing optimal convergence is disclosed. The convergence is computed by aggregating information from multiple fronts using approaches, such as pixel similarity/distance, optical flow, and a combination of both.
This description will begin with an introduction to convergence and to the terminology which will be used in the discussion to follow. Convergence is a function which scales an image coordinate by a given amount, known as the convergence value. The scaling is done for all pixel coordinates with respect to an image center, which serves as the origin of the scaling.
If c(x,y) is the convergence at point (x,y), x,y being the coordinates of a point in one of the two images being stitched, then an output image, I_out, is computed using c(x,y) and input image I_in (x,y) by using the following expressions:
x
i
=x−cx
y
i
=y−cy
where cx,cy are the coordinates of the center points of a fish-eye image, and (xi,yi) are normalized image coordinates, that is to say, xi,yi are the coordinates relative to the image centers, which serve the origins of the images.
Then,
dx=x
i
*c(x,y)
dy=y
i
*c(x,y)
where dx and dy are the changes in the coordinates of the input image points (xi,yi) following application of the convergence.
Finally, the coordinates of the corresponding point in the output image are obtained from:
x
o
=dx+cx
y
o
=dy+cy
Thus, an input image is transformed to an output image using a scaling factor; that is
I_out(xo,yo)=I_in(xi,yi)
Objects in the overlapping seam regions of the images must optimally be at a farther depth, that is, located far away from the camera, for stitching to be successful. Convergence scaling is used to align images at different depths for scaling.
Since objects at different depths in the seam region are not usually aligned in stitching, the convergence value is varied across the seam region, in the horizontal and vertical directions, to align the objects.
In the discussion to follow, the following abbreviations will be used:
Having provided this background material, the present invention may be summarized as a hybrid approach for computing the convergence of a seam region, in particular, segment regions of a seam region, by using (a) convergence estimates from an SSD-based search and (b) convergence estimates from optical flow, to arrive at a final convergence value towards image alignment for stitching.
The convergence estimate from optical flow is further verified using a flow-confidence criterion. The flow confidence criterion basically checks the consistency of the optical flow between the seam region segments from the left and right cameras. That is, in a segment of the seam region, the similarity of the foreground region flow is computed from left-image and right-image regions. This is referred to as Consistency-1.
The convergence estimates from both optical flows, right-image and left-image, are verified to lie between pre-defined limits. These are convLow and convHigh, which are minimum and maximum values of the convergence. This is referred to as Consistency-2.
If Consistency-1 and Consistency-2 are satisfied for a given segment of a seam region, then the final convergence for that segment is computed as the average of the flow-based convergences. Otherwise, the convergence is estimated by using an SSD distance-based method.
The convergence estimate from the SSD distance-based method is further verified to lie within a range of convergence values between convLow and convHigh, whereas the convergence range is computed using the maxflow for that segment in the seam region. These limits, convLow and convHigh, are empirically tuned and calculated. In the implementation corresponding to the present invention, they are computed using a linear interpolation function from the given maxflow value as follows:
If this last estimate fails, then the final convergence (finalConv) is the same as the minimum convergence boundary value (convLow) computed from the max flow for that segment.
The preceding paragraphs will now be discussed in greater detail with reference being made to the figures, of which
Referring first to
For every image, for the seam region, and for a segment, that is, a vertical division (strip) of the seam region, a representative maxflow is computed, as indicated in block 12, where reference is made to
Using maxflow_1, the convergence limits are computed. These are convLow and convHigh, which denote the lower and upper bounds of the final convergence. Let finalConv=flowConv be an initialization value for the final convergence.
The first part of the determination of flow convergence consistency is carried out as follows: the convergence estimate from all the optical flow representatives, maxflow_1, maxflow_2, and so forth, is computed as follows:
flowConv1=(x1+maxflow_1−cx1)/(x1−cx1)
flowConv2=(x2+maxflow_2−cx2)/(x2−cx2)
where
flowConv1 is the flow convergence estimate for the segment of image1,
flowConv2 is the flow convergence estimate for the segment of image2,
x1 and x2 are the x-coordinate averages in the segment, and
cx1 and cx2 are the x-coordinate image centers of two overlapping cameras.
If (|(flowConv1−flowConv2)|<flowconvergence_diff_threshold), then consistency_1_enable=true (or yes). The flowconvergence_diff_threshold is computed empirically and is equal to 0.035 in the implementation corresponding to the present invention.
The second part of the determination of flow convergence is carried out as follows: the consistency between the optical flows of the overlapping region is computed as follows:
If (|(maxflow_1−maxflow_2)|<flowConsistency_threshold), then consistency_2_enable=true (or yes), and the convergence estimate from flow is reliable. These two consistency checks are represented in block 14 in
If the two parts of the determination of flow convergence are true (or yes), then the final convergence (finalConv) is the average of the flowConv1 and flowConv2 (flowConv), as indicated in block 18 of
However, if at least one part of the determination of flow convergence consistency is not true, then the convergence estimate using a distance-based search (distConv) is computed and evaluated for consistency using the flow information, as indicated in block 20 of
The distConv is computed as follows: let images I1 and I2 be images from cameras C1 and C2. In general, images I1 and I2 are images captured by two cameras having a common overlap region.
The seam regions from images I1 and I2 are considered, and a set of convergence values (N) are applied to these seam regions or strips. The resulting strips are denoted by I1y and I2y. This step generates N strips corresponding to each strip I1y and I2y.
The following equation is used to compute the convergence which gives minimum SSD between the strips.
SSD-based search: argmin(Σ((I1y(x)−I2y(x)))2)
The convergence obtained by distance-based convergence is then validated against the convLow and convHigh bounds, at block 20, as follows:
The present method has the advantage of being independent of calibration information, and inherits the best properties of both distance-based convergence and flow-based convergence methods and hence produces better results than either of these methods. All of these benefits are applicable to the mono scopic panorama case and generate improved results with respect to not using convergence at all.
Camera 1340 and Camera 2350 provide input to the apparatus 300. A stitched image obtained by combining the images obtained by Camera 1340 and Camera 2350 in accordance with the process described above is sent to the output 360.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as an apparatus, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit”, “module”, or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon.
Any combination of one or more computer-readable medium/media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium does not include propagating signals and may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, and so forth, or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language, such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer, for example, through the Internet using an Internet Service Provider.
Aspects of the present invention are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable-medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of any and all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of various embodiments has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
Various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings. However, any and all modifications of the teachings of this disclosure will still fall within the scope of the non-limiting embodiments of this invention.
Although described in the context of particular embodiments, it will be apparent to those skilled in the art that a number of modifications and various changes to these teachings may occur. Thus, while the invention has been particularly shown and described with respect to one or more embodiments thereof, it will be understood by those skilled in the art that certain modifications or changes may be made therein without departing from the scope of the invention as set forth above, or from the scope of the claims to follow.
