Photographers, graphic artists, and others desiring to capture an image of a scene, person, device, or event can use a device such as a still camera, a video camera, a web-cam, or the like to record the desired image. At times, the field of view of the captured image is too small, either too narrow (horizontal deficiency), too short (vertical deficiency), or both, for the photographer or user. The photographer can take two or more images while panning across the desired scene and can combine the multiple images into a composite image, or panorama, of the entire scene.
The resultant images can be cut and pasted together, whether manually, electronically, or digitally to combine the separate images or pictures into a single image. However, slight variations in the separate images can cause the cut and paste single image to appear as a mosaic instead of a contiguous, single image of the scene. The variations in the separate images can be the result of differences in projection angle and/or motion within the scene between the times the images were captured.
Various techniques have been proposed for blending images together to form a single image. One such method for blending overlapping images together utilizes Laplacian and Gaussian pyramids. An algorithm-based technique for blending overlapping images together can also be used.
Additional techniques have been disclosed that utilize both blending and warping to process images. For example, a blended image can be distorted with warping. Alternately, an image can first be morphed and then the colors of the warped image can be blended.
Exemplary embodiments are directed to a method and system for blending images into a single image, including selecting two images having overlapping content; dividing the two images into strips; selecting a strip in each of the two images where the two images overlap each other; determining differences between the overlapping two strips; determining a line through the overlapping strips where the differences between the overlapping strips are minimized; and blending the two images together along the minimized line to create a single image.
An additional embodiment is directed toward a method for blending two images into a single image, including dividing two images into strips along a common plane; selecting a strip in each image where the two images overlap; determining a line through the overlapping strips where differences between the overlapping strips are minimized; blending the two images along the determined minimized line to create a single image; and warping the single image to minimize blurring along the blending line.
Alternative embodiments provide for a computer-based system for blending images into a single image, including a computer configured to divide two images having overlapping content into strips; select a strip in each of the two images where the two images overlap each other; determine pixel difference values between the overlapping two strips; determine a line through the overlapping strips where the sum of the pixel difference values between the overlapping strips are minimized; and blend the two images together along the minimized line to create a single image.
A further embodiment is directed to a system for blending images into a single image, including means for dividing two images having overlapping content into strips in at least one region of overlap; means for calculating difference values between the pixels of the two images in the at least one region of overlap; means for determining a cut line through the two images where the difference values are minimized; and means for blending the two images along the cut line to create a single image.
Yet a further embodiment is directed to a system for blending images into a single image, including a first computing module dividing two images having overlapping content into strips in at least one region of overlap; a second computing module calculating difference values between the pixels of the two images in the at least one region of overlap; a third computing module determining a cut line through the two images where the difference values are minimized; and a fourth computing module blending the two images along the cut line to create a blended single image.
Another embodiment provides for a computer readable medium encoded with software for blending images into a single image, wherein the software is provided for selecting two images having overlapping content; dividing the two images into strips where the two images overlap each other; selecting a strip in each of the two images; determining the differences between the overlapping two strips; determining a line through the overlapping strips where the differences between the overlapping strips are minimized; and blending the two images together along the minimized line to create a single image.
The accompanying drawings provide visual representations which will be used to more fully describe the representative embodiments disclosed herein and can be used by those skilled in the art to better understand them and their inherent advantages. In these drawings, like reference numerals identify corresponding elements, and:
Referring initially to
The system of
Exemplary embodiments are compatible with various networks 114, including the Internet, whereby the images can be downloaded across the network for processing on the computer 100. The resultant single images can be uploaded across the network 114 for subsequent storage and/or browsing by a user who is situated remotely from the computer 100.
One or more images are input to a processor 112 in a computer 100 according to exemplary embodiments. Means for receiving the images for processing by the computer 100 can include any of the recording and storage devices discussed above and any input device coupled to the computer 100 for the reception of images. The computer 100 and the devices coupled to the computer 100 as shown in
These processor(s) and the computing modules and/or software guiding them can comprise the means by which the computer 100 can select a strip in each of two images where the images overlap each other and can determine a cut line through the two strips where the images can be joined to form a single image. For example, separate means in the form of software modules within the computer 100 can control the processor(s) 112 for selecting the best strip in each image where the images overlap and for blending the images along a cut line in the strips for creating a single image of a scene. The computer 100 can include a computer-readable medium encoded with software or instructions for controlling and directing processing on the computer 100 for directing blending of images into a single image.
The computer 100 can include a display, graphical user interface, personal computer 116 or the like for controlling the processing of the classification, for viewing the classification results on a monitor 120, and/or for listening to all or a portion of sound signals associated with the images over the speakers 118. One or more images are input to the computer 100 from a source of images as captured by one or more still cameras 102, video cameras 104, or the like and/or from a prior recording of a scene or event stored on a medium such as a tape 107 or CD 108. While
Embodiments can also be implemented within the capture/recording devices 102, 104, and 106 themselves so that the images can be blended concurrently with, or shortly after, the images being recorded. Further, exemplary embodiments of the image blending system can be implemented in electronic devices other than the computer 100 without detracting from the features of the system. For example, and not limitation, embodiments can be implemented in one or more components of an image projection system, such as in a CD/VCD/DVD player, a VCR recorder/player, etc. In such configurations, embodiments of the image blending system can blend images prior to or concurrent with the display of the single image(s).
The computer 100 optionally accepts as parameters one or more variables for controlling the processing of exemplary embodiments. As will be explained in more detail below, exemplary embodiments can apply one or more control parameters to guide the image blending processing to customize the blending and warping of the images to create single images according to the preferences of a particular user. Parameters for controlling the blending and warping processes can be retained on and accessed from storage 122. For example, a user can select, by means of the computer or graphical user interface 116, parameters for establishing how smoothly, or seamlessly, exemplary embodiments should blend two images together to produce a single image of a scene. These control parameters can be input through a user interface, such as the computer 116 or can be input from a storage device 112, memory of the computer 100, or from alternative storage media without detracting from the features of exemplary embodiments. Single images blended by exemplary embodiments can be written into a storage media 124 in the forms of files, catalogs, libraries, and/or databases in a sequential and/or hierarchical format. The processor 112 operating under control of exemplary embodiments can output the results of the image blending process, including printed images, summaries, and statistics, to a printer 130.
The functionality of an embodiment for automatically blending images to create a single image can be shown with the following exemplary flow description:
Blending Images into a Single Image:
Referring now to
One strip from each of the two images is selected in step 204 where the images overlap. In
(R1−R2)2+(G1−G2)2+(B1−B2)2=Diff Value
A relative difference value for determining the image differences between strip1 (404) of image1 (400) and strip2 (406) of image2 (402) is calculated by first finding the sum of the squared differences of each of the red, green, and blue (RGB) color pixel values for each pair of corresponding pixels between the two strips and then adding together all such difference values for that pair of strips. The difference value can also be viewed as a mean squared error value of the degree of mismatch between the two images at their respective strips. Each strip is selected to have the same number of pixels so the mean squared color difference algorithm can be applied throughout each strip. Under this technique, the differences in the red, green, and blue pixel values for each pair of corresponding pixels in the two strips are squared and summed to produce the image pixel difference value representing the measure of difference between the two image strips, such as 404 and 406. By iteratively calculating the mean squared difference value between selected strips from the two images, exemplary embodiments can select a pair of strips, one from each image 400 and 402, where the image difference between the two strips is minimized and, accordingly, the overlap between the two images is the closest match at the selected strips. As can be seen by comparing strips 404 and 406, the strip of the images shown by these strips is very similar, although not identical, and is representative of an area within the image scene where the two images, 400 and 402, overlap. While the mean squared pixel difference value has been disclosed for selecting overlapping image strips 404 and 406, exemplary embodiments are not so limited, and any known technique for matching and/or comparing images can be utilized to select a strip from each of two overlapping images where the difference or mismatch in the overlap is minimized. For example, a pixel by pixel intensity or luminescent comparison between two sample strips can be utilized to select two overlapping and matching strips. The width of the strips can be reduced by the user to reduce the computational time and resources to perform the steps of strip selection. Similarly, the number of strips and the size of the image region to be divided into strips can be limited to control processing time and resources.
Referring now to
Referring now to
At step 208, represented by the figures of
The first step, step 300, for determining a line where the images should be cut for being combined into a single image is to calculate a difference value for each pixel pair between the two overlapping strips. In some embodiments, as discussed above, this step has already been completed during the process of determining the two overlapping strips to be selected for finding the best matching strip between the two images. For example, a mean squared difference algorithm can be used to determine a difference value between each pair of pixels in the corresponding overlapping strips 504 and 506. Next, at step 302, the pixels of the difference image shown in
At step 306, the next unassigned pixel with the greatest difference value is processed and mapped to the region 700 according to the adjacency of the pixel to a region. If the pixel is adjacent to region 701 and is not adjacent to any other region, the pixel is assigned, or mapped, to region 701. If the pixel is adjacent to region 702 and is not adjacent to any other region, the pixel is assigned to region 702. If the pixel is adjacent to both regions 701 and 702, the pixel is arbitrarily assigned to either region 701 or 702. For example, the dual adjacent pixels can alternately be assigned first to region 701 and next to region 702. If the pixel is adjacent to no other regions, a new region 706 is created within 700; and the pixel is assigned to the new region 706. If the pixel is not adjacent to either region 701 or region 702 but is adjacent to one of the aforementioned new regions 706, the pixel is assigned to the adjacent new region 706. Further, if the pixel is adjacent to two or more new regions 706, those regions are combined into a single new region 706.
If the pixel is adjacent to region 701 and is also adjacent to one or more new regions 706, then the pixel is assigned to region 701; and all the pixels in the other adjacent regions 706 are also assigned to region 701. Correspondingly, if the pixel is adjacent to region 702 and is also adjacent to one or more new regions 706, then the pixel is assigned to region 702; and all the pixels in the other adjacent regions 706 are also assigned to region 702. If the pixel is adjacent to both regions 701 and 702 and is also adjacent to one or more new other regions 706, then the pixel is arbitrarily assigned to either region 701 or 702, as discussed above. In this situation, all the pixels in the other adjacent regions 706 are assigned to the same region, 701 or 702, to which the pixel was assigned.
All remaining pixels in the difference image 606 of
The mapping of all of the pixels of the difference image 606 produces the image of
In another exemplary embodiment, a plurality of images exceeding two images can be blended into a single image. In this embodiment, two images at a time are selected for blending into a single image. The resultant single image is then reinput to the image blending process as one of two selected images, with a new, unprocessed image constituting the second selected image. This process continues until all images comprising the scene, event, or the like have been processed, resulting in a final single image blended from the plurality of images. In like manner, separate frames of a video sequence can be blended into a single image of the video.
In an alternate embodiment, represented by
To improve the fit between the two images along the cut line, the two images can be warped, or spatially bent or morphed, in the areas of disagreement to better align the pixels in the mismatch areas. Referring to
For each area of disagreement along the cut line that exceeds a threshold, exemplary embodiments can iteratively warp the images in the areas of disagreement along a plurality of common planes by a plurality of magnitude of warp until the minimum pixel disagreement is found. For example, the warping shown in
Although exemplary embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principle and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Number | Name | Date | Kind |
---|---|---|---|
5659489 | Baldur | Aug 1997 | A |
5991461 | Schmucker et al. | Nov 1999 | A |
6075905 | Herman et al. | Jun 2000 | A |
6078701 | Hsu et al. | Jun 2000 | A |
6097854 | Szeliski et al. | Aug 2000 | A |
6148118 | Murakami et al. | Nov 2000 | A |
6271847 | Shum et al. | Aug 2001 | B1 |
6351269 | Georgiev | Feb 2002 | B1 |
6359617 | Xiong | Mar 2002 | B1 |
6362832 | Stephan et al. | Mar 2002 | B1 |
6366699 | Kuwano et al. | Apr 2002 | B1 |
6373995 | Moore | Apr 2002 | B1 |
6381376 | Toyoda | Apr 2002 | B1 |
6392658 | Oura | May 2002 | B1 |
6411742 | Peterson | Jun 2002 | B1 |
6434265 | Xiong et al. | Aug 2002 | B1 |
6434280 | Peleg et al. | Aug 2002 | B1 |
6487323 | Bonnet et al. | Nov 2002 | B1 |
6496606 | Boutroux et al. | Dec 2002 | B1 |
6532036 | Peleg et al. | Mar 2003 | B1 |
6549651 | Xiong et al. | Apr 2003 | B2 |
6563529 | Jongerius | May 2003 | B1 |
6568816 | Mayer, III et al. | May 2003 | B2 |
6570623 | Li et al. | May 2003 | B1 |
6590574 | Andrews | Jul 2003 | B1 |
6592225 | Wagner et al. | Jul 2003 | B2 |
6628283 | Gardner | Sep 2003 | B1 |
6714689 | Yano et al. | Mar 2004 | B1 |
6717608 | Mancuso et al. | Apr 2004 | B1 |
6720997 | Horie et al. | Apr 2004 | B1 |
6762769 | Guo et al. | Jul 2004 | B2 |
6771304 | Mancuso et al. | Aug 2004 | B1 |
6813391 | Uyttendaele et al. | Nov 2004 | B1 |
6941029 | Hatori | Sep 2005 | B1 |
7006707 | Peterson | Feb 2006 | B2 |
7006709 | Kang et al. | Feb 2006 | B2 |
7035478 | Crandall et al. | Apr 2006 | B2 |
7085435 | Takiguchi et al. | Aug 2006 | B2 |
7096428 | Foote et al. | Aug 2006 | B2 |
7098914 | Katayama et al. | Aug 2006 | B1 |
7130490 | Elder et al. | Oct 2006 | B2 |
7142725 | Komiya et al. | Nov 2006 | B2 |
7197192 | Edwards | Mar 2007 | B2 |
7373017 | Edwards et al. | May 2008 | B2 |
7375745 | Rai et al. | May 2008 | B2 |
7424218 | Baudisch et al. | Sep 2008 | B2 |
7529429 | Rother et al. | May 2009 | B2 |
7535497 | Ouchi | May 2009 | B2 |
7593854 | Belrose | Sep 2009 | B2 |
7620909 | Park et al. | Nov 2009 | B2 |
7653261 | Blake et al. | Jan 2010 | B2 |
20030076406 | Peleg et al. | Apr 2003 | A1 |
20030235344 | Kang et al. | Dec 2003 | A1 |
20040057633 | Mai et al. | Mar 2004 | A1 |
20050226531 | Silverstein et al. | Oct 2005 | A1 |
20060072851 | Kang et al. | Apr 2006 | A1 |
20060072852 | Kang et al. | Apr 2006 | A1 |
Number | Date | Country | |
---|---|---|---|
20050226531 A1 | Oct 2005 | US |