Patent Application
20100265313
Not Applicable
Not Applicable
Not Applicable
A portion of the material in this patent document is subject to copyright protection under the copyright laws of the United States and of other countries. The owner of the copyright rights has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the United States Patent and Trademark Office publicly available file or records, but otherwise reserves all copyright rights whatsoever. The copyright owner does not hereby waive any of its rights to have this patent document maintained in secrecy, including without limitation its rights pursuant to 37 C.F.R. §1.14.
1. Field of the Invention
This invention pertains generally to camera devices, and more particularly to automatic generation of panoramic images.
2. Description of Related Art
Panoramic photography provides image capture with elongated fields of view, and is sometimes referred to as wide format. In one sense an image that presents a field of view which approaches or is greater than that of the human eye, which is about 160° by 75°, can be termed panoramic. Although the term “panoramic” can be generally used for various forms of wide (or long) format images, including those cropped to a wide or long format, the discussion herein uses the term to mean photos which are extended in one or more directions (e.g., horizontally or vertically) beyond that captured by the image sensor.
Although cameras have been available which utilize multiple lenses and shutters, it will be appreciated that these are expensive, bulky and are not readily available to even professional photographers, and certainly not to the average photographer.
The creation of panoramic images using conventional cameras presently involves a process of capturing a number of subject images with fixed exposure and appropriate overlap/alignment and then either cutting pieces of physical images to fit together in a panorama or doing the same thing in a photo editor. In either case it is difficult and laborious to achieve quality results.
People tend to consider the use of panoramic imaging as they stand before a stretching panorama extending vertically or horizontally, which cannot be captured with a single image. As noted traditional methods of creating a panorama require a lot of user effort and may not assure a good panoramic picture. Another problem arises in that the photographer can not be sure during the process of capturing the images whether or not the images so captured will provide the desired panorama when assembled. The user may capture images which are not suitable for creating a panorama, such as forgetting to fix the focus, not staying in the fixed plane of the panorama, failing to overlap the shots sufficiently, tilting the camera off-plane of the panorama, and so forth. When the user finally sits down to work on these images it will be too late to correct these issues by obtaining additional images.
Accordingly a need exists for a system and method of automatically generating high quality panoramic images without user effort. These needs and others are met within the present invention, which overcomes the deficiencies of previously developed panoramic techniques.
The present invention is a camera and method of automatically generating panoramic images within a camera in response to the capturing of a plurality of images. One of the keys of the present invention is that the acquisition of the images by the camera is directed to the panoramic object and/or coupled to panoramic processing. The camera itself not only automatically processes subject images into a panorama but assures that sufficient overlap is obtained between sequential images being captured.
In one embodiment of the invention, once panoramic mode is selected the photographer only need hold down the shutter as they pan (e.g., vertically, or horizontally) across the subject. The camera automatically assures that the proper fixed focus is maintained across the shot, along with sufficient overlap, and assures sufficient frames. All the user has to do with the camera is to pan the image while taking the “snapshot”.
This invention provides a solution for any users, from novice to expert, allowing them to create high quality panoramic pictures with little effort or no additional effort in response to a method which utilizes the combination state-of-art CMOS sensor and digital image processing techniques to directly generate panoramic images.
The following terms are generally described in relation to the specification, and are not to be interpreted toward constraining specific recitations of the specification.
The term “pan” as recited herein applies to the process of moving the camera across the spatial extent of the desired image to be captured, whether that arises in response to a horizontal motion (traditional pan) or vertical motion (traditional tilt), or a combination of vertical and horizontal motion.
The term “register” has numerous definitions, and as recited herein describes the detection or estimation of camera motion as “registering camera motion”, and is also used in describing the alignment of adjacent images in creating the panorama. It should be appreciated that in the image processing arts the term “register” is often used more restrictively to connote only image registration processes.
The invention is amenable to being embodied in a number of ways, including but not limited to the following descriptions.
One embodiment of the invention is an apparatus for capturing panoramic images, comprising: (a) means for capturing a sequence of (still, video, or combination of still and video) digital images (e.g., at least one imager and focus control) while panning the camera in any desired direction or combination of directions; (b) a computer with memory coupled to the computer; and (b)(i) programming adapted for execution on the computer for, (b)(ii) capturing image frames within a sequence of pair-wise overlapping image frames as the camera is panned across a desired panoramic view to be captured in a panoramic image, (b)(iii) registering (detecting/estimating) panning motion, (b)(iv) controlling the capture of each sequential frame within the sequence of pair-wise overlapping image frames in response to detected panning motion to assure proper pair-wise overlap without gaps between the image frames, and optionally selecting a subset of overlapping image frames as desired, and (b)(v) combining the sequence of pair-wise overlapping image frames to create the panoramic still photograph for output by the camera.
It should be noted that the panoramic image extends further in at least one dimension (e.g., horizontal, vertical, diagonal, arbitrary, 2D (vertical and horizontal area), and so forth) than a conventional, non-panoramic, image captured by the apparatus. Thus, the panoramic image can span any desired spatial area of a subject.
Panoramic images can be created without the user fussing with getting a proper image or performing a manual stitching and blending operation, as the apparatus can perform all of these steps automatically. Panoramic output can be generated in any desired format for communicating data to an external electronic device configured for printing, storing and/or communicating the data.
The apparatus according to the present invention can support any desired user interface which is adapted to allow the user to select panoramic image capture, and optionally select characteristics for altering the capture and image combining process.
Registration of panning motion within the apparatus can be performed in response to optical sensing, physical motion sensing, or a combination thereof.
When combining the images, programming corrects pixels at the interface between adjacent images, thus blending the images to reduce any appearance of a “seam”.
Super-resolution processing techniques can be utilized on image frames, prior to or after creation of one or more panoramic images to enhance resolution of the results.
One embodiment of the invention is a camera configured for automatically creating panoramic still images in response to panning a desired subject, comprising: (a) an electronic imaging element within a camera adapted for capturing a sequence of digital images; (b) a computer with memory coupled to the computer, the computer configured for controlling the electronic imaging element of the camera; and (c) programming adapted for execution on the computer for, (c)(i) capturing a sequence of pair-wise overlapping image frames within the electronic imaging element as the camera is panned across a desired panoramic view to be captured in a panoramic image, (c)(ii) controlling image frame overlap during capture, or selecting a set of sequential overlapping image frames, in response to detecting panning motion to assure proper pair-wise overlap without gaps arising between image frames, and (c)(iii) combining the set of sequential overlapping image frames to create the panoramic still photograph for output by the camera.
One embodiment of the invention is a method of automatically capturing a panoramic still photograph within a camera, comprising: (a) capturing a sequence of images spanning a desired subject area being panned; (b) assuring that the edges of adjacent images in the sequence overlap one another in response to capturing sufficient images based on camera motion as the desired subject area is being panned; (c) combining the sequence of images to create at least one panoramic still photographic image.
The present invention provides a number of beneficial aspects which can be implemented either separately or in any desired combination without departing from the present teachings.
An aspect of the invention is a method and apparatus for automatically creating panoramic images in response to combining a plurality of images captured during panning while in a panoramic mode.
Another aspect of the invention is that it allows panoramic images to be automatically created in response to collection of a sufficient number of image frames which span a desired subject area.
Another aspect of the invention is to allow average “point-and-shoot” photographers to create desired panoramic effects.
Another aspect of the invention is to allow photographers to create panoramic image results without the need of additional equipment or process steps.
Another aspect of the invention is the ability to create panoramic images without the need of highly precise camera equipment and control mechanisms.
Another aspect of the invention is that it allows the photographer to create a panoramic image with the camera assuring a proper overlap of adjacent images.
Another aspect of the invention is a method that can be integrated within digital still and/or video camera devices.
A still further aspect of the invention is that it can be applied to numerous forms of image collecting and processing devices.
Further aspects of the invention will be brought out in the following portions of the specification, wherein the detailed description is for the purpose of fully disclosing preferred embodiments of the invention without placing limitations thereon.
The invention will be more fully understood by reference to the following drawings which are for illustrative purposes only:
Referring more specifically to the drawings, for illustrative purposes the present invention is embodied in the apparatus generally shown in
A camera apparatus according to the present invention utilizes advanced digital image sensing (e.g., CMOS image sensors), and advanced computational chips in cameras that provide sufficient processing power for performing advanced image processing to generate panoramic images. The camera allows even novice photographers to create a panoramic photographic image spanning any desired area of the subject. The method of creating the panorama applies multi-frame processing to go beyond the limitations of existing digital cameras to provide automated capture and processing so that panoramic images can be output.
Prior to combining the images from the input image sequence, it is preferred in some cases that the programming of the invention compensate for non-panning motion and/or tilting prior to combining the set of sequential images into said panoramic image. For example algorithms can be executed to reduce the introduction of camera shake into the panorama, such as by correcting images or eliminating selected captures if sufficient other images exist from which to generate the panorama. If a rectangular panorama is to be created then portions of the input images can be cropped prior to combination of the images. This image cropping can be according to any desired shape, for example according to lens geometry, and so forth. It should be appreciated that any form of preprocessing and combination thereof can be additionally performed without departing from the teachings of the present invention.
By way of example and not limitation, one of the methods for “stitching together”, or combining, the overlapping images frames is by a matching and blending process having the following general steps. (1) Matching the overlapping portion of a first frame with the second frame, toward optimizing some matching criterion in the overlapped area (e.g., minimizing pixel error). Some examples include the following: minimizing mean-squared or mean-absolute error; maximizing cross correlation, normalized cross correlation or phase correlation; estimating the optical flow between the images; fitting a parameterized global model to the results of matching according to the aforementioned criteria; or applying the optical flow equations constrained according to a desired global model. It should be appreciated that one or more of the frames may be adjusted in size or lighting and so forth toward assuring a proper match. It will be noted that the positioning of the overlap takes into account off-axis displacements, such as vertical displacements or rotation in a horizontal panoramic pan, to assure matching. (2) Blending of the pixel areas on one or both sides of the “seam” to assure no visible discontinuity. Preferably, the blending process includes a random error diffusion so as to further obscure the area of the seam. Alternatively, the blending process can optimize various image criteria near the seams, such as smoothness or similarity in gradients with the input images. It will be appreciated by one of ordinary skill in the art that a number of techniques can be utilized for blending pixels without departing from the teachings of the present invention.
The motion of the camera can be registered, such as detecting motion and the rate of that motion, in a number of different ways without departing from the present invention. In at least one implementation, sequential image frames are compared to determine the amount of overlap and thus the relative camera motion for the panorama. In one variant, a low resolution image sensor or a low resolution mode of the normal image sensor, is utilized for quickly obtaining sufficient detail to ascertain a pan distance between frames, such as by determining how much matching exists, and thus overlap, between frames. In at least one other embodiment, the motion of the camera is physically registered, such as in response to an acceleration sensor temporally processed to estimate camera motion in any desired axis of motion. Alternatively, gravimeter, inertial sensing, and so forth may be used such as in combination with motion analysis software so that images are captured for the panorama based on the panning speed of camera. Use of physical sensing allows for optimizing the image collection rate for a given panning rate. In other embodiments a combination of physical motion sensing and optical sensing are used for determining camera motion.
In one mode of the invention the apparatus provides the user with options for controlling how the images are combined, and/or cropped. In one mode the user can select rectangular format (or otherwise set pan limits to horizontal and/or vertical) in which the programming combines the images in a horizontal or vertical direction and automatically crops away image portions that would lie outside of a resultant rectangular viewing area (e.g., producing an image shaped as in
If sufficient image overlap is provided in the series of images, then the programming can elect to eliminate one or more captured images from the sequence of images, such as those which suffer from one or more image “issues”. For example these issues may include the shaky hand of the photographer so that one or more of the images exhibit more blur than the other images. Other artifacts may show up during a set of images, such as a camera flash in the field of view, a bug flitting across the image sensor, and other anomalies which would adversely impact the quality of the resultant panorama.
It should also be appreciated that the teachings of the present invention can provide for generating a panoramic image spanning any arbitrary direction or path. For example, embodiments of the present invention can be configured to generate a panoramic image in response to the user holding the shutter down while circumscribing a serpentine pattern (e.g., S-shaped). In one mode of the invention the camera can even allow the user to intentionally tilt the camera during image panning to provide a mosaic tiling effect which is popular in certain instances.
In an advanced embodiment, the technique herein allows the user to even capture both vertical and horizontal panorama, such as to effectively increase camera resolution. For example a geologist wanting to examine strata at a site could select a panoramic mode and then pan left to right, then down, and right to left, and down, then back left to right, and so forth in any pattern effective toward covering a given area. The camera according to the present invention can provide a very large (high resolution) still which spans the subject and allows the user to magnify these as desired to view the minor details.
In one embodiment, a mode is provided that allows the user to select a tiled panoramic output. For example, in the above instance the huge area panorama created could be very unwieldy (e.g., >20-1000 MB) for transferring, viewing and so forth. A tiling mode thus allows the user to select a tiled mode in which the image is divided into images of a selected size which fit perfectly with another, because they are tiled after the panorama is generated. Programming in one or more embodiments allows the user to output a scaled down panorama, (e.g., fitting one normal image space) such as for previewing the panorama or for categorizing purposes. In one aspect, the scaled down panorama is output with an overlying grid in which the position of each of the underlying images is shown, with or without indicia (e.g., file name).
Embodiments of the present invention can be configured to generate the panoramic images in response to the use of many different user interface configurations. In one very basic configuration, the user simply selects panoramic mode, such as through pushing a button or moving a selector, then holds the shutter down as they pan the desired area from which a panoramic image is to be produced. More sophisticated user interfaces allow the user to select aspects of how the capturing is to be performed, the tradeoffs between image resolution and frequency of capture, the amount of stitching to be performed (e.g., ratio of shot overlap), the blending methods and parameters for seamlessly blending the images together (or selecting to allow the seams to be seen, as desired), whether correction of tilt is to be performed (e.g., seen going from
Embodiments of the present invention can also provide a choice of panoramic image outputs to the user. In one mode the user is shown a series of panoramic images created from the same input image sequence. In another mode, the user is shown one panoramic image and can select that the camera show them additional constructions of the panorama. As there exist different ways of cropping, blending and altering resolutions, these mechanisms give the user a set of automated choices. The user can elect to save one or more of these results, which are generated from the same sequence of images. In at least one implementation, advanced controls allow the user to specify a priority on how the different panoramic forms are to be presented for their selection, or whether they want that feature activated at all.
It will be appreciated that the panning motion can be registered by utilizing a physical sensor, by optical sensing, or a combination thereof. It should be appreciated that in this embodiment the video frames may be analyzed to determine the panning motion since the prior video frame and thus the panning speed as well as the timing on when the next still image frame is to be captured. In step 36 the still image frames are combined using the video image frames to aid in stitching together the still image frames. When utilizing video frames having significant overlap, it is relatively easy to align (register) the video frames. This registration information can be accumulated for the video frames captured at times between the two still images, until the registration of the two images becomes readily available. The approximation provides initial conditions for the precise registration of the two still images, reducing registration search space and hence complexity.
Alternative embodiments of the invention also allow for multiple sets of images to be combined into multiple panoramic images which are then combined using super-resolution techniques into a final panoramic image. For example consider frames n1 through n8 collected during a single pan. Overlapping frames n1, n3, n5 and n7 are then selected as part of a first set of frames, while frames n2, n4, n6 and n8 are selected as part of a second set of frames. The first set is combined into a first panoramic still, and also the second set into a second panoramic still. Both panoramas, covering substantially the same region, are then combined using super-resolution into a single higher resolution panoramic output. One benefit of this reversed-order approach is that the super-resolution process itself provides for obscuring the seams in the separate panoramic images. It should be appreciated that this aspect may be combined with any implementation taught herein and variations thereof.
It should be noted that the term “aspect ratio” refers to the width of a picture (or screen) in relation to its height. Ratios are typically expressed in the form “width×height”. For example, a 4×3 (more commonly expressed 4:3) ratio means the picture is 4 units wide by 3 units high. Actual physical size (number of pixels) of the picture is irrelevant for aspect ratio calculation, as the aspect ratio refers only to the relationship between width and height. A 4:3 ratio is commonly referred to as a standard definition capture, while a 16:9 ratio is typically referred to today as the format for so-called “High-Definition”. Other “panoramic” video formats also exist, such as “Cinemascope”® which has a 21:9 format. It will be appreciated that the ability to extend the frame in any desired direction by the present invention can be utilized to upscale from any smaller format to any larger format, such as from 4:3→16:9, or from 16:9→21:9, and so forth without limitation.
In this method a video is captured 70 as the base for the video panorama. Preferably, high resolution panoramic still pictures are captured 72 (e.g., periodically and/or triggered by camera or subject motion) while the video is being captured. It is preferred that at least two still images be captured during a video segment. Camera motion is again preferably registered 74 for aiding in determining when to capture the still images. Adjacent video frames are then combined using super-resolution up-scaling (or digital zoom) techniques 76 to extend the resolution of the video frames. Generated high resolution video frames are then combined as per block 78 into panoramic video frames which are wider (or taller as may be desired in select applications) than the original video frames. Still images can be utilized in this method during both the up-scaling process and for guiding the stitching process during combination of the video frames generated in response to super-resolution techniques. Although this embodiment could be implemented within the camera itself, the high processing overhead required of a video sequence lends itself more particularly to off-line processing as guided by the collected video and still frames, as well as the optional motion information.
In addition, an optional motion sensor 108 is shown, by which panning motion can be registered (detected). Motion sensor 106 may comprise an optical sensor or physical sensor (e.g., inertia, acceleration, etc.), although the image sensor utilized for collecting video and/or still images may be utilized.
An optional second imaging system comprising imager 110 and focus/zoom 112 are shown by way of example, so as to allow certain applications to collect the video and still frames using different imaging devices. This has the benefit of maximizing framing rates of video and still image capture as the functions are not sharing a single capture element. However, it should be appreciated that the video and still image output can be collected from the same imager, such as in response to use of different modes of operation that allow selecting either video or still frames. In at least one implementation, the difference between the video and still frames are the resolution, although the aspect ratio and other characteristics may be different.
The above figure is shown by way of example and not limitation. It should be appreciated that the method according to the present invention can be implemented on various image capture and processing devices which are configured for capturing/receiving a sequence of images and generating panoramic output. The invention is not directed to processing of MPEG videos and its associated motion vector processing. The method is preferably implemented on the imaging device itself, although it may be implemented in post processing, such as in generating a panoramic video output. The present invention can be implemented in hardware and/or software, depending on the target system. It should be appreciated that the invention can be implemented as software on any camera or system which provides the requisite optics and sufficient processing power.
Panoramic images may be displayed on an image display of the apparatus (or printed) and/or output from the apparatus in any desired format, including still image file formats, video formats, data formats, native storage formats, and the like.
Panoramic image techniques taught herein may be utilized in combination with other image processing techniques.
In one variation, the panoramic image techniques taught herein may be utilized in combination with simulated long exposure methods taught elsewhere by the Applicant. In combining these techniques panoramic images may be generated within the camera, or in response to automated post-processing, which also simulate long exposures, producing either still images or video frames as output. It will be appreciated that both panoramic processing and simulation of long exposures can be facilitated in response to obtaining a sufficient number of images while panning the camera, or by using the dual (still-video) image capture techniques.
The present invention provides methods and apparatus for automatically generating panoramic images in response to the input of a sequence of overlapping images captured during panning. Inventive teachings can be applied in a variety of apparatus and applications, including cameras (still and/or video), video processing equipment and software, video playback devices, and so forth.
As can be seen, therefore, the present invention includes the following inventive embodiments among others:
1. An apparatus for capturing panoramic images, comprising:
(a) means for capturing a sequence of digital images;
(b) a computer with memory coupled to said computer; and
(c) programming adapted for execution on said computer for,
2. An apparatus as recited in embodiment 1, wherein said panoramic image extends further in at least one dimension than a conventional, non-panoramic, image captured by said apparatus.
3. An apparatus as recited in embodiment 1, wherein said panoramic image can span any desired spatial area of a subject.
4. An apparatus as recited in embodiment 1, wherein panoramic images are created by said apparatus without need of externally processing a set of images captured by said apparatus.
5. An apparatus as recited in embodiment 1, wherein said means comprises:
an imager configured for electronically capturing images; and
a focus control adapted for controlling focal length of images captured by said imager.
6. An apparatus as recited in embodiment 1, further comprising programming for selecting a set of sequential overlapping image frames from within said sequence of pair-wise overlapping image frames for being combined.
7. An apparatus as recited in embodiment 1, further comprising a user interface adapted to allow a user to select a panoramic image capture mode.
8. An apparatus as recited in embodiment 1, wherein said panning motion is registered in response to optical sensing.
9. An apparatus as recited in embodiment 1, further comprising a motion sensor whose output is used by said programming for registering panning motion.
10. An apparatus as recited in embodiment 1, further comprising programming for compensating for non-panning motion and/or tilting prior to combining said sequence of pair-wise overlapping image frames into said panoramic image.
11. An apparatus as recited in embodiment 1, wherein said sequence of pair-wise overlapping image frames comprise frames of a video sequence.
12. An apparatus as recited in embodiment 1, wherein the panoramic output of said apparatus comprises panoramic digital photographs.
13. An apparatus as recited in embodiment 1:
14. An apparatus as recited in embodiment 1, further comprising programming for correcting pixel interfacing between pair-wise overlapping image frames after combining said sequence of pair-wise overlapping image frames.
15. An apparatus as recited in embodiment 1, further comprising programming for performing super-resolution techniques on image frames to enhance resolution.
16. A camera configured for automatically creating panoramic still images in response to panning a desired subject, comprising:
(a) an electronic imaging element within a camera adapted for capturing digital images;
(b) a computer with memory coupled to said computer, said computer configured for controlling said electronic imaging element of said camera; and
(c) programming adapted for execution on said computer for,
17. An apparatus as recited in embodiment 16, further comprising programming for correcting pixel interfacing to create a smooth seamless blend of the sequence of pair-wise overlapping image frames after combining said sequence of images.
18. An apparatus as recited in embodiment 16, further comprising programming for performing super-resolution techniques on image frames to enhance resolution.
19. A method of automatically capturing a panoramic still photograph within a camera, comprising:
capturing a sequence of images spanning a desired subject area being panned;
assuring that edges of adjacent images in the sequence of images overlap one another in response to capturing sufficient images based on camera motion as a desired subject area is being panned; and
combining the sequence of images to create at least one panoramic still photographic image.
20. A method as recited in embodiment 19, further comprising executing one or more super-resolution techniques on image frames to enhance resolution.
Although the description above contains many details, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the presently preferred embodiments of this invention. Therefore, it will be appreciated that the scope of the present invention fully encompasses other embodiments which may become obvious to those skilled in the art, and that the scope of the present invention is accordingly to be limited by nothing other than the appended claims, in which reference to an element in the singular is not intended to mean “one and only one” unless explicitly so stated, but rather “one or more.” All structural and functional equivalents to the elements of the above-described preferred embodiment that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Moreover, it is not necessary for a device or method to address each and every problem sought to be solved by the present invention, for it to be encompassed by the present claims. Furthermore, no element, component, or method step in the present disclosure is intended to be dedicated to the public regardless of whether the element, component, or method step is explicitly recited in the claims. No claim element herein is to be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited using the phrase “means for.”
