Patent Application
20070081081
In traditional film photography, it is known to take multiple pictures of a field of view, each at different angles, and then overlap the resulting prints to obtain a single larger image having a wider field of view. Thus, each photograph is an image that is overlapped with preceding and/or successive images to produce the larger, final image. The overlapping technique can be used to provide extra wide-format pictures, commonly referred to as “panorama” pictures. However, the overlapping technique can also be used to generate extra tall pictures and pictures that are extra large, providing a wider angle of view in both the horizontal and vertical directions.
One unique aspect of digital imaging is the ability to digitally process and manipulate the image after the image is stored. Often, this entails transferring image data to a general purpose computer and manipulating the image using imaging software. It is known, for example, to take overlapping images of a scene and then digitally stitch the images together to form a single larger image. The images may even be automatically aligned by computer software by detecting edges and using other known techniques, then combined into a single larger image.
Modern hand-held digital imaging devices include dedicated digital cameras, as well as cell phones, personal digital assistants (PDAs), and other devices incorporating digital imaging functionality. The imaging system in these devices includes an image sensor and various electronics to pass the image from the sensor to a display and/or to memory. Some digital imaging devices such as mid-range and high-end digital cameras include functionality to assist the photographer to produce overlapping images. For example, in a “panorama mode,” it is known to provide in a liquid crystal display (LCD) both the previous image, and a live view, to permit the photographer to manually determine where to position the camera to provide an appropriate amount of overlap. In some prior art devices, the user may select the panning direction (up, down, left, or right) and the camera would then orient the previous image in the display so that the overlapped portion of the previous image would be adjacent the live image, to further assist the photographer. However, in these cases, the photographer is still required to manually align each successive overlapping image with the previous one, and each image must be separately stored by pressing the shutter release.
There therefore exists an unmet need to provide an automated yet reliable mechanism for producing overlapping digital images and for creating composite images using overlapping images.
Broadly speaking, the present invention fills these needs by providing an imaging device capable of automating the capture of overlapping images.
It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a device, or a method. Several inventive embodiments of the present invention are described below.
In one embodiment, a method and apparatus for capturing a plurality of overlapping images using an imaging device are described. The method includes receiving user input indicating a start of a multi-image capture and storing a current image. Each time an amount of rotation of the imaging device about at least one axis exceeds a displacement angle, a new current image is stored. User input indicates an end of the multi-image capture.
Other aspects and advantages will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.
The embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings, and like reference numerals designate like structural elements.
The timing control signals and data lines, such as line 141 communicating between graphics controller 140 and display 160, are shown as a single line but may in fact be several address, data, and control lines and/or a bus. All communication lines shown in the figures will be presented in this manner except as noted to reduce the complexity and better present various novel aspects of imaging device 100.
Imaging module 110 includes an image sensor 112 positioned adjacent to a lens (not shown) such that light is focused on and forms an image on the sensor. Imaging module 110 and image sensor 112 may be combined into a single integrated circuit or exist as separate integrated circuits. Image sensor 112 may be a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) type image sensor that converts light into electronic signals that represent the level of light at each pixel. Other image sensors that are known or may become known that are capable of converting an image formed by light onto a surface into electronic signals representative of the image may also be used. Imaging module 110 then converts these electronic signals into image data, which is passed to graphics controller 140. Imaging module 110 may have varying resolutions depending upon the application. In one embodiment, image sensor 112 comprises a two-dimensional array of pixel sensors in which each pixel sensor has a color filter in front of it in what is known as a color filter array (CFA). One common type of CFA is the Bayer filter in which every other pixel has a green filter over it in a checkerboard pattern, with remaining pixels in alternate rows having blue and red filters. Other types of color image sensors are available or may become available that are contemplated for use with imaging device 100. In addition, the present invention may also be used with a gray-scale image sensor used for taking black and white photographs.
Graphics controller 140 receives image data from imaging module 110, and, in accordance with instructions from host CPU 165, can send the image data to display 160 or host CPU 165. Graphics controller 140 may include image processing capabilities such as image compression technology for converting image data received from imaging module 110 into compressed image data.
Display 160 can be any form of display capable of displaying an image. In one embodiment, display 160 comprises a liquid crystal display (LCD). However, other types of displays are available or may become available that are capable of displaying an image that may be used in conjunction with imaging device 100. Although imaging module 110 and display 160 are presented as being part of imaging device 100, it is possible that one or both of imaging module 110 and display 160 are external to or even remote from each other and/or graphics controller 140. For example, if imaging device 100 can be used as a security camera or baby monitor, it may be desirable to provide a display 160 that is separable from or remote to the imaging module 10 to provide monitoring capability at a remote location. In another embodiment, e.g., for a compact camera, display 160 is not provided. In this case, the photographer may rely on an optical viewfinder or other means for aligning image sensor 112 with the intended subject.
Host CPU 165 performs digital processing operations and communicates with graphics controller 140. In one embodiment, host CPU 165 comprises an integrated circuit capable of executing software retrieved from memory 167. This software provides imaging device 100 with functionality when executed on host CPU 165. Host CPU may also be a digital signal processor (DSP) or other processing device.
Memory 167 may be internal or external random-access memory or non-volatile memory. Memory 167 may be non-removable memory such as flash memory or other EEPROM, or magnetic media. Alternatively, memory 167 may take the form of a removable memory card such as ones widely available and sold under such trademarks as “SD Card,” “Compact Flash,” and “Memory Stick.” Memory 167 may also be any other type of machine-readable removable or non-removable media. Memory 167 may be remote from imaging device 100. For example, memory 167 may be connected to imaging device 100 via a communications port (not shown). For example, imaging device 100 may include a BLUETOOTH® interface or an IEEE 802.11 interface, commonly referred to as “Wi-Fi.” Such an interface may connect imaging device 100 with a host (not shown) for uploading image data to the host. If imaging device 100 is a communications device such as a cell phone, it may include a wireless communications link to a carrier, which may then store data in hard drives as a service to customers, or transmit image data to another cell phone or email address. Memory 167 may be a combination of memories. For example, it may include a removable memory card for storing image data, and a non-removable memory for storing data and software executed by host CPU 165.
Host CPU 165 is also in communication with user input 150, motion sensor 152, and focus and zoom servos 154. In one embodiment, user input device 150 comprises a shutter button 205 (see
Motion sensor 152 provides electronic signals to host CPU 165 indicating a relative rotation about at least one axis. In one embodiment, motion sensor 152 comprises a gyroscopic motion sensor, such as the Epson@ XV-3500 Gyro Sensor available from Epson Electronics America, Inc. of San Jose, Calif. This gyroscopic motion sensor is a vibration type sensor having no rotating parts. Other motion sensors or absolute position sensors, such as ones sensitive to the Earth's magnetic field and/or gravity, may also be used to determine relative rotation of imaging device 100.
Returning to
Changing the focal length will affect the angle of view 156 (α). The angle of view may be measured horizontally, vertically, or diagonally, and will vary with the dimensions and aspect ratio of image sensor 112, as well as the focal length. In addition, imaging device 100 may include electronic zoom functionality, which can also affect angle of view 156. An electronic zoom is a feature of some imaging devices allowing a subset of pixel sensors at the center of image sensor 112 to define the image. As the zoom factor is electronically increased, a smaller and smaller subset of pixel sensors are employed, which causes a corresponding reduction in the angle of view. Thus, the angle of view will vary with both the optical zoom setting, which varies the focal length, and the electronic zoom setting, which effectively varies the size of the sensor.
In operation, a photographer may save a single image by orienting imaging device 100 such that a desired image is aligned with image sensor 112 of imaging module 110. Graphics controller 140 then passes resulting image data to either or both of display 160 and host CPU 165 for storage in memory. Imaging module 110 and/or graphics controller 140 may include image processing circuitry for compressing the image using an image compression algorithm such as the well known JPEG image format. It is also possible to have a system wherein the sensor data is not compressed at all, but stored in a “RAW” uncompressed format, and stored in this format in memory 167 for later processing in camera or using a general-purpose computer. In one mode of operation, display 160 is continuously updated with an image most recently received by imaging module 110. When the user inputs a desire to send data representing a current image to memory 167, the user will interact with user input device 150 causing an image received by imaging module 110 to be passed by graphics controller 140 to host CPU 165, and stored in memory 167. In one embodiment, imaging device 100 has at least two modes of operation. In one mode, only single images are taken for each interaction with user input device 150. In another mode of operation, imaging device 100 takes a series of overlapping images to be stitched together to form a single image having a larger angle of view.
Instead of or in addition to taking single still images, imaging device 100 may be capable of generating a video stream. In this case, graphics controller 140 may receive an image periodically, e.g., 30 times a second, which is then encoded using Moving Picture Experts Group (MPEG) or other encoding technology and stored in memory 167. In the case of a video recording device, motion sensor 152 may be used both for capturing overlapping still images of a scene as will be described in more detail below, and for electronically compensating for camera shake when recording a video as is known in the art.
Image stitching algorithms wherein two or more overlapping images are combined into a single seamless image are known in the art. In general terms, in an image stitching operation, overlapping images are compared with each other to determine where the images overlap in an initial alignment operation. This may be performed computationally. For example, some algorithms identify and match up edges within each image. An edge is a line between two contrasting areas. For example, an edge might exist between a mountain and the sky or various features within the mountain. Once edges are identified for each image, the edges are compared from one image to the next to determine likely matches. If a match is made, then the overlapping images are digitally stitched together to form final image 324 in a merging operation. The merging operation may include some transformation/dewarping of the images to compensate for the differing perspectives between images. This transformation may comprise computationally flattening the projection of the sequence of images taken on the arc for presenting a final flat two dimensional image. Displacement angle information from motion sensor 152 may be used in the transformation operation. The merging operation can also include a final cropping so that misalignments between successive images can be hidden.
The alignment and merging operations can be done independently of one another. For example, each image may be aligned and merged with a previous image as each image is captured during multi-image capture. In another embodiment, all images 322a-322d may be first aligned in a first step, then merged in a second step. In another embodiment, each image is aligned with a previous image during and/or after the multi-image capture and then all the alignment information is gathered and each image is placed into a final larger image, which is then stored in memory.
In another embodiment, images are stored separately in memory 167 to be later stitched together using a general purpose computer. In this embodiment, host CPU may add meta data to each image file header to identify each overlapping image 322a-322d as one of a plurality of overlapping images. The meta data may include identifying the image number in the sequence of images for each image and/or a displacement vector in the header file for each image after the first in the series. The displacement vector can provide the approximate number of pixels vertically and/or horizontally, of image displacement from the previous image in the series. This can be used by a stitching algorithm as a hint or starting location for aligning overlapping images. If imaging device has a three-axis motion sensor, then a rotation angle about z-axis 210 (
In operation 336, the user orients imaging device 100 so that the view finder 214 or LCD 216 (
In operation 338, the user presses shutter button 205 (
In operation 342, shutter button 205 (
In operation 354, imaging device 100 receives a user panorama mode selection. In one embodiment, this operation is skipped and the user mode is inferred by how long the user holds down shutter button 205 or its equivalent. If imaging device infers the user mode, then the procedure skips to operation 356. Otherwise, the procedure flows to operation 356 upon selection by the user of the panorama mode.
In operation 356, imaging device 100 determines whether shutter button 205 (
In operation 358, an displacement angle Δα is determined based on current zoom setting (if any). If imaging device 100 has a zoom feature, then the angle of view can be calculated based on the zoom setting. As discussed above, the zoom setting may include an optical zoom, which varies the focal length, and/or a digital zoom, which varies the effective size of the sensor. Displacement angle Δα is determined for vertical and horizontal directions based on the vertical and horizontal angle of view. For example, if a field of view is 30° horizontally and 20° vertically (for an imaging device having a 3:2 aspect ratio) then the displacement angle Δα may be calculated as 2Δα/3 or 20° horizontal and 14° vertical. After calculating displacement angle Δα, the procedure flows to operation 360.
In operation 360, imaging device determines autofocus and exposure settings. Autofocus settings and exposure settings can be determined in any convenient manner consistent with the foreseeable application of the device. In one embodiment, autofocus is achieved by taking one or more preview images and, using software and/or hardware implemented algorithms, analyzing appropriate regions of the image for contrast. In addition to determining an appropriate autofocus setting, imaging device 100 can also determine an appropriate exposure setting. Exposure setting can be determined concurrently with autofocus using the preview image and identifying the brightness level of the scene and compensating for the brightness level. If a flash is used, then a pre-flash can be used when taking the pre-image to assist in both autofocus and exposure. After autofocus and exposure is set, the procedure continues with operation 362.
In operation 362, the current image is stored. Referring to
In a second embodiment, the current image data may be compressed into a compressed-image file format, to be opened and later stitched with previous and/or subsequent images using a general purpose computer. In this case, host CPU 165 may add header information to the image file to store metadata to indicate that the image is one of a series of overlapping images, and to indicate the position of the image in the series. In this regard, a counter is incremented each time operation 362 is executed and the counter value is added to the header for the corresponding image. In addition, the filename used to identify the image may be modified to indicate that the image is one of a series of overlapping images, e.g., by appending a letter or number to the filename. The metadata may also include a displacement vector and other information to assist computer software when stitching the overlapping images together. The displacement vector can be expressed as a number of pixels in the x and y directions, and, if the z-axis is monitored by host CPU 165, then an angle of rotation about the z-axis can also be provided. Rotation about the z-axis by the photographer will result overlapping images being rotated with respect to each other. When stitching the images together, this rotation will require digitally rotating the images to compensate, so that objects visible in the images, such as the horizon, line up properly. After storing the current image, the procedure flows to operation 364.
In operation 364, host CPU determines whether the camera has rotated by the displacement angle ha. To do this, host CPU keeps track of relative rotation of the camera about y-axis 208 and/or x-axis 206 (
In operation 366, host CPU determines whether shutter button 205 (
In operation 368, the current image is stored in the manner described above with respect to operation 362. If imaging device 100 is configured to infer the operating mode based on the amount of time shutter button 205 is held down, then this operation is skipped if the shutter button has not been held longer than some threshold length of time, e.g., one-half of a second or a second. After storing the current image, or skipping the operation of storing the image if the shutter button 205 is not been down for the threshold length of time, the procedure ends as indicated by ending block 370.
In the case where LCD 216 is available, it may be utilized differently than in prior panorama-capable cameras. Specifically, LCD 216 may provide feedback to the photographer as to the progress of the panorama image. In one embodiment, the camera takes and stores successive overlapping images to be stitched together at a later time using a general purpose computer. However, the display can provide an estimated finished stitched image using data from motion sensor 152 (
In another embodiment, images are aligned and stitched together in imaging device 100. In this example, LCD display 216 provides feedback as to the progress of the image capture as well as image stitching. In this case, LCD display 216 displays a live preview image 224 as well as estimated alignments of combined previous images. Images not yet aligned and stitched together may be shaded or colored (not shown in
The methods and systems described above may be augmented to provide additional features. For example, an optional mode may be entered into wherein the focus is adjusted for each of the overlapping images. Another mode may be entered into for generating a spherical image of a virtual scene. In this case, the photographer may be required to rotate imaging device 100 a complete 360° then tilt the camera up (or down) and rotate another 360°, and continue this process until the camera is pointing up or down. Once these images are stored, they can then be combined to generate a virtual scene in a computer that allows a user to view the scene at any angle. In another embodiment, perspective correction can be applied during image stitching. In this example, motion sensor 152 (
Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
