1. Technical Field of the Invention
This invention relates generally to deblurring of still images such as those captured by digital cameras, and more specifically to deblurring still images generated as a multi-frame full resolution image and especially those produced by a camera having a rolling shutter, meaning that the pixels of the frame are captured at different times.
2. Background Art
A digital camera includes a lens mechanism having a fixed or variable shutter and a fixed or variable aperture, and a sensor having a fixed or variable sensitivity. The terms ISO and SOS refer to different but related methods of measuring the effective, useable speed of a sensor. This disclosure will refer simply to these as “sensor speed” so as to not specifically distinguish between ISO and SOS characteristics. The term “shutter speed” will refer to the duration of the opening of the shutter.
Adjusting a digital camera's sensor speed upward enables the use of a faster shutter speed but tends to increase the noisiness of the resulting image, and adjusting it downward tends to decrease the noisiness but increases the required exposure duration.
Slow shutter speeds tend to result in blurred images, caused by motion of the camera and/or the photographed subject. This motion causes a time-dependent smearing in which the photons captured from a specific subject location fall upon different sensor locations. This will be referred to as “motion blur”.
Some cameras have been adapted to reduce noise and motion blur by using what may be termed “multi-frame compositing”. Rather than capturing a photograph by taking a single frame at e.g. 1/30 second shutter duration, which is likely to produce significant motion blur, an extremely rapid burst of e.g. three frames is taken at 1/90 second each, and the three frames are overlayed to produce the final image.
In the simplistic case, the frames are simply directly overlayed and the corresponding pixel values are added or perhaps averaged, depending upon the camera. While this helps reduce noise in a sqrt(n) manner for n averaged frames, it does not reduce motion blur.
To reduce motion blur, some cameras apply motion compensation to the burst frames, attempting to move each to the same registration versus the sensor. Typically, this is done using edge detection techniques which identify object edges and other localized regions in the image which exhibit high “activity levels”, meaning lots of difference from surrounding pixels. Most edge detection is done primarily or exclusively with the luminance (Y′) values of the pixels in a YUV or Y′CbCr color space, ignoring the chrominance values. Brightness differences are generally a much better indicator of an edge than are color differences. In these cameras, the motion vector estimation is performed on the entire image (or, more exactly, the entire Nth frame versus the entire 1st frame), or perhaps a portion of the image, to determine a single motion vector, and the entire Nth frame is moved in the opposite vector and composited with the 1st frame.
Most cameras, especially higher priced ones, use a “full frame” shutter which effectively exposes the entire sensor from the same opening time to the same closing time (except of course for the extremely small time differences caused by the motion of the aperture mechanism, which is so small as compared to the exposure duration as to be ignorable for these purposes).
More recently, some inexpensive cameras have been equipped with a “rolling shutter” mechanism which exposes different regions of the sensor at meaningfully different times. For example, a rolling shutter may expose each pixel for 1/90 second but take 1/9 second to expose the entire sensor. In other words, it takes 1/9 second to travel from the first line of the sensor to the last line of the sensor. Because each pixel is exposed for only 1/90 second, each pixel will not have significant motion blur. However, because there is 1/9 second between when the first row of pixels is exposed and when the last row of pixels is exposed, the subject and/or camera may have moved significantly.
What is needed is an improved camera apparatus and method for recompositing multi-frame images, which compensate for the motion-induced de-registration caused by a rolling shutter.
The invention will be understood more fully from the detailed description given below and from the accompanying drawings of embodiments of the invention which, however, should not be taken to limit the invention to the specific embodiments described, but are for explanation and understanding only.
The digital camera includes a sensor which may be a CCD charge coupled device, a CMOS circuit, or any other suitable means of digitally capturing photons (in or out of the visible spectrum) from a subject and from them generating an electronic representation of the subject. The sensor is equipped with a rolling shutter mechanism which, in this instance, scans the sensor from top to bottom. The camera includes a processor, which may be characterized as a digital signal processor, a microprocessor, a microcontroller, or the like. The processor may be constructed as a single, monolithic, integrated circuit, or as two or more separate ICs or other logic.
The processor is coupled to send control signals to the sensor and to receive pixel data from the sensor. The camera further includes a memory, which typically includes some amount of flash or other non-volatile memory, some of which may be removable, as well as a smaller amount of SRAM or DRAM configured as a high speed buffer for storing pixel data from the sensor as it is written to the slower non-volatile memory. Optionally, some or all of the buffer may be fabricated on the same IC as the processor, like an on-die cache memory. The details of the memory organization are not germane to this invention, so the memory is shown simply as a single entity. In some embodiments, the pixel data may travel from the sensor to the processor and then from the processor to the memory. In other embodiments, the sensor may be equipped with its own logic for performing direct memory access (DMA) to the memory, bypassing the processor.
The reader should continue to refer to
At a high level, the camera captures an image (“takes a picture”) by capturing a rapid burst of images (“frames”) from the sensor, and then amalgamating them into a final image which is written to a multi-image storage area in the memory.
It is highly desirable to reduce or eliminate errors in registration of the multiple frames.
The designer may make a variety of tradeoffs in configuring the operational routines that are performed by the camera, according to the parameters of the application at hand. For example, the buffer memory (or total memory other than the multi-image storage area) may be small, forcing the programmer to optimize his code for memory footprint. Or, the processor may have only a small amount of computational power or throughput, forcing the programmer to adjust parameters so that image quality may be sacrificed for faster execution speed.
In order to reduce computation and simplify motion detection, it is preferable to perform motion detection in the Y′CbCr or other luminance-with-color color space, using only the Y′ luma component in calculating frame-to-frame motion. The alternative is to perform motion detection on each color, and then somehow decide which color represents the motion. The latter approach complicates processing for no real gain in final image quality. If the sensor produces e.g. an RGB image, the camera first converts the RGB image to Y′CbCr format for further processing.
Because of the rolling shutter, the first frame itself might well have registration errors of its various swaths with respect to its first swath (which, being closest in time to when the user activated the shutter, is by definition correct). In some embodiments, the camera could use a frame other than the first frame as the reference frame. In other embodiments, including the one described below, the first frame is taken as the reference frame, and other frames are aligned to it in order to reduce noise.
The camera copies the first frame to an accumulator, which is a memory structure which advantageously has more bits per pixel than does a given frame, to permit the accumulation (addition) of multiple frames of the image. If each image will arrive as four frames and has a conventional 8 bits per subpixel (color component of a pixel), the designer could choose to configure the accumulator to have 10 bits per pixel (4*2^8=2^10) which minimizes the memory footprint of the accumulator, or he could simply use the next power of 2 size (2^16) which uses more memory but may yield faster computation. Optionally, the processor may subsample the chroma components of the image, to reduce the accumulator's memory footprint.
The camera also saves a full resolution reference (FullResRef) copy of the Y′ luma component in another memory structure. The FullResRef serves as the reference against which subsequent images will be compared to detect motion. Thus, in one embodiment, motion detection is based solely on the Y′ luma values and not on the chroma values. In other embodiments, motion detection could be performed using the full color space, in which case the FullResRef storage would need to be made larger to accommodate also storing the Cb and Cr values for each pixel, rather than just the Y′ value.
In one embodiment, motion detection is limited to integer pixel displacements only. In other embodiments, finer-grained detection could be employed.
This inventor has observed that a thumbnail of the FullResRef typically contains enough information that it can be used to greatly reduce the amount of work required to identify enough “interesting” pixel regions against which subsequent images can effectively be compared for motion detection. Well-known image decimation techniques can be used to create a LowRes image having the desired aspect ratio (and X and Y values) from the FullResRef image with its potentially independent aspect ratio (and X and Y values).
In one embodiment, pixels having an activity level above a predetermined threshold are marked as good candidates, and all others are marked as poor candidates. In another embodiment, the threshold is dynamically determined such that a predetermined percentage of the total pixels are marked as good candidates. In one such embodiment, this is done on a line by line basis. In the particular example shown, 1's mark pixels deemed good candidates, and the threshold has been dynamically adjusted on a line by line basis such that each line has between 3 and 8 of its 32 pixels marked as good candidates. The designer can select the threshold or range according to the parameters of the camera, such as the full resolution, size and speed of memory, computational throughput and latency of the processor, lens aperture, shutter speed, number of frames in a multi-frame image, and so forth.
In one embodiment, once the CM has been constructed and its values populated, the thumbnail is no longer needed and its memory can be released, and the CM is retained for the duration of the processing of the current multi-frame image. In another embodiment, the thumbnail is retained and the CM is recreated as needed, so the CM's storage may be periodically released. The designer can make many such tradeoffs, depending upon the dictates of the application at hand.
Once the CM has been completed, the processor can use it to perform a baseline analysis of the first frame.
For each swath in succession, the points on its key line are analyzed in a two-pass process. The first pass initializes then creates a histogram, and the second pass uses the histogram to identify the top N % most edge-like pixels and record them in an Edge Matrix (EM), where N is a predetermined value for example.
In the first pass, a stride point is analyzed by first converting its X,Y location in the frame into a corresponding position in the thumbnail—or, more precisely, in the CM. If the CM value at that position indicates that the pixel is not a good candidate (unlikely to be an edge), that point is simply skipped. If the CM value indicates that the pixel is a good edge candidate, then a more robust, fine-grain analysis is done in the full resolution frame.
In one mode, the fine-grain analysis includes all immediate neighbors of the pixel at the point in question. The immediate neighbors constitute a 3-by-3 array of pixels centered at the point. If the fine-grain analysis indicates that the pixel is still a good edge candidate, a histogram entry is incremented. In one embodiment, the histogram entry is selected by using the value max(3-by-3 array)−min(3-by-3 array) as an index into the histogram.
After all stride points in the key line have been analyzed, the histogram represents the swath. The processor inspects the histogram and identifies an index (CMI) such that a predetermined target activity level percentage (TAL) of the stride points in that swath have an activity level above that indexed by the CMI. For example, it may be desirable to perform re-registration using only the 10% most edge-like stride points. The value of TAL can be selected by the designer according to the processing power etc. constraints of the camera.
Then, once the CMI has been identified according to the TAL, the second pass can be made, to find the TAL % most edge-like stride points in the swath. A fine-grain (3-by-3) analysis is again made for each stride point whose CM counterpart indicates it to be a good candidate, and if the activity level is greater than Histogram[CMI], that stride point's entry in the Edge Matrix is set to indicate that it is actually an edge point against which future frames will be compared for motion. Alternatively, if memory permits, the results of the first pass fine-grain analysis could be saved and then simply read during the second pass, rather than being recalculated.
The second pass is complete, the histogram memory can be released, and the Edge Matrix will be used for comparing future frames' swaths against the FullResRef copy of the first image's Y′ luma values.
The amount of shift will depend on the amount of camera movement and the speed of the rolling shutter. The amounts shown in the drawings are for illustration only. And, for ease of illustration, the drawings illustrate a simplistic example in which the camera movement is only in the horizontal plane. In reality, the camera may also move vertically, and may even twist.
This is the significance of de-registration. Unless frames/swaths are correctly re-registered, averaging them with the original frame will blur the image in unpredictable degrees dependent upon the amount of camera motion and the scene itself.
For each swath, the corresponding row of the Edge Matrix is consulted. For any stride point of the FullResRef identified as being in the top TAL % (by being marked as a 1), the same location is selected in the new frame. Any suitable pattern matching/motion estimation methodology can be used, to find the pixels in the new frame's swath which most closely resemble (by having the best pattern match of their neighbors versus the FullResRef) the first frame's corresponding swath. The searching may advantageously start from the displacement vector found on the previous swath, if this is not the first swath. A search may also be centered around a zero displacement in case the previous swath's motion vector is unreliable. If this swath is the first swath, then a wider search around the zero displacement may be needed in order to establish frame-to-frame motion. Searching then progresses systematically outward from the point in either case, because likelihood decreases as the distance from the center increases. For each match found, a motion vector is calculated, which describes how that stride point's pixel region moved from the first frame to the current frame. After all the EM-identified pixels have been thus examined, their resulting motion vectors are compared, to generate a single motion vector ascribed to the whole swath. Any suitable algorithm may be used in doing so. For example, preferably the mode (most common) motion vector would be selected.
In one mode, the new frame's swath may individually be re-registered in the inverse of the swath's overall motion vector, and the result added to the corresponding swath of the accumulator.
In another mode, overall motion vectors are calculated for all of the swaths of the current frame, and then some intelligence is applied against them as a set, before doing any re-registration and accumulation. For example, it is extremely unlikely that consecutive swaths would be correctly found to have moved “left 2 pixels”, “left 2 pixels”, “left 2 pixels”, “right 17 pixels”, and “left 2 pixels”. In that case, it may be determined that the “right 17 pixels” vector is simply an error, and that swath's overall motion vector may be changed to e.g. the average of those above and below it.
In one embodiment, the missing pixels are filled by accumulating the corresponding pixel values from the FullResRef first frame. Alternatively, the missing pixel filling can be done by scaling the corresponding pixel locations in the accumulator by the value F/(F−1), where F is the number of frames received so far (and the reference first frame is at F=1), and where averaging starts with frame F=2 so no divide-by-zero occurs. The first method is preferable in order to avoid magnifying errors in the accumulator. When using only Y for motion calculations, chrominance reference frame data is not usually kept in order to save memory, so the alternative method must be used in this case.
After all of the frames have been thus processed, the accumulator represents a scaled version of the final image. The accumulator is normalized by dividing its pixel values by F, the total number of frames in the multi-frame image. Then the accumulator contents can be written to the camera's multi-image storage, after being converted back to RGB space if necessary. In some cameras, it may also be desirable to first perform other conventional operations such as automatic white balancing, sharpening, cropping, generating EXIF information, and so forth, before storing the result as the final image.
Motion blur has been reduced by shooting a multi-frame burst of short-exposure frames rather than a single long-exposure frame. Noise has been reduced by having been averaged out, as random noise is unlikely to affect the same pixel in the same manner multiple times, and sensor pixel specific noise will be spread around the final image by the re-registration process. And rolling shutter artifacts have been reduced by re-registering swaths of the second and later images.
In one embodiment, the second and subsequent frame are compared against a saved copy of the first frame, rather than against the accumulator, to avoid motion detection errors caused by accumulation (of errors).
Tables 1-4 list pseudo-code such as may be used in performing one embodiment of the method of this invention, which may differ in some respects from that described above.
When one component is said to be adjacent another component, it should not be interpreted to mean that there is absolutely nothing between the two components, only that they are in the order indicated.
The various features illustrated in the figures may be combined in many ways, and should not be interpreted as though limited to the specific embodiments in which they were explained and shown.
Those skilled in the art, having the benefit of this disclosure, will appreciate that many other variations from the foregoing description and drawings may be made within the scope of the present invention. Indeed, the invention is not limited to the details described above. Rather, it is the following claims including any amendments thereto that define the scope of the invention.
In the claims, numbering and lettering is for convenience and clarity, and should not be interpreted as requiring any particular order of steps in a method. Ordering, if any, should be implied only where it is implicit in the recitation. For example, “(1) obtain a chicken, and (2) cook the chicken” does require an ordering, but “(1) set the table, and (2) cook dinner” does not; in the latter case, the two steps may be performed in either order or even in parallel.
Number | Name | Date | Kind |
---|---|---|---|
3904818 | Kovac | Sep 1975 | A |
4253120 | Levine | Feb 1981 | A |
4646251 | Hayes et al. | Feb 1987 | A |
4685071 | Lee | Aug 1987 | A |
4739495 | Levine | Apr 1988 | A |
4771470 | Geiser et al. | Sep 1988 | A |
4920428 | Lin et al. | Apr 1990 | A |
4987496 | Greivenkamp, Jr. | Jan 1991 | A |
5175430 | Enke et al. | Dec 1992 | A |
5261029 | Abi-Ezzi et al. | Nov 1993 | A |
5305994 | Matsui et al. | Apr 1994 | A |
5387983 | Sugiura et al. | Feb 1995 | A |
5475430 | Hamada et al. | Dec 1995 | A |
5513016 | Inoue | Apr 1996 | A |
5608824 | Shimizu et al. | Mar 1997 | A |
5652621 | Adams, Jr. et al. | Jul 1997 | A |
5793433 | Kim et al. | Aug 1998 | A |
5878174 | Stewart et al. | Mar 1999 | A |
5903273 | Mochizuki et al. | May 1999 | A |
5905530 | Yokota et al. | May 1999 | A |
5995109 | Goel et al. | Nov 1999 | A |
6016474 | Kim et al. | Jan 2000 | A |
6078331 | Pulli et al. | Jun 2000 | A |
6111988 | Horowitz et al. | Aug 2000 | A |
6118547 | Tanioka | Sep 2000 | A |
6128000 | Jouppi et al. | Oct 2000 | A |
6141740 | Mahalingaiah et al. | Oct 2000 | A |
6151457 | Kawamoto | Nov 2000 | A |
6175430 | Ito | Jan 2001 | B1 |
6252611 | Kondo | Jun 2001 | B1 |
6256038 | Krishnamurthy | Jul 2001 | B1 |
6281931 | Tsao et al. | Aug 2001 | B1 |
6289103 | Sako et al. | Sep 2001 | B1 |
6314493 | Luick | Nov 2001 | B1 |
6319682 | Hochman | Nov 2001 | B1 |
6323934 | Enomoto | Nov 2001 | B1 |
6392216 | Peng-Tan | May 2002 | B1 |
6396397 | Bos et al. | May 2002 | B1 |
6438664 | McGrath et al. | Aug 2002 | B1 |
6469707 | Voorhies | Oct 2002 | B1 |
6486971 | Kawamoto | Nov 2002 | B1 |
6504952 | Takemura et al. | Jan 2003 | B1 |
6584202 | Montag et al. | Jun 2003 | B1 |
6594388 | Gindele et al. | Jul 2003 | B1 |
6683643 | Takayama et al. | Jan 2004 | B1 |
6707452 | Veach | Mar 2004 | B1 |
6724423 | Sudo | Apr 2004 | B1 |
6724932 | Ito | Apr 2004 | B1 |
6737625 | Baharav et al. | May 2004 | B2 |
6760080 | Moddel et al. | Jul 2004 | B1 |
6785814 | Usami et al. | Aug 2004 | B1 |
6806452 | Bos et al. | Oct 2004 | B2 |
6839062 | Aronson et al. | Jan 2005 | B2 |
6856441 | Zhang et al. | Feb 2005 | B2 |
6891543 | Wyatt | May 2005 | B2 |
6900836 | Hamilton, Jr. | May 2005 | B2 |
6950099 | Stollnitz et al. | Sep 2005 | B2 |
7009639 | Une et al. | Mar 2006 | B1 |
7015909 | Morgan, III et al. | Mar 2006 | B1 |
7023479 | Hiramatsu et al. | Apr 2006 | B2 |
7088388 | MacLean et al. | Aug 2006 | B2 |
7092018 | Watanabe | Aug 2006 | B1 |
7106368 | Daiku et al. | Sep 2006 | B2 |
7133041 | Kaufman et al. | Nov 2006 | B2 |
7133072 | Harada | Nov 2006 | B2 |
7146041 | Takahashi | Dec 2006 | B2 |
7221779 | Kawakami et al. | May 2007 | B2 |
7227586 | Finlayson et al. | Jun 2007 | B2 |
7245319 | Enomoto | Jul 2007 | B1 |
7305148 | Spampinato et al. | Dec 2007 | B2 |
7343040 | Chanas et al. | Mar 2008 | B2 |
7486844 | Chang et al. | Feb 2009 | B2 |
7502505 | Malvar et al. | Mar 2009 | B2 |
7580070 | Yanof et al. | Aug 2009 | B2 |
7626612 | John et al. | Dec 2009 | B2 |
7627193 | Alon et al. | Dec 2009 | B2 |
7671910 | Lee | Mar 2010 | B2 |
7728880 | Hung et al. | Jun 2010 | B2 |
7750956 | Wloka | Jul 2010 | B2 |
7817187 | Silsby et al. | Oct 2010 | B2 |
7859568 | Shimano et al. | Dec 2010 | B2 |
7860382 | Grip | Dec 2010 | B2 |
7912279 | Hsu et al. | Mar 2011 | B2 |
8049789 | Innocent | Nov 2011 | B2 |
8238695 | Davey et al. | Aug 2012 | B1 |
8456547 | Wloka | Jun 2013 | B2 |
8456548 | Wloka | Jun 2013 | B2 |
8456549 | Wloka | Jun 2013 | B2 |
8471852 | Bunnell | Jun 2013 | B1 |
20010001234 | Addy et al. | May 2001 | A1 |
20010012113 | Yoshizawa et al. | Aug 2001 | A1 |
20010012127 | Fukuda et al. | Aug 2001 | A1 |
20010015821 | Namizuka et al. | Aug 2001 | A1 |
20010019429 | Oteki et al. | Sep 2001 | A1 |
20010021278 | Fukuda et al. | Sep 2001 | A1 |
20010033410 | Helsel et al. | Oct 2001 | A1 |
20010050778 | Fukuda et al. | Dec 2001 | A1 |
20010054126 | Fukuda et al. | Dec 2001 | A1 |
20020012131 | Oteki et al. | Jan 2002 | A1 |
20020015111 | Harada | Feb 2002 | A1 |
20020018244 | Namizuka et al. | Feb 2002 | A1 |
20020027670 | Takahashi et al. | Mar 2002 | A1 |
20020033887 | Hieda et al. | Mar 2002 | A1 |
20020041383 | Lewis, Jr. et al. | Apr 2002 | A1 |
20020044778 | Suzuki | Apr 2002 | A1 |
20020054374 | Inoue et al. | May 2002 | A1 |
20020063802 | Gullichsen et al. | May 2002 | A1 |
20020105579 | Levine et al. | Aug 2002 | A1 |
20020126210 | Shinohara et al. | Sep 2002 | A1 |
20020146136 | Carter, Jr. | Oct 2002 | A1 |
20020149683 | Post | Oct 2002 | A1 |
20020158971 | Daiku et al. | Oct 2002 | A1 |
20020167202 | Pfalzgraf | Nov 2002 | A1 |
20020167602 | Nguyen | Nov 2002 | A1 |
20020191694 | Ohyama et al. | Dec 2002 | A1 |
20020196470 | Kawamoto et al. | Dec 2002 | A1 |
20030035100 | Dimsdale et al. | Feb 2003 | A1 |
20030067461 | Fletcher et al. | Apr 2003 | A1 |
20030122825 | Kawamoto | Jul 2003 | A1 |
20030142222 | Hordley | Jul 2003 | A1 |
20030146975 | Joung et al. | Aug 2003 | A1 |
20030169353 | Keshet et al. | Sep 2003 | A1 |
20030169918 | Sogawa | Sep 2003 | A1 |
20030197701 | Teodosiadis et al. | Oct 2003 | A1 |
20030218672 | Zhang et al. | Nov 2003 | A1 |
20030222995 | Kaplinsky et al. | Dec 2003 | A1 |
20030223007 | Takane | Dec 2003 | A1 |
20040001061 | Stollnitz et al. | Jan 2004 | A1 |
20040001234 | Curry et al. | Jan 2004 | A1 |
20040032516 | Kakarala | Feb 2004 | A1 |
20040066970 | Matsugu | Apr 2004 | A1 |
20040100588 | Hartson et al. | May 2004 | A1 |
20040101313 | Akiyama | May 2004 | A1 |
20040109069 | Kaplinsky et al. | Jun 2004 | A1 |
20040189875 | Zhai et al. | Sep 2004 | A1 |
20040218071 | Chauville et al. | Nov 2004 | A1 |
20040247196 | Chanas et al. | Dec 2004 | A1 |
20050007378 | Grove | Jan 2005 | A1 |
20050007477 | Ahiska | Jan 2005 | A1 |
20050030395 | Hattori | Feb 2005 | A1 |
20050046704 | Kinoshita | Mar 2005 | A1 |
20050099418 | Cabral et al. | May 2005 | A1 |
20050111110 | Matama | May 2005 | A1 |
20050175257 | Kuroki | Aug 2005 | A1 |
20050185058 | Sablak | Aug 2005 | A1 |
20050238225 | Jo et al. | Oct 2005 | A1 |
20050243181 | Castello et al. | Nov 2005 | A1 |
20050248671 | Schweng | Nov 2005 | A1 |
20050261849 | Kochi et al. | Nov 2005 | A1 |
20050286097 | Hung et al. | Dec 2005 | A1 |
20060050158 | Irie | Mar 2006 | A1 |
20060061658 | Faulkner et al. | Mar 2006 | A1 |
20060087509 | Ebert et al. | Apr 2006 | A1 |
20060119710 | Ben-Ezra et al. | Jun 2006 | A1 |
20060133697 | Uvarov et al. | Jun 2006 | A1 |
20060176375 | Hwang et al. | Aug 2006 | A1 |
20060197664 | Zhang et al. | Sep 2006 | A1 |
20060274171 | Wang | Dec 2006 | A1 |
20060290794 | Bergman et al. | Dec 2006 | A1 |
20060293089 | Herberger et al. | Dec 2006 | A1 |
20070091188 | Chen et al. | Apr 2007 | A1 |
20070147706 | Sasaki et al. | Jun 2007 | A1 |
20070171288 | Inoue et al. | Jul 2007 | A1 |
20070236770 | Doherty et al. | Oct 2007 | A1 |
20070247532 | Sasaki | Oct 2007 | A1 |
20070285530 | Kim et al. | Dec 2007 | A1 |
20080030587 | Helbing | Feb 2008 | A1 |
20080043024 | Schiwietz et al. | Feb 2008 | A1 |
20080062164 | Bassi et al. | Mar 2008 | A1 |
20080101690 | Hsu et al. | May 2008 | A1 |
20080143844 | Innocent | Jun 2008 | A1 |
20080231726 | John | Sep 2008 | A1 |
20090002517 | Yokomitsu et al. | Jan 2009 | A1 |
20090010539 | Guarnera et al. | Jan 2009 | A1 |
20090037774 | Rideout et al. | Feb 2009 | A1 |
20090116750 | Lee et al. | May 2009 | A1 |
20090128575 | Liao et al. | May 2009 | A1 |
20090160957 | Deng et al. | Jun 2009 | A1 |
20090257677 | Cabral et al. | Oct 2009 | A1 |
20100266201 | Cabral et al. | Oct 2010 | A1 |
Number | Date | Country |
---|---|---|
1275870 | Dec 2000 | CN |
0392565 | Oct 1990 | EP |
1449169 | May 2003 | EP |
1378790 | Jul 2004 | EP |
1447977 | Aug 2004 | EP |
1550980 | Jul 2005 | EP |
2045026 | Oct 1980 | GB |
2363018 | May 2001 | GB |
61187467 | Aug 1986 | JP |
62151978 | Jul 1987 | JP |
07015631 | Jan 1995 | JP |
8036640 | Feb 1996 | JP |
08-079622 | Mar 1996 | JP |
2000516752 | Dec 2000 | JP |
2001052194 | Feb 2001 | JP |
2002-207242 | Jul 2002 | JP |
2003-085542 | Mar 2003 | JP |
2004-221838 | Aug 2004 | JP |
2005094048 | Apr 2005 | JP |
2005-182785 | Jul 2005 | JP |
2005520442 | Jul 2005 | JP |
2006025005 | Jan 2006 | JP |
2006086822 | Mar 2006 | JP |
2006-094494 | Apr 2006 | JP |
2006121612 | May 2006 | JP |
2006134157 | May 2006 | JP |
2007019959 | Jan 2007 | JP |
2007-148500 | Jun 2007 | JP |
2007-233833 | Sep 2007 | JP |
2007282158 | Oct 2007 | JP |
2008085388 | Apr 2008 | JP |
2008113416 | May 2008 | JP |
2008277926 | Nov 2008 | JP |
2009021962 | Jan 2009 | JP |
1020040043156 | May 2004 | KR |
1020060068497 | Jun 2006 | KR |
1020070004202 | Jan 2007 | KR |
03043308 | May 2003 | WO |
2004063989 | Jul 2004 | WO |
2007056459 | May 2007 | WO |
2007093864 | Aug 2007 | WO |
Entry |
---|
Donald D. Spencer, “Illustrated Computer Graphics Dictionary”, 1993, Camelot Publishing Company, p. 272. |
Duca et al., “A Relational Debugging Engine for Graphics Pipeline, International Conference on Computer Graphics and Interactive Techniques”, ACM SIGGRAPH Jul. 2005, pp. 453-463. |
gDEBugger, graphicRemedy, http://www.gremedy.com, Aug. 8, 2006, pp. 1-18. |
Ko et al., “Fast Digital Image Stabilizer Based on Gray-Coded Bit-Plane Matching”, IEEE Transactions on Consumer Electronics, vol. 45, No. 3, pp. 598-603, Aug. 1999. |
Ko, et al., “Digital Image Stabilizing Algorithms Basd on Bit-Plane Matching”, IEEE Transactions on Consumer Electronics, vol. 44, No. 3, pp. 617-622, Aug. 1988. |
Morimoto et al., “Fast Electronic Digital Image Stabilization for Off-Road Navigation”, Computer Vision Laboratory, Center for Automated Research University of Maryland, Real-Time Imaging, vol. 2, pp. 285-296, 1996. |
Paik et al., “An Adaptive Motion Decision system for Digital Image Stabilizer Based on Edge Pattern Matching”, IEEE Transactions on Consumer Electronics, vol. 38, No. 3, pp. 607-616, Aug. 1992. |
Parhami, Computer Arithmetic, Oxford University Press, Jun. 2000, pp. 413-418. |
S. Erturk, “Digital Image Stabilization with Sub-Image Phase Correlation Based Global Motion Estimation”, IEEE Transactions on Consumer Electronics, vol. 49, No. 4, pp. 1320-1325, Nov. 2003. |
S. Erturk, “Real-Time Digital Image Stabilization Using Kalman Filters”, http://www,ideallibrary.com, Real-Time Imaging 8, pp. 317-328, 2002. |
Uomori et al., “Automatic Image Stabilizing System by Full-Digital Signal Processing”, vol. 36, No. 3, pp. 510-519, Aug. 1990. |
Uomori et al., “Electronic Image Stabiliztion System for Video Cameras and VCRS”, J. Soc. Motion Pict. Telev. Eng., vol. 101, pp. 66-75, 1992. |
Weerasinghe et al.; “Method of Color Interpolation in a Single Sensor Color Camera Using Green Channel Separation”; Visual Information Proessing lab, Motorola Australian Research Center; IV 3233- IV3236, 2002. |
“A Pipelined Architecture for Real-Time Correction of Barrel Distortion in Wide-Angle Camera Images”, Hau, T. Ngo, Student Member, IEEE and Vijayan K. Asari, Senior Member IEEE, IEEE Transaction on Circuits and Systems for Video Technology: vol. 15 No. Mar. 3, 2005 pp. 436-444. |
“Calibration and removal of lateral chromatic aberration in images” Mallon, et al. Science Direct Copyright 2006; 11 pages. |
“Method of Color Interpolation in a Single Sensor Color Camera Using Green Channel Seperation” Weerasighe, et al Visual Information Processing Lab, Motorola Austrailan Research Center pp. IV-3233-IV3236, 2002. |
D. Doo, M. Sabin “Behaviour of recrusive division surfaces near extraordinary points”; Sep. 1978; Computer Aided Design; vol. 10, pp. 356-360. |
D. W. H. Doo; “A subdivision algorithm for smoothing down irregular shaped polyhedrons”; 1978; Interactive Techniques in Computer Aided Design; pp. 157-165. |
Davis, J., Marschner, S., Garr, M., Levoy, M., Filling holes in complex surfaces using volumetric diffusion, Dec. 2001, Stanford University, pp. 1-9. |
E. Catmull, J. Clark, “recursively generated B-Spline surfaces on arbitrary topological meshes”; Nov. 1978; Computer aided design; vol. 10; pp. 350-355. |
J. Stam; “Exact Evaluation of Catmull-clark subdivision surfaces at arbitrary parameter values”; Jul. 1998; Computer Graphics; vol. 32; pp. 395-404. |
Krus, M., Bourdot, P., Osorio, A., Guisnel, F., Thibault, G., Adaptive tessellation of connected primitives for interactive walkthroughs in complex industrial virtual environments, Jun. 1999, Proceedings of the Eurographics workshop, pp. 1-10. |
Kumar, S., Manocha, D., Interactive display of large scale trimmed NURBS models, 1994, University of North Carolina at chapel Hill, Technical Report, pp. 1-36. |
Kuno et al. “New Interpolation Method Using Discriminated Color Correlation for Digital Still Cameras” IEEE Transac. On Consumer Electronics, vol. 45, No. 1, Feb. 1999, pp. 259-267. |
Loop, C., DeRose, T., Generalized B-Spline surfaces of arbitrary topology, Aug. 1990, Sigraph 90, pp. 347-356. |
M. Halstead, M. Kass, T. DeRose; “efficient, fair interpolation using catmull-clark surfaces”; Sep. 1993; Computer Graphics and Interactive Techniques, Proc; pp. 35-44. |
T. DeRose, M., Kass, T. Troung; “subdivision surfaces in character animation”; Jul. 1998; Computer Graphics and Interactive Techniques, Proc; pp. 85-94. |
Takeuchi, S., Kanai, T., Suzuki, H., Shimada, K., Kimura, F., Subdivision surface fitting with QEM-based mesh simplification and reconstruction of approximated B-spline surfaces, 2000, Eighth Pacific Conference on computer graphics and applications, pp. 202-212. |
Keith R. Slavin; Application as Filed entitled “Efficient Method for Reducing Noise and Blur in a Composite Still Image From a Rolling Shutter Camera”; Application No. 12069669; Filed Feb. 11, 2008. |
Goshtasby, Ardeshir, “Correction of Image Distortion From Lens Distortion Using Bezier Patches”, 1989, Computer Vision, Graphics and Image Processing, vol. 47, pp. 358-394. |
Number | Date | Country | |
---|---|---|---|
20090201383 A1 | Aug 2009 | US |