Method for correcting an acquired image

Information

  • Patent Grant
  • 11257192
  • Patent Number
    11,257,192
  • Date Filed
    Monday, December 23, 2019
    4 years ago
  • Date Issued
    Tuesday, February 22, 2022
    2 years ago
Abstract
A method of correcting an image obtained by an image acquisition device includes obtaining successive measurements, Gn, of device movement during exposure of each row of an image. An integration range, idx, is selected in proportion to an exposure time, te, for each row of the image. Accumulated measurements, Cn, of device movement for each row of an image are averaged across the integration range to provide successive filtered measurements, G, of device movement during exposure of each row of an image. The image is corrected for device movement using the filtered measurements G.
Description
FIELD OF THE INVENTION

The present invention relates to a method for correcting an acquired image.


DETAILED DESCRIPTION

Referring to FIG. 1, image acquisition devices 10 typically comprise an image processing pipeline (IPP) 12 which obtains acquired image data from an image sensor (not shown), conducts basic processing of the raw image data, such as color balancing, and writes acquired images or portions of images via a system bus 20 to system memory 14. Image frames can be acquired by the IPP 12 at frame rates from 24 fps up to from 60 fps to even 240 fps.


Such image acquisition devices 10 can include downstream dedicated image processing units which can analyse acquired images and process such images either to extract information from the images or to correct the images. Such processing can includes face detection and tracking, object recognition or distortion correction such as disclosed in PCT Application WO2014/005783 (Ref: FN-384). In the present specification, such processing units, which can be dedicated hardware modules or a generic central processing unit (CPU), are indicated as processing unit (PU) 16 which is capable of running either low-level firmware/software or in the case of the CPU, application software, capable of obtaining image information from memory 14 and further processing the images.


In the present specification, we refer to images being provided by IPP 12, however, it will be appreciated that these can comprise either individually acquired images or images within a video sequence.


It is known for image acquisition devices 10 to include inertial measurement units (IMU) 18 which record a trajectory of device movement during image acquisition, enabling processing unit(s) 16 to use that information to correct an acquired image to take into account blur caused by involuntary or unwanted device motion during image capture or to stabilize video sequences, for example, as disclosed in Sung Hee & Park Marc Levoy “Gyro-Based Multi-Image Deconvolution for Removing Handshake Blur”, pp 3366-3373, Conference on Computer Vision and Pattern Recognition, 23-28 Jun. 2014. IMUS 18 can include any combination of gyroscopes, accelerometers and magnetometers. Typically involuntary or unwanted device motion blur is caused by human hand shake or tremor as distinct from a deliberate sweeping movement used by photographers or videographers when attempting either to simulate motion blur or to capture an image of a moving object.


Referring to FIG. 2A, some image acquisition devices employ a rolling shutter technique where one or more lines (rows) of an image are read from an image sensor during successive exposure intervals. Each line or group of lines R1,R2,R3,R4 . . . of the image sensor is exposed over certain period of time tl1, meaning the light intensity is integrated over that period of time. If the acquisition device is moved during the exposure time, the total line exposure is a sum of light coming from different directions. If an image acquisition device is subjected to relatively high frequency vibrational movement during image acquisition using rolling shutter, i.e. a frequency (f) higher than the total exposure time for the image frame tf, then a vertical straight edge within a scene can appear to oscillate as indicated by the line 22.


Let us first assume a perfect synchronization between IMU samples and the starts of exposure for each of the (groups of) line(s) R1,R2,R3,R4 . . . of the image sensor. When the exposure time is very short, i.e. where the frequency of vibration (f) is less than 1/tl1, where tl1 is exposure time for a line, the captured image represents a short moment in time for a given camera orientation. In that case, motion of image features correlates very well with the motion recorded by the IMU sensors during the exposure time for a given (group of) line(s)—each being generally linear or at least monotonic. Thus, by delaying an IMU signal by ½ of the exposure time tl1, such an image can corrected according to the IMU sensor data.


However, with the increasing exposure time, tl2>tl1, such a correction becomes inappropriate. So referring to FIG. 2B, a vertical line captured with a rolling shutter where vibration is at a frequency (f) higher than 1/tl2, without correction, appears as a blurred line 24 within an image. Correction performed according to the IMU sensor data now starts to introduce overcompensation, which in the extreme makes the corrected result look worse, by further distorting the line 24, pulling alternating segments of the line in opposite directions without unblurring the line detail.


So while relatively slower (lower frequency) camera movements can still compensated properly even with a long exposure image where tl2>tl1, if the camera is vibrating at frequencies higher than 1/t, the exposure time for lines of the image, overcompensation is particularly visible.


This becomes a particular problem in drone mounted cameras where the vibration caused by the drone rotors can be of the order of 100 Hz. For images acquired at high light levels, exposure times of as little as 1 ms may be employed and so conventional correction of these images is unaffected by the vibration. However, for images acquired in lower light environments, exposure levels of up to 30 ms or so may be required and these images can be heavily distorted if corrected using conventional techniques.


It is tempting to address the above problem by making correction amplitude a direct function of exposure time, i.e. to damp the degree of correction in proportion to exposure time. However, this does not provide a satisfactory solution, as this affects all vibration frequencies in the same way and so for example, correction of longer-exposed images subject to low frequency shake becomes particularly unsatisfactory.


SUMMARY

According to the present invention, there is provided a method of correcting an image according to claim 1.


Embodiments filter data provided by an inertial measurement unit to provide for stabilization of images with varying exposure times which can be subject to vibration frequencies higher than the readout frequency of an image sensor.


The invention is based on the observation that for longer exposure time images, mechanical vibrations, typically greater than human hand shake or tremor frequencies and for example caused by drone motors, cause oscillating motion blur within acquired images, including images within a video sequence.


Correction of such longer exposure time images which are subject to high frequency vibration using the embodiment of the present invention does not necessarily remove high frequency motion blur from the image, but it prevents further distortion of the image.


The same results cannot be achieved using low-pass filters typically built into gyroscopes within device IMUS. These are meant to prevent any aliasing artefacts caused by sampling and to reduce signal noise. Typically they offer several pre-defined filter settings for applications like inertial navigation or applications which depend on device orientation where the frequency components of a measured signal are relatively low. Those filters are implemented as IIR filters with a phase delay that is uncorrelated with the exposure time and can be different for different signal frequencies.





BRIEF DESCRIPTION OF THE DRAWINGS

An embodiment of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:



FIG. 1 is a block diagram of an image acquisition device according to an embodiment of the invention; and



FIG. 2A and FIG. 2B illustrate the effect of acquisition device movement during exposure of respective images from a rolling shutter exposure.





DESCRIPTION OF THE PREFERRED EMBODIMENT

In embodiments of the present invention, IMU sensor data is integrated over the exposure time of the lines of an acquired image so tending to average the record of device movement. As a result, correction to each line of an image is calculated for average camera orientation during the exposure time of the line. Importantly, the range of the integrator (or the corner frequency of the integrator) is a function of the exposure time of the image.


In one embodiment the IMU sensor data comprises gyroscope samples and the method operates as follows:

    • 1. Successive gyroscope samples Gn are added to a cumulative buffer, where each new element is a sum of the current value Gn and all previous sample values added to the buffer. The cumulative value corresponding to sample Gn is denoted as Cn
    • 2. An index to the cumulative buffer is calculated dependent on the gyroscope sampling frequency F and exposure time te as follows: idx=F*te
    • 3. An average gyroscope rate is calculated: G=(Cn−Cn-idx)/idx
    • 4. The average rate G is used as the numerical integration of camera orientation. It is these filtered values G which are used subsequently in place of the trajectory of corresponding original samples Gn for correction of corresponding lines of an image in an otherwise conventional fashion. It will be seen that this mode of filtering does not require any additional corrections and will automatically adapt to changing exposure time.


It will be seen that for short exposure images, idx will be short and so the linear approximation G of device movement during exposure time will provide similar values to the original samples G and so will be appropriate for correcting such images. On the other hand, as exposure times increase, the averaging will have the effect of not overcorrecting an image subject to high frequency vibration, but can still provide useful correction for images subjected to human hand shake or tremor.


While the above example has been described in terms of gyroscope samples, the same technique can be applied to sample values from all IMU sensors (gyroscope, accelerometer, magnetometer) in implementations where full sensor fusion is required.


The above approach can be employed whether images are acquired using a rolling shutter technique or not. Where lines of an image are exposed successively, then successive corresponding IMU measurements filtered as above can be employed to correct for device movement during exposure of those lines; whereas for an image where all lines are exposed at the same time, then the same IMU measurements filtered as above are applied to all lines.


The filtered IMU sensor signals described above can be employed in electronic image stabilisation (EIS) schemes such as disclosed in co-filed U.S. patent application Ser. No. 15/048,149, entitled “A method of stabilizing a sequence of images” (Reference: IO002-0474-US-01) in place of raw sensor signals conventionally employed to mitigate problems caused by high frequency vibration of the camera during image capture.

Claims
  • 1. A method comprising: receiving sensor data representing movement of a device, the sensor data comprising a set of measurements of device movement during exposure of a first row and a second row of an image;determining an integration range based at least in part on an exposure time of the first row and the second row of the image;determining an average measurement of device movement based at least in part on the set of measurements and the integration range; andcorrecting the image for device movement based at least in part on the average measurement.
  • 2. A method according to claim 1, wherein determining the integration range is further based at least in part on a sampling frequency associated with the first row or the second row of the image.
  • 3. A method according to claim 1, wherein determining the integration range is further based at least in part on a first sampling frequency associated with the first row and a second sampling frequency associated with the second row.
  • 4. A method according to claim 1, wherein at least one of the set of measurements of device movement comprises a gyroscope measurement, an accelerometer measurement, or a magnetometer measurement.
  • 5. A method according to claim 1, wherein the set of measurements of device movement comprises a first measurement of device movement associated with a first sensor and a second measurement of device movement associated with a second sensor.
  • 6. A method according to claim 1, wherein the image is acquired using a rolling shutter technique.
  • 7. A method according to claim 1, wherein the image is part of a video sequence and wherein correcting the image for device movement using the average measurement comprises stabilizing multiple images of the video sequence.
  • 8. A method according to claim 1, wherein correcting the image for device movement using the average measurement comprises correcting blur in at least a portion of the image to account for device motion during image capture.
  • 9. A system comprising: one or more processors; andmemory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform actions comprising: receiving sensor data representing movement of a device, the sensor data comprising a set of measurements of device movement during exposure of a first row and a second row of an image;determining an integration range based at least in part on an exposure time of the first row and the second row of the image;determining a measurement of device movement based at least in part on the set of measurements and the integration range; andcorrecting the image for device movement based at least in part on the measurement.
  • 10. A system according to claim 9, wherein at least one of the set of measurements of device movement comprises a gyroscope measurement, an accelerometer measurement, or a magnetometer measurement.
  • 11. A system according to claim 9, wherein the set of measurements of device movement comprises a first measurement of device movement associated with a first sensor and a second measurement of device movement associated with a second sensor.
  • 12. A system according to claim 9, wherein determining the integration range is further based at least in part on a sampling frequency associated with the first row or the second row of the image.
  • 13. A system according to claim 9, wherein the image is acquired using a rolling shutter technique.
  • 14. A system according to claim 9, wherein the image is part of a video sequence and wherein correcting the image for device movement using the measurement comprises stabilizing multiple images of the video sequence.
  • 15. A system according to claim 9, wherein correcting the image for device movement using the measurement comprises correcting blur in at least a portion of the image to account for device motion during image capture.
  • 16. An image acquisition device comprising: a sensor to receive sensor data representing movement of a device, the sensor data comprising a set of measurements of device movement during exposure of a first row and a second row of an image;one or more processors; andmemory storing processor-executable instructions that, when executed by the one or more processors, cause the image acquisition device to perform actions comprising: determining an integration range based at least in part on an exposure time of the first row and the second row of the image;determining an average measurement of device movement based at least in part on the set of measurements and the integration range; andcorrecting the image for device movement based at least in part on the average measurement.
  • 17. A image acquisition device according to claim 16, wherein at least one of the set of measurements of device movement comprises a gyroscope measurement, an accelerometer measurement, or a magnetometer measurement.
  • 18. A image acquisition device according to claim 16, wherein the set of measurements of device movement comprises a first measurement of device movement associated with a first sensor and a second measurement of device movement associated with a second sensor.
  • 19. A image acquisition device according to claim 16, wherein the image is acquired using a rolling shutter technique.
  • 20. A image acquisition device according to claim 16, wherein the image is part of a video sequence and wherein correcting the image for device movement using the average measurement comprises stabilizing multiple images of the video sequence.
CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 120 as a Continuation of U.S. application Ser. No. 15/942,789 filed Apr. 2, 2018, which is a Continuation of U.S. application Ser. No. 15/048,224 filed on Feb. 19, 2016, issued on Apr. 3, 2018 as U.S. Pat. No. 9,934,559 which relates to co-filed U.S. application Ser. No. 15/048,149, entitled “A method of stabilizing a sequence of images” (Reference: IO002-0474-US-01) the disclosures of each of which are hereby incorporated by reference.

US Referenced Citations (23)
Number Name Date Kind
9743001 Stec Aug 2017 B1
9934559 Stec Apr 2018 B2
10515439 Stec Dec 2019 B2
20060140503 Kurata et al. Jun 2006 A1
20080166115 Sachs et al. Jul 2008 A1
20130044230 Zhou Feb 2013 A1
20130044241 Zhou Feb 2013 A1
20130329064 Price et al. Dec 2013 A1
20140009568 Stec Jan 2014 A1
20140362256 Schulze et al. Dec 2014 A1
20150146033 Yasugi May 2015 A1
20150262344 Stec Sep 2015 A1
20150341576 Gu et al. Nov 2015 A1
20160006935 Zhou et al. Jan 2016 A1
20160057350 Uezono Feb 2016 A1
20160105648 Eline et al. Apr 2016 A1
20160360111 Thivent et al. Dec 2016 A1
20170041542 Ono Feb 2017 A1
20170243330 Stec Aug 2017 A1
20170244881 Stec Aug 2017 A1
20180330478 Stec Nov 2018 A1
20190158716 Kerr May 2019 A1
20210088803 O'Sullivan Mar 2021 A1
Foreign Referenced Citations (1)
Number Date Country
WO2014005783 Jan 2014 WO
Non-Patent Literature Citations (3)
Entry
Karpenko, Alexandre, et al. “Digital video stabilization and rolling shutter correction using gyroscopes.” CSTR 1 (2011): 2.
Hee Park, Sung, and Marc Levoy. “Gyro-based multi-image deconvolution for removing handshake blur.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014.
Hanning, Gustav, et al. “Stabilizing cell phone video using inertial measurement sensors.” Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE, 2011.
Related Publications (1)
Number Date Country
20200143523 A1 May 2020 US
Continuations (2)
Number Date Country
Parent 15942789 Apr 2018 US
Child 16724877 US
Parent 15048224 Feb 2016 US
Child 15942789 US