The present invention relates to a method for dynamically calibrating an image capture device.
Referring now to
Once the relationship between DAC code and lens position is determined, for example, there can be a linear relationship between the two, the camera module can be calibrated by adjusting the DAC codes for infinity and macro distances:
DACFAR[t]−physical lens position to focus at far (infinity) distance at time [t] [1]
DACNEAR[t]−physical lens position to focus at near (macro) distance at time [t] [2]
These calibration parameters can be determined during a production line process (PLP) and their values stored in a non-volatile memory 20 inside the camera module 12 or elsewhere in the camera.
Thus, the auto-focus module 10 can determine the required DAC code to be supplied to the lens actuator 18 as a function of the distance to the target object as well as DACNEAR[t] and DACFAR[t].
It is known that the camera module 12 may be affected by operating conditions such as SAG (gravity influence) or thermal (temperature influence) and WO 2016/000874 (Ref: FN-396-PCT) discloses some methods to compensate for SAG and thermal effects by adjusting DACNEAR[t] and DACFAR[t] according to operating conditions.
Nonetheless, there may be other components contributing to calibration error including inaccuracies, due to some limitations of the PLP or, as disclosed in WO 2016/000874 (Ref: FN-395-PCT), camera module performance drifting over time, for example, due to device aging or even device on-time.
If PLP, SAG or thermal errors are not compensated accordingly, the DAC code computed by AF module will not provide proper focus on the target object.
The camera module may then be required to hunt for focus and this both impacts adversely on focus speed as well as causing an unacceptable lens wobble effect within a preview stream.
It is an object of the present application to mitigate these problems.
According to the present invention there is provided a method for dynamically calibrating an image capture device according to claim 1.
According to a second aspect there is provided a computer program product comprising a computer readable medium on computer readable instructions are stored and which when executed on an image capture device are arranged to perform the steps of claim 1.
According to a third aspect there is provided an image capture device configured to perform the steps of claim 1.
The present method runs on an image capture device, possibly within a camera module, and dynamically compensates for calibration errors while the user is operating the device.
The method does not affect the production line process and collects the necessary data while the user is operating the device, without adversely affecting the user experience. The method can improve auto-focus speed and minimize lens wobble when estimating the calibration error and then updating the calibration parameters.
The method can be triggered from time to time to check if the calibration parameters haven't been affected by for example, camera ageing, and if so, perform the necessary corrections.
An embodiment of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:
Calibration errors, other than those caused by SAG or thermal effects and referred to herein generally as PLP errors can be quantified as follows:
where:
DACFARPLP and DACNEARPLP are the stored calibration parameters for the camera module (CM). This can be measured and determined at production time, or they can be updated from time to time during camera operation as disclosed in WO 2016/000874 (Ref: FN-395-PCT).
Thus, DACFAR[t] and DACNEAR[t] are the desired corrected calibration parameters at time [t], while ERRFARPLP and ERRNEARPLP are the respective errors in these parameters.
To illustrate the impact of the errors in equations [3] and [4] against the final focus position, let us assume a target object is placed at distance [D] from the camera. Typically, in a handheld image capture device such as a consumer camera, smartphone, tablet computer or equivalent, the object of interest is a human face. The corresponding lens position [DACD] to focus at distance [D] is given by the following formula:
Assuming a linear DAC function, the additional parameters together with their formula are detailed in table 1:
Nonetheless, it will be appreciated that the invention is also applicable to a non-arithmetic, but nonetheless linear relationship between DAC codes and lens position.
Equations [9], [10] and [11] are derived from thin lens equation:
Replacing [6], [7], [8] in [5], the new formula for computing the DAC value becomes:
Replacing [9], [10], [11] in [13] and [14], the final formula which estimates the DAC value is given by:
ERRD is the overall error generated by PLP errors (ERRFARPLP and ERRNEARPLP).
Referring now to
Sensor temperature [TCRT] and device orientation [OCRT] are used to adjust the original calibration parameters to compensate for the CM being affected by SAG (gravity) or thermal effects, as disclosed in WO 2016/000874 (Ref: FN-395-PCT). To briefly explain how to compensate SAG and thermal effects, assume that during production, the CM orientation was OPLP and the sensor temperature was TPLP.
If the CM is affected by SAG, OCRT≠OPLP, then the original calibration parameters are converted into the OCRT range. In the present description, this transformation function is represented as SAG below:
[DACFARPLP,DACNEARPLP]O
If the CM is not affected by SAG, the calibration parameters will remain unchanged:
[DACFARPLP,DACNEARPLP]O
If the CM is affected by thermal effect, TCRT # TPLP, then the calibration parameters after SAG correction are converted into the TCRT range. Again, a transformation function, called TH below, can be used:
[DACFARPLP,DACNEARPLP]T
If the CM is not affected by thermal effects, then the calibration parameters resulting from any SAG correction will remain unchanged:
[DACFARPLP,DACNEARPLP]T
Note that in each case SAG( ) and TH( ) can involve lookup tables, and again details of how to adjust the DACFARPLP and DACNEARPLP values to take into account temperature and orientation are disclosed in WO 2016/000874 (Ref: FN-395-PCT).
At this point, PLP errors are unknown. For estimating them, data is collected during AF module operation as follows:
For large PLP errors, the DACD
For small errors, the DACD
The goal of dynamic compensation method is to use the above data (provided by steps 1, 2 and 3) to estimate PLP errors. Once the estimation is done, the calibration parameters (DACFARPLP, DACNEARPLP) will be properly updated and the lens position provided by [16] will be the focus position. Focus sweeping will not be necessary anymore, and thus the AF module speed will be improved and the lens wobble effect reduced.
One way to define sufficiently good accuracy, is to restrict the errors of [DCRT] and [DACD
To understand the meaning of DOFD
DACSTEP is the absolute difference between the corresponding DAC values at DF and DN distances. Using [15], [16] and assuming ERRD
DACSTEP should be a constant value (should not vary with the distance D).
The second step of the compensation method is to estimate the errors [3] and [4] and to update the calibration parameters. It requires, two input records (TCRT, OCRT, DCRT, DACD
[D1
where:
[T1, O1, D1, DACD
[T2, O2, D2, DACD
[D1
[D2
ERRD
ERRD
where:
DACD
Using [17] and replacing [ERRD, D] with [ERRD
To simplify the above system, the following substitutions will be done:
The new system becomes:
PLP errors can now be estimated with the following formulae:
where:
ERRD
∝D
The new updated calibration parameters (which should be used further to improve AF module speed and reduced lens wobble effect) are:
In a second embodiment of the present invention, instead of directly measuring a distance to an object in a scene being imaged, the distance can be estimated based on an assumed dimension of an object being imaged, for example, a face. More details about estimating the distance based on face information or indeed any recognizable object with a known dimension can be found in U.S. Pat. No. 8,970,770 (Ref: FN-361) and WO 2016/091545 (Ref: FN-399), the disclosures of which are herein incorporated by reference.
However, as disclosed in WO 2016/091545 (Ref: FN-399), care should be taken when doing so to ensure that the object is not a false image of an object, for example, a billboard showing a large face, or a small printed face or a small child's face, where the assumed dimension may not apply. Thus, the second embodiment aims to provide dynamic compensation to estimate ERRPLP while taking into account that false faces may be present in a scene.
Let assume the distance from the face to the image acquisition device is [D]. The current estimated distance [DEST] to that face is computed with the following formula:
where:
For those human faces (where ed≈70 mm), formula [1] will provide a good estimation of the distance (DEST≈D).
For false faces (ex. a small printed face with ed≈20 mm), formula [1] will provide a wrong distance because it assumes that ed=70 mm.
The lens position [DACD] to focus at distance [D] is given by the following formula:
DACD=DACD
where:
DACD
ERRD=ERRPLP+ERRD
Note that in this example, near and far PLP errors are assumed to be almost the same (ERRFARPLP≈ERRNEARPLP≈ERRPLP) and ERRD
Referring now to
The second step of dynamic compensation process is to estimate ERRPLP and to update the calibration parameters. The embodiment attempts to image a given face at two separate distances, although in variants of the embodiment, measurements from images of different faces could be employed. In any case as in the first embodiment, two input records (TCRT, OCRT, DEST, DACD
[D1
where:
where
As before, the first condition (a), requires that the two distances should be different.
The second condition (b), requires that the object being imaged indeed exhibits the assumed dimension so that a given face to be a live human face, ed≈70 mm. In this case, the errors should not be larger than the maximum error (N*DACSTEP) and they should be quite similar (the difference should not be higher than half DACSTEP). This condition assures that ERRD
If the current face is false (condition (b) is not respected), and so compensation must not be done until a new valid face is received.
If conditions (a) and (b) are respected, them ERRPLP will be estimated as follows:
The new updated calibration parameters (which should be used further to improve AF speed and reduced lens wobble effect) are:
As indicated, the maximum estimated error should not be higher than N*DACSTEP (ERRPLP≤N*DACSTEP). The value of N can be determined by the image acquisition device or camera module manufacturer according to how tightly they wish the estimated compensation process to operate. Thus, the larger the value of N the greater the possibility of calibrating based on a poorly estimated distance to an object.
It will be appreciated that many variations of the above described embodiments are possible and that for example features and functions described in relation to the first embodiment are applicable to the second embodiment and vice versa where possible.
Number | Name | Date | Kind |
---|---|---|---|
8970770 | Nanu et al. | Mar 2015 | B2 |
20090102403 | Lule | Apr 2009 | A1 |
20090202235 | Li | Aug 2009 | A1 |
20160014404 | Krestyannikov | Jan 2016 | A1 |
20160147131 | Richards | May 2016 | A1 |
20160165129 | Malaescu | Jun 2016 | A1 |
20170155896 | Malaescu | Jun 2017 | A1 |
20180041754 | Nanu | Feb 2018 | A1 |
Number | Date | Country |
---|---|---|
3151048 | Apr 2017 | EP |
WO2016000874 | May 2015 | WO |
WO2016091545 | Jun 2016 | WO |
Entry |
---|
European Patent Office, Extended European Search Report for EP App. No. 18170272.1 dated Oct. 26, 2018 for “Method for Dynamically Calibrating an Image Capture Device”, 13 pages. |