This invention relates to methods for processing a digital image.
Global distortion, or blurring, of a picture can arise from various factors such as distortion due to out-of-focus optics and linear translation or shaking of the camera during the exposure time.
Blurring of a digital image may be described by means of a convolution:
B0(x)=∫dx′F(x′)h(x−x′) (1)
where B0(x) is the intensity of the pixel at the address x=(x,y) in the distorted picture, x being a two-dimensional vector, F(x) is the intensity of the pixel x in the undistorted image, and h(x) is the so-called point spread function (PSF) that describes the distortion. The function B0(x) is typically obtained as the output from a digital camera. The PSF for an image distorted by out-of-focus optics, for example, can be described in a first approximation by a function h that is constant inside a circle of radius r and h(x)=0 outside the circle.
Rectifying a distorted image involves determining the function F given the functions B0 and h. The convolution (1) can be Fourier transformed to yield
{tilde over (B)}0(q)={tilde over (F)}(q)·{tilde over (h)}(q) (2)
where “{tilde over ( )}” represents the Fourier transform. Hence,
In principle, therefore, F(x) may be obtained from the inverse Fourier transform of {tilde over (B)}(q)/{tilde over (h)}(q). In practice, however, this solution is characterized by a very low signal-to-noise ratio (SNR), due to amplification of noise at frequencies q at which {tilde over (h)}(q) is very small.
In the following description and set of claims, two explicitly described, calculable or measurable variables are considered equivalent to each other when the two variables are substantially proportional to each other.
The present invention provides a method for rectifying a distorted digital image B0(x) to produce a rectified image F(x). In accordance with the invention, noise is removed from the function {tilde over (B)}0(q) before applying the inverse Fourier transform to the right side of equation. (3). A noise function N is used to evaluate the amount of noise for functions {tilde over (B)}(q) that deviate slightly from {tilde over (B)}0(q) and a new function {tilde over (B)}(q) is selected that deviates slightly from {tilde over (B)}0(q) and that minimizes the noise function N. In a preferred embodiment of the invention, the noise N in an image B(x) is obtained based upon a calculation involving the gradient of the function P obtained by inverse Fourier transform of {tilde over (B)}(q)/{tilde over (h)}(q). In a most preferred embodiment, the noise N is calculated according to the equation:
N=∫∇P(x)·∇P′(x)dx (4)
where “*” indicates complex conjugate. Equation (4) may be written in the equivalent form
N=∫dq·q2∥{tilde over (D)}(q)∥2·∥{tilde over (B)}0(q)∥2 (4)
where D(x) is the so-called deconvolution filter (DCF) defined as 1/h(x), wherein h(x) is the point spread function characteristic of the distortion, and q is a two-dimensional vector in Fourier space. The rectified image F(x) may then be obtained by calculating the inverse Fourier transform of the right side of equation (3) using the thus obtained B(x). However, a pattern characteristic of the DCF may have been superimposed on the obtained F(x). This pattern originates in the DCF and is correlated with it. In the most preferred embodiment, this pattern is removed.
The invention thus concerns a method for processing a digital image B1, the image B1 being a convolution of an image F and a point spread function h, comprising removing noise from the image B1 so as to produce an image B′ of reduced noise, and calculating F based upon B′.
The invention further concerns a method for processing a deconvoluted image B, the image B having been deconvoluted according to a deconvolution filter D, the method comprising reducing correlation between the image and the deconvolution filter.
Yet still further the invention concerns a method for processing a digital image B1, the image B1 being a convolution of an image F and a point spread function h comprising the steps of:
By yet still another aspect the invention concerns a method for obtaining a radius r of a point spread function h describing an out-of-focus distortion of a digital image B, the method comprising a step of calculating a gradient at a plurality of pixels in the image B.
By a further aspect the invention concerns a program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for processing a digital image B1, the image B1 being a convolution of an image F and a point spread function h, comprising removing noise from the image B1 so as to produce an image B′ of reduced noise, and calculating F based upon B′.
Further the invention concerns a program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for processing a deconvoluted image B, the image B having been deconvoluted according to a deconvolution filter D, the method comprising reducing correlation between the image and the deconvolution filter.
Yet still further the invention concerns a program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for processing a digital image B1, the image B1 being a convolution of an image F and a point spread function h comprising the steps of.
Further the invention concerns a program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for obtaining a radius r of a point spread function h describing an out-of-focus distortion of a digital image B, the method comprising a step of calculating a gradient at a plurality of pixels in the image B.
In order to understand the invention and to see how it maybe carried out in practice, a preferred embodiment will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:
In a preferred embodiment, as shown in
In accordance with the invention, a noise function N is used to evaluate the amount of noise for functions {tilde over (B)}(q) that deviate slightly from {tilde over (B)}1(q) and a new function {tilde over (B)}(q) is selected that deviates slightly from {tilde over (B)}1(q) and that minimizes the noise function N. In a preferred embodiment of the invention, the noise N in an image {tilde over (B)}(q) is obtained based upon a calculation involving the gradient of the function P obtained by inverse Fourier transform of {tilde over (B)}(q)/{tilde over (h)}(q). In a most preferred embodiment, the noise N is calculated according to the equation:
N=∫∇P(x)·∇*(x)dx
or equivalently, N=∫dq·q2∥{tilde over (D)}(q)∥2·∥{tilde over (B)}(q)∥2 (4)
where D(x) is the deconvolution filter defined as 1/h (x), where h(x) is the point spread function characteristic of the distortion.
A function B(x) of essentially minimal noise that deviates only slightly from B1(x) may be found by evaluating the variation of N with respect to B*(q), (∂N/∂{tilde over (B)}*(q)). For example, a sequence of functions {tilde over (B)}1(q) may be generated iteratively by:
In the most preferred embodiment, N is given by (4) so that
Thus in the most preferred embodiment, (N given by (4)), for finite α, i=n for large n, and ε=α/n, {tilde over (B)}n(q) is given by:
and in the limit,
{tilde over (B)}α(q) is preferably multiplied by a factor so that it acquires the same norm as {tilde over (B)}1(q) to produce a function {tilde over (B)}(q) and a new function {tilde over (P)}1(q)={tilde over (B)}(q)/{tilde over (h)}(q) is then obtained.
F(x) may now be obtained by inverse Fourier transform of {tilde over (P)}1(q). However, a pattern characteristic of the DCF may have been superimposed on {tilde over (P)}1(q). This pattern originates in the DCF and is correlated with it. In the most preferred embodiment, this pattern is removed from the function {tilde over (P)}1(q) before applying the inverse Fourier transform and return to x-space.
In order to remove this pattern, the correlation between the pattern and the DCF is decreased, An overall correlation function C is used to evaluate the correlation between the DCP and {tilde over (P)}(q) for functions {tilde over (P)}(q) that deviate slightly from {tilde over (P)}1(q) and a new function {tilde over (P)}(q) is selected that deviates slightly from {tilde over (P)}1(q) and that minimizes the correlation function C. In a preferred embodiment of the invention, the correlation C between the DCF and a function {tilde over (P)}(q) is calculated according to the equation:
C=∫dq∥{tilde over (D)}(q)∥2·∥{tilde over (P)}(q)∥2 (7)
The variation of C with respect to {tilde over (P)}*(q), ∂C/∂{tilde over (P)}* is used to change {tilde over (P)}(q) in order to reduce C. for example, a sequence of functions {tilde over (P)}i (q) may be generated iteratively by:
In the most preferred embodiment, C is given by (7) so that
Thus, in the most preferred embodiment (C given by (7)), for finite β, i=n for large n, and ε=β/n,
and in the limit,
{tilde over (P)}β(q) is multiplied by a factor to produce a {tilde over (P)}(q) having the same norm as {tilde over (P)}1(q)·F(x) is then obtained by inverse Fourier transfer of {tilde over (P)}(q).
After removing the superimposed pattern correction, the function F(x) may be converted from the linear intensity range to an intensity space suited for the eye. However, where the original B0(x) was near 0, the slope of B0(x) was very steep so that when values of B0(x) near 0 are converted for discrete representation, much information is lost. Thus, despite the fact that the resolution of the data is nominally 8 bit, with dark colors (corresponding to low values of B), the resolution of the data is equivalent to 3 bit.
Because much information has been lost in the dark regions, converting F(x) to an intensity space suited for the eye is preferably not performed simply by applying the inverse of the transformation previously applied by the camera to B(x) in order to convert it to the linear range. Instead, image F(x) is preferably first mapped into an interval, and then the inverse transformation is applied Thus, for example, if the transformation A(B(x)/A)γ
The PSF for an image distorted by out-of-focus optics can be described in a first approximation by a function h that is constant inside a circle of radius r and h(x)=0 outside the circle. The radius r may be determined from a distorted image B(x) by the following algorithm. The algorithm makes use of the fact that in an unfocused picture, boundaries (referred to herein as “steps”) are gradual and not abrupt.
A step parallel to the y-axis at x=x0 of an image is described by a Heaviside function Θ(x−x0) independent of y. The mathematical description of the removal from focus of this step is then given by:
I(x)=∫dx′dy′h(x−x′,y−y′)Θ(x′−x0)
where the intensity I in the vicinity of the step is independent of y and h(x,y) is the PSF function. Integration with respect to y yields
The slope of the step at x=x0 is
In accordance with the invention, r is found by determining the slope s(x) at each pixel x where a step exists in the image B1(x), where s is defined as the absolute value of the gradient of B1(x) divided by the height of the step at x. The radius r(x) at x is
Edges in the image B1(x) are detected by any method of edge detection as is known in the art, for example, as disclosed in Crane, R., “Simplified Approach to Image Processing, A: Classical and Modern Techniques in C”; Chapter 3. Prentice Hall, 1996, Calculated radii r(x) for pixels x at an edge in the image B1(x) are normalized by dividing by the height of the step at the edge. The normalized radii are arranged in a histogram. The radius of the PSF is obtained essentially equal to the radius of maximum frequency in the histogram.
The radius of the PSF of the distortion was first found as follows. Edges in the image were detected and radius r(x) was calculated for each pixel x located at an edge as described above. Each calculated radius was normalized by dividing it by the height of the step at x.
{tilde over (B)}α(q) was then obtained according to equation (6) using α=0.001, and then normalized so as to have the same norm as {tilde over (B)}1(q), A function {tilde over (P)}1(q) was then obtained by applying the right side of equation (3) using {tilde over (B)}α(q) for {tilde over (B)}0(q)·{tilde over (P)}β(q) was then obtained according to equation (9) using β=0.01, and then normalized so as to have the same norm as {tilde over (P)}1(q). The function F(x) was then obtained by inverse Fourier transform of {tilde over (P)}1(q). The rectified image F(x) is shown in
It will also be understood that the system according to the invention may be a suitably programmed computer. Likewise, the invention contemplates a computer program being readable by a computer for executing the method of the invention. The invention further contemplates a machine-readable memory tangibly embodying a program of instructions executable by the machine for executing the method of the invention.
Number | Name | Date | Kind |
---|---|---|---|
4532548 | Zwirn | Jul 1985 | A |
5023641 | Merrick | Jun 1991 | A |
5535291 | Spencer et al. | Jul 1996 | A |
5580728 | Perlin | Dec 1996 | A |
5748491 | Allison et al. | May 1998 | A |
5867410 | Smallcombe et al. | Feb 1999 | A |
6240219 | Gregory | May 2001 | B1 |
6333990 | Yazici et al. | Dec 2001 | B1 |
6545714 | Takada | Apr 2003 | B1 |
6567570 | Steinle et al. | May 2003 | B1 |
20020145671 | Alon et al. | Oct 2002 | A1 |
Number | Date | Country |
---|---|---|
0 466 277 | Jan 1992 | EP |
1 079 612 | Feb 2001 | EP |
Number | Date | Country | |
---|---|---|---|
20020145671 A1 | Oct 2002 | US |