This application claims the benefit of Taiwan application Serial No. 98102331, filed Jan. 21, 2009, the subject matter of which is incorporated herein by reference.
1. Field of the Invention
The invention relates in general to a restoration method for restoring a motion blurred image, and more particularly to a method of restoring a motion blurred image using bi-level regions.
2. Description of the Related Art
In the modern age, in which the technology is changing with each passing day, photographing an image (e.g., motion image) using a digital camera, a digital still camera or a mobile phone with a photographing function has become a general behavior in the daily life of the modern human beings.
In photographing, the user may fix the digital still camera to a foot stand to stabilize the digital still camera and prevent the digital still camera from shaking, in order to obtain the better image quality. A foldable foot stand is available so that the inconvenience of carrying the foot stand can be reduced, but the user may feel inconvenient in carrying the foot stand. However, the blurred image may be obtained in photographing due to shake of user's hand if the digital still camera is not fixed to the foot stand. Thus, a restoration method for restoring blurred images is needed to restore the blurred image for getting high quality image.
An example of the application is directed to an image restoration method for restoring a blurred image; and a motion blur kernel is estimated based on an intrinsic bi-level image region of the blurred image. During the blind image restoration, blur kernel is estimated and image is restored.
Another example of the application is directed to an image restoration method for iteratively obtaining a blur kernel in estimate of the blur kernel. A kernel prior is introduced in estimate of the blur kernel, to stabilize the solution.
Still another example of the application is directed to an image restoration method for restoring an image according to image priors, which is advantageous to the achieving of the better restoration result and can reduce the ringing artifact.
According to examples of the present invention, an image restoration method applied to an image acquisition device is provided. The method includes: receiving a blurred image; selecting at least one first image region of the blurred image; performing thresholding and brightness compensation on the first image region to obtain a second image region; estimating a blur kernel of the blurred image based on the first image region, the second image region and a first image prior; restoring the second image region based on the blur kernel, a second image prior and a third image prior; and restoring the blurred image based on the blur kernel and the third image priors and outputting the restored blurred image if the restored second image region is converged.
The invention will become apparent from the following detailed description of the preferred but non-limiting embodiments. The following description is made with reference to the accompanying drawings.
In an embodiment of the invention, a motion blurred image is restored, wherein a blur kernel may be estimated based on an intrinsic bi-level image region of the blurred image to perform restoration. In the blind image restoration, the blur kernel is estimated and the image is restored. In estimate of the blur kernel, the blur kernel is iteratively obtained. The image region is restored according to image priors, which are advantageous to the achieving of the better restoration result so that the ringing artifact can be reduced. In addition, the kernel prior is introduced in estimate of the blur kernel, to stabilize the solution. In here, the so-called blind image restoration represents that the blur kernel is unknown when the blurred image is inputted. That is, the blur kernel is estimated from the inputted blurred image.
Then, in step 115, at least one blur bi-level image region is selected form the blurred image. The blur bi-level image region is a blurred image region which only contains two colors and their mixtures. Herein, the blurred image region is a rectangular image region, for example. The blurred image region is transferred to gray level.
Next, in step 125, the image region is restored. Before a blur kernel is estimated, the result obtained in the step 120 is the result obtained in the step 125, that is, the image region obtained after thresholding is regarded as the image obtained after the first restoration.
Next, in step 130, it is judged whether the restored image region converges. If not, the procedure goes to step 135. On the contrary, if yes, the procedure goes to step 145.
In the step 135, the blur kernel is estimated based on the blurred region (selected in the step 115), and the image region (obtained after the restoration in the step 125). In the embodiment, the estimated blur kernel is a two-dimensional gray-level image (or a rectangular image), which represents the track of camera shaking.
Thereafter, in step 140, the image region is restored based on the estimated blur kernel.
Then, the procedure goes back to the step 130 to judge whether the restored image region converges.
If the restored image region converges, which means that the estimated blur kernel is unchanged, in the step 145, the overall blurred image is restored based on the estimated blur kernel. Next, in step 150, the restored image is outputted. Taking a digital still camera as an example, the restored image is what viewed by user on a display of the digital still camera, and the original blurred image will not be viewed by the user.
In addition, in the embodiment of the invention, the blur kernel is iteratively estimated until the restored region converges, which is only one of many possible implementations. Other methods may also be adopted. For example, the blur kernel is iteratively estimated until the estimated blur kernel converges; or the iteration times are restricted.
Now, how to estimate the blur kernel and restore the region (and the overall image) in the embodiment will be described in the following.
In the embodiment, the blur kernel estimation and the bi-level image restoration are merged in the MAP (Maximum A Posteriori) formula using a possibility model. The so-called bi-level image represents the image region after thresholding. As shown in
In the embodiment, the image restoration is based on the Bayesian possibility model. Based on the Bayesian possibility model and the assumption that the restored image F is independent from the blur kernel H, the posteriori P(F,H|G) may be written as follows:
P(F,H|G)∝P(G|F,H)P(F)P(H) (1),
wherein P(G|F, H) represents the likelihood, P(F) represents the prior of the restored image F and P(H) represents the prior of the blur kernel H. Herein, it is to find the restored image F and the blur kernel H, which make a maximum posteriori P(G|F, H). G represents the blurred image. In this embodiment, the restored image F is what obtained after the restoration step 125 of
If the restored image F and the blur kernel H are given, then the likelihood of the blurred image G may be determined according to the image noise N=G−FH. More particularly, it is assumed that the image noise N is in independent and identically distribution (i.i.d.), and the distribution of the gray levels of the pixels may be emulated as the Gaussian distribution. Under this assumption, the likelihood P(G|F, H) may be represented as:
wherein η represents the standard deviation of the Gaussian distribution, ∥∥ represents the 2-norm operator, ∥G−FH∥ is a fidelity constraint which represents the difference between the blurred image and the convolution F
H. The convolution F
H is the convolution of the (restored) image F with the estimated blur kernel H. That is, the blurred image may be represented by the convolution of the (restored) image F with the estimated blur kernel H.
The blind image restoration is an ill-posed problem. In order to overcome the illness, the image prior is introduced into the image restoration in this embodiment. In the model of this embodiment, the image prior P(F) of the restored image F includes two components. That is, P(F) may be represented as follows:
P(F)∝Pb(F)Ps(F) (3),
wherein Pb(F) represents a bi-level prior and PS(F) represents a sparse prior. Details of the two image priors will be described in the following.
In the embodiment, after the image F is normalized, Pb(F) may be represented as:
wherein λ1 represents a rate parameter, Fi represents a image intensity of pixels of the image F after the image F is normalized. In normalization, the image intensity of pixels of the image F is linearly transformed to a zone ranging from −1 to 1 (black color corresponds to −1, and white color corresponds to 1). ΩF represents a set of all pixels in the image F. It is to be noted that the parameter |1−Fi2| preferably approaches 0. That is, after the restored image F is normalized, the image intensity of each pixel is preferably equal to 1 or −1. That is, it is preferred that the restored image only includes two colors (white and black).
In the embodiment, based on observation, the differentiation of the bi-level image also follows the heavy-tailed distribution. This represents that the image gradient is almost equal to zero or a very small value. Thus, the sparse prior Ps(F) may be modeled, according to the heavy-tailed function, as follows:
wherein λ2 represents a rate parameter, x and y respectively represent horizontal differentiation and vertical differentiation, and function Φ represents the heavy-tailed function. Herein, it is assumed that Φ(x)=|x|0.8.
To stabilize the solution, the kernel prior P(H) is defined as follows:
wherein λ3 represents a rate parameter, ΩH represents the set of all pixels in the blurred kernel.
Synthesizing the above-mentioned formulas, the posteriori has to be maximized to obtain high quality restored image during the image restoration:
If the MAP formula is given, in order to solve the formula (7), a logarithm value of the right portion on the formula (7) is obtained and then a negative value of the logarithm value is obtained, which may be represented as:
wherein α1=2η2λ1, α2=2η2λ2, and α3=2η2λ3.
Thus, the solution for the maximized posteriori formula (7) is transformed into a solution for a minimum of E(F,H) in the formula (8). That is, if the minimum of E(F,H) can be obtained, which means the solution for the maximized posteriori formula (7) is obtained, then a high quality image restoration result is obtained. The minimum of E(F,H) may be obtained as follows. An optimum ideal image F is obtained assuming the blur kernel H is given. In addition, an optimum blur kernel H is obtained assuming the ideal image F is given. The methods will be described in the following.
Obtain the Optimum Ideal Image F when the Blur Kernel H is Given
Assuming that the currently estimated blur kernel H is fixed (i.e., the kernel H is regarded as known), the formula (8) is minimized to obtain the optimum ideal image F.
Assuming that the image noise N is neglected (i.e., the image noise is set as 0), N=G−FH may be rewritten, and the convolution is regarded as a matrix multiplication:
G=FH
g=Chf (9),
wherein Ch is an L×L two-dimensional matrix and is determined by the estimated blur kernel, and f and g are one-dimensional vectors which represent the image F and the blurred image G. Herein, L is a product of the height and the width of the blurred image. After the constant item (known item) is removed, E(F,H) may be simplified as EF(f) and represented as follows:
wherein Cgh and Cgv represent matrixes determined according to the horizontal derivative filter [1, −1] and the vertical derivative filter [1, −1].
In order to minimize EF(f) of the formula (10), we derivate the formula (10) with respect to f by setting Φ(x)=|x|2. Consequently, the derivative of the formula (10) with respect to f is taken and set to be equal to 0 so as to obtain a set of linear equations as follows:
wherein Wkh0 and Wkv0 are both a diagonal weighting coefficient matrix, whose initialization is an identity matrix, and Wb is a diagonal binary mask and may be represented as follows:
W
b(i,i)=1 when −1<fi<1 (12A)
W
b(i,i)=−1 otherwise (12B)
The formula (II) is calculated using the conjugate gradient method. Next, an iterative re-weighted least squares process (IRLS) is adopted to optimize the formula (II). The re-weighting term may be represented as follows:
W
kh
t=diag(max(Cghf,ε)0.8-2) (13A)
W
kv
t=diag(max(Cgvf,ε)0.8-2) (13B)
wherein t represents the tth iteration in IRLS, and the parameter ε prevents the division by zero. Parameters α1 and α2 are reduced over iteration.
Obtain the Optimum Blur Kernel H when the Ideal Image F is Given
Next, how the optimum blur kernel H is determined with respect to the given ideal image F in this embodiment will be described. Similarly, the image noise N is neglected, G=FH
g=Ah is written, and the convolution is regarded as a matrix multiplication:
G=FH
g=Ah (14),
wherein the matrix A is composed of the image F, h represents an one-dimensional vector of K2, K represents a dimension (length or width thereof) of the blur kernel, and g represents an one-dimensional vector of the blurred image G. For the given image F, E(F,H) may be simplified as EH(h) represented as follows:
E
H(h)=∥g−Ah∥22+α3∥h∥22 (15),
In order to minimize EH(f) of the formula (15), the derivative of the formula (15) with respect to h is taken and set to be equal 0 so as to obtain a set of linear equations as follows:
wherein I is an identity matrix.
In summary, the embodiment of the invention iteratively estimates the blur kernel based on the blurred image to obtain the optimum blur kernel. Then, the estimated blur kernel is utilized to restore the blurred image. In estimate of the blur kernel, image priors may be introduced. These image priors are advantageous to the achieving of a better restoration result, and can reduce the ringing artifact.
While the invention has been described by way of example and in terms of a preferred embodiment, it is to be understood that the invention is not limited thereto. On the contrary, it is intended to cover various modifications and similar arrangements and procedures, and the scope of the appended claims therefore should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements and procedures.
Number | Date | Country | Kind |
---|---|---|---|
098102331 | Jan 2009 | TW | national |