Patent Application
20110096205
This invention relates generally to the field of denoising digital images, and more particularly to a method to denoise digital images having signal-dependent noise.
Digital images captured by digital cameras are corrupted by “noise,” wherein noise can be defined as unwanted random variations in the digital image. Noise can arise from a number of sources such as sensor shot noise and fixed-pattern noise. The amount of noise in a digital image can depend on many factors such as sensor design, exposure level, and digital image processing applied to the image. Denoising algorithms are often used to “denoise” captured digital images (i.e., reduce the level of noise) in order to improve the signal-to-noise (SNR) ratio of the captured digital images. Noise in digital camera images generally depends on the signal level (i.e., the image pixel values); this type of noise is commonly referred to as “signal-dependent noise.” Traditional denoising algorithms assume the level of noise to be independent of the image pixel values. As a result, such algorithms are inadequate to deal with signal-dependent noise.
One method described by Rudin et al., in the article “Nonlinear total variation based noise removal algorithms” (Physica D, Vol. 60, pp. 259-268, 1992) uses total variation minimization to generate denoised image.
Another method taught by Foi et al., in the article “Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images” (IEEE Transactions on Image Processing, Vol. 16, pp. 1395-1411, May 2007) analyzes noisy image using a shape-adaptive discrete cosine transform (DCT).
Coifman et al., in the article “Translation-invariant de-noising” (Lecture Notes in Statistics: Wavelets and Statistics, Springer Verlag, New York, pp. 125-150, 1995) use translation invariant thresholding of the wavelet coefficients of the noisy image to produce a denoised image.
All of the above denoising algorithms assume noise to be independent of the image pixel values and are therefore inadequate to deal with signal-dependent noise.
Hirakawa et al., in the article “Image denoising for signal-dependent noise” (IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 2, pp. 29-32, 2005) teaches a rather elegant signal-dependent denoising technique. In this approach, the total least square (TLS) approach is used for modeling the uncertainties in the noisy image and to reduce the signal-dependent noise. However, this approach is computationally demanding and too slow for many applications.
Thus, there exists a need for an efficient signal-dependent denoising algorithm that preserves salient features of the image.
The present invention represents a method for producing a noise-reduced digital image captured using a digital imaging system having signal-dependent noise characteristics, comprising using a digital processor to perform at least some of the steps of:
a) capturing one or more noisy digital images of a scene, wherein said at least one noisy digital image has signal-dependent noise characteristics;
b) defining a functional relationship to relate the one or more noisy digital images to a noise-reduced digital image estimate, wherein the functional relationship includes at least two sets of unknown parameters, and wherein at least one of the sets of unknown parameters relates to the signal-dependent noise characteristics;
c) defining an energy function responsive to the functional relationship which includes at least:
d) using an optimization process to determine a noise-reduced image responsive to the energy function.
It is an advantage of the present invention that it deals with signal-dependent noise in digital images, which is more realistic for digital camera applications as compared to traditional signal-independent noise models.
It is an additional advantage that by using a spatial penalty term, the resulting noise-reduced image preserves the salient features with improved accuracy.
It is a further advantage of the present invention that it can process multiple captures of the scene having signal-dependent noise, which enables the noise to be reduced with improved accuracy.
In addition to the embodiments described above, further embodiments will become apparent by reference to the drawings and by study of the following detailed description.
The present invention will be more readily understood from the detailed description of exemplary embodiments presented below considered in conjunction with the attached drawings, of which:
The invention is inclusive of combinations of the embodiments described herein. References to “a particular embodiment” and the like refer to features that are present in at least one embodiment of the invention. Separate references to “an embodiment” or “particular embodiments” or the like do not necessarily refer to the same embodiment or embodiments; however, such embodiments are not mutually exclusive, unless so indicated or as are readily apparent to one of skill in the art. The use of singular and/or plural in referring to the “method” or “methods” and the like is not limiting.
The phrase, “digital content record”, as used herein, refers to any digital content record, such as a digital still image, a digital audio file, a digital video file, etc.
It should be noted that, unless otherwise explicitly noted or required by context, the word “or” is used in this disclosure in a non-exclusive sense.
The data processing system 110 includes one or more data processing devices that implement the processes of the various embodiments of the present invention, including the example processes of
The data storage system 140 includes one or more processor-accessible memories configured to store information, including the information needed to execute the processes of the various embodiments of the present invention, including the example processes of
The phrase “processor-accessible memory” is intended to include any processor-accessible data storage device, whether volatile or nonvolatile, electronic, magnetic, optical, or otherwise, including but not limited to, registers, floppy disks, hard disks, Compact Discs, DVDs, flash memories, ROMs, and RAMs.
The phrase “communicatively connected” is intended to include any type of connection, whether wired or wireless, between devices, data processors, or programs in which data may be communicated.
The phrase “communicatively connected” is intended to include a connection between devices or programs within a single data processor, a connection between devices or programs located in different data processors, and a connection between devices not located in data processors at all. In this regard, although the data storage system 140 is shown separately from the data processing system 110, one skilled in the art will appreciate that the data storage system 140 may be stored completely or partially within the data processing system 110. Further in this regard, although the peripheral system 120 and the user interface system 130 are shown separately from the data processing system 110, one skilled in the art will appreciate that one or both of such systems may be stored completely or partially within the data processing system 110.
The peripheral system 120 may include one or more devices configured to provide digital content records to the data processing system 110. For example, the peripheral system 120 may include digital still cameras, digital video cameras, cellular phones, or other data processors. The data processing system 110, upon receipt of digital content records from a device in the peripheral system 120, may store such digital content records in the data storage system 140.
The user interface system 130 may include a mouse, a keyboard, another computer, or any device or combination of devices from which data is input to the data processing system 110. In this regard, although the peripheral system 120 is shown separately from the user interface system 130, the peripheral system 120 may be included as part of the user interface system 130.
The user interface system 130 also may include a display device, a processor-accessible memory, or any device or combination of devices to which data is output by the data processing system 110. In this regard, if the user interface system 130 includes a processor-accessible memory, such memory may be part of the data storage system 140 even though the user interface system 130 and the data storage system 140 are shown separately in
A priori information 205 is received in receive a priori information step 204. The a priori information 205 contains the prior information about the content of a noise-reduced output image 211 that is to be produced according to the method of the present invention. The a priori information 205 will be discussed in more detail later. A define spatial mapping step 206 defines a spatial mapping 207. The spatial mapping 207 is a functional relationship relating the one or more noisy digital images 203 to the noise-reduced output image 211. A define energy function step 208 uses the spatial mapping 207 to compute an energy function 209. In the art, energy functions 209 are sometimes called optimization functions or cost functions. Finally, an optimization process step 210 uses the one or more noisy digital images 203 and the energy function 209 to produce the noise-reduced output image 211.
The individual steps outlined in
I
k(x,y)=I0(x,y)+K0+K1I0(x,y))ηk(x,y) (1)
where Ik(x, y) is a pixel value of the kth noisy digital images 203 at pixel location (x,y), ηk (x, y) is a unit amplitude random noise function for the kth noisy digital images 203 at pixel location (x,y), and I0(x, y), K0 is a constant relating to the signal-independent noise amplitude, and K1 is a constant relating to the signal-dependent noise amplitude. (This spatial mapping is similar to one defined by Hirakawa et al., in the aforementioned article “Image denoising for signal-dependent noise.)”
The spatial mapping 207 given in Eq. (1) can be rearranged as follows:
I
k(x,y)=βk(x,y)I0(x,y)+K0ηk(x,y) (2)
where βk(x, y)=1+K1ηk (x, y) incorporates the signal-dependent noise characteristics. The individual equations for each of the noisy digital images 203 can be combined into vector form to yield:
I
(x,y)=β(x,y)I0(x,y)+K0η(x,y) (3)
where I(x, y) is a vector of the pixels from the one or more noisy digital images 203 at pixel location (x,y), η(x, y) is a function parameter vector of the random noise functions at pixel location (x,y), and β(x, y) is a function parameter vector relating to the signal-dependent noise characteristics The function parameter vector β(x, y) can be expressed by the following equation:
β(x,y)=1+K1η(x,y) (4)
where 1 is a vector containing all 1s.
DF(x,y)=(β(x,y)I0(x,y)−I(x,y))t(β(x,y)I0(x,y)−I(x,y)) (5)
where DF(x, y) is the value of the data fidelity term 303 at pixel location (x,y), and (•)t denotes a transpose operation.
A define spatial fidelity term step 304 uses the a priori information 205 to define a spatial fidelity term 305. The spatial fidelity term 305 encourages sharp edges in the noise-reduced output image 211 (
where SF(x, y) is the value of the spatial fidelity term 305 at pixel location (x,y), and ∇I0(x, y) is the gradient of the noise-reduced output image 211 (
A construct energy function step 306 combines the data fidelity term 303 and the spatial fidelity term 305 to construct the energy function 209. The construct energy function step 306 can be performed in any appropriate way known to those skilled in the art. One way to construct an energy function 209 with the construct energy function step 306 according to a preferred embodiment of the present invention can be described using the following equation:
where g(I0(x, y), β(x, y), λ, α, γ) is the energy function 209, and λ, α and γ are weighting parameters. The weighting parameter λ is used to weight the data fidelity term, and the weighting parameter α s used to weight the spatial fidelity term. The weighting parameter γ is used to weight an optional third term which encourages β(x, y) to be smooth.
The optimization process step 210 of
An estimate scaled noise images step 404 uses the intermediate noise-reduced image 403 and the one or more noisy digital images 203 to produce a corresponding set of one or more scaled noise images 405. The scaled noise images 405 can be computed in any appropriate way known to those skilled in the art. One way to compute the scaled noise images 405 according to a preferred embodiment of the present invention can be described using the following equation:
where SN(x, y) is a vector of the pixel values of the scaled noise images 405 at pixel location (x,y), IINR(x, y) is the value of the intermediate noise-reduced image 403 at pixel location (x,y) and I(x, y), λ, and γ have been defined earlier. This equation follows from the definition of the energy function defined in Eq. (7) using derivations that are well-known in the optimization art. It can be obtained by taking the derivative of Eq. (7) with respect to β(x, y), then setting it equal to zero and solving for β(x, y).
An estimate noise reduced image step 406 uses the intermediate noise-reduced image 403, the one or more scaled noise images 405 and the one or more noisy digital images 203 to produce a noise-reduced image 407. The noise-reduced image 407 can be computed in any appropriate way known to those skilled in the art. One way to compute the noise-reduced image 407 according to a preferred embodiment of the present invention can be described using the following equation:
where INR(x, y) and IINR(x, y) are pixel values of noise-reduced image 407 and intermediate noise-reduced image 403, respectively, at pixel location (x,y), τ and α are empirically determined weighting parameters, “•” represents the dot product, “|•|” is the absolute value operator, and V is the gradient operator. This equation also follows from the definition of the energy function defined in Eq. (7). It can be obtained by taking the derivative of Eq. (7) with respect to I0(x, y), then setting the derivative equal to zero and applying a gradient descent method
A define convergence criterion step 408 specifies a convergence criterion 409 to terminate the algorithm. The convergence criterion 409 can be determined in any appropriate way known to those skilled in the art. In a preferred embodiment of the present invention, the convergence criterion 409 is satisfied when the algorithm is repeated for a predetermined number of iterations. Alternate forms of convergence criteria are well known to those skilled in the art. As an example, the convergence criterion 409 can specify that the algorithm is terminated if the mean square difference between the intermediate noise-reduced image 403 and the noise-reduced image 407 is less than a predetermined threshold. Alternatively, the convergence criterion 409 can specify that the algorithm is terminated if the mean square difference between the intermediate noise-reduced image 403 and the noise-reduced image 407 is less than a predetermined threshold, but is terminated after the algorithm is repeated for a predetermined number of iterations even if the mean square difference condition is not satisfied.
The convergence test 410 checks whether the convergence criterion 409 is satisfied. If the convergence criterion 409 is satisfied then the algorithms is terminated and the noise-reduced image 407 is selected as the final noise-reduced output image 211. Otherwise, the intermediate noise-reduced image 403 is set to be equal to the noise-reduced image 407 and the entire process is repeated until the convergence criterion is satisfied.
In a preferred embodiment of the present invention, a plurality of noisy digital images 203 are used to determine the noise-reduced output image 211. This has the advantage that it provides multiple instances of the image noise, and therefore provides information that is useful to reduce the noise. The plurality of images is preferably captured sequentially during a short time interval in order to minimize any relative motion between the digital camera and objects in the scene. In the case where there may be some amount of relative motion between the times that the noisy digital images 203 are captured, the noisy digital images 203 can be spatially registered first and the proposed invention can then be applied to the registered images. The image registration can be performed in any appropriate way known to those skilled in the art. In a preferred embodiment of the present invention, the image registration is performed by selecting one of the noisy digital images 203 as a reference digital image and applying translation, rotation or scaling operations to the other noisy digital images 203 to align them with the reference digital image. In cases where the images contain moving objects, only portions of the digital images may be misaligned. In this case, one or more subsets of image pixels that are misaligned can be determined and appropriate translation, rotation or scaling operations can be applied to each subset of image pixels.
In another embodiment of the present invention, only a single noisy digital image 203 is used in the determination of the noise-reduced output image 211. The same basic methods described with reference to
It is to be understood that the exemplary embodiments disclosed herein are merely illustrative of the present invention and that many variations of the above-described embodiments can be devised by one skilled in the art without departing from the scope of the invention. It is therefore intended that all such variations be included within the scope of the following claims and their equivalents.
