None.
The present invention relates generally to a multi-layered image enhancement technique.
A multitude of different techniques have been developed for processing and filtering two-dimensional images and two-dimensional video sequences. In particular, many image processing techniques have been developed to modify a relatively low resolution image to a relatively high resolution image or otherwise enhancing the existing resolution, while maintaining image clarity without the introduction of excessive artifacts.
If an image is blurred or degraded by a well-understood process, such as shot noise occurring during transmission, the image can usually be enhanced by developing a model of the source of degradation, then reconstructing the original image using the model. However, in many circumstances, a source of degradation of the image cannot be modeled and, hence, the image cannot be faithfully reconstructed. Thus, generic image enhancement is problematic since the type of noise content therein may be hard to determine or otherwise characterize.
What is desired therefore is a multi-stage non-linear enhancement technique that is suitable for a variety of different types of noise content.
The foregoing and other objectives, features, and advantages of the invention may be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.
Referring to
Referring to
The input frame 200 is provided to a non-linear smoothing filter λ1202. As a general matter, non-linear filters tend to be more effective than linear filters, but likewise non-linear filters tend to be more destructive than linear filters if not suitably controlled. The non-linear smoothing filter λ1202 is suitable to reduce the amount of noise in the image. Preferably, the non-linear smoothing filter λ1 is suitable to reduce limited levels of noise in the image, which is suitable for image content that has quite limited amounts of noise. In this manner, the image content is not significantly degraded as a result of the application of the non-linear smoothing filter λ1 for image content with quite limited amounts of noise. The non-linear smoothing filter λ1 provides an output base layer λ1204 which is the result of the smoothing filter.
The input frame 200 is provided to a non-linear smoothing filter λ2210. The non-linear smoothing filter λ2210 is suitable to reduce the amount of noise in the image. Preferably, the non-linear smoothing filter λ2 is suitable to reduce medium levels of noise in the image, which is suitable for image content that has texture based amounts of noise which is a greater amount of noise than the quite limited amounts of noise. In this manner, the image content is not significantly degraded as a result of the application of the non-linear smoothing filter λ2 for image content with texture based amounts of noise. The non-linear smoothing filter λ2 provides an output base layer λ2212 which is the result of the smoothing filter.
The input frame 200 is provided to a non-linear smoothing filter λ3220. The non-linear smoothing filter λ3220 is suitable to reduce the amount of noise in the image. Preferably, the non-linear smoothing filter λ3 is suitable to reduce significant levels of noise in the image, which is suitable for image content that has significant amounts of noise which is a greater amount of noise than the quite limited amounts of noise and a greater amount of noise than the texture based amounts of noise. In this manner, the image content is not as significantly degraded as a result of the application of the non-linear smoothing filter λ3 for image content with such significant amounts of noise. The non-linear smoothing filter λ3 provides an output base layer λ3222 which is the result of the smoothing filter. For example, the output base layer λ3222 may resemble a cartoon image.
The input frame 200 may bypass 224 the non-linear smoothing filters 202, 210, 220. In this manner, a reference image is maintained that does not have any (or an insignificant amount) of filtering applied thereto. In this manner, the image content is not degraded (or insignificantly degraded) as a result of the application of one of the non-linear smoothing filters.
Additional non-linear smoothing filters λn may be included that have different amounts of noise removal, as desired. One or more linear filters may be included, if desired. As it may be observed, each of the different non-linear smoothing filters provides a different amount of smoothing to the same input frame 200. In this manner, the system has available to it a set of output images that have different amounts of smoothing applied thereto, without having to pre-process the image content to determine which particular smoothing filter should be applied at the exclusion of other smoothing filters. In addition, the arrangement of a plurality of different non-linear smoothing filters, each processing the same video frame during overlapping time periods, is suitable for parallel processing.
It is desirable to determine the amount of smoothing that was applied by each smoothing filter. In particular, it is desirable to determine the amount of smoothing that was applied by a particular non-linear smoothing filter relative to one or more of the other non-linear smoothing filters and/or the original input image.
To determine the amount of smoothing applied to the input frame 224 with respect to the output 204 of the non-linear smoothing filter λ1202 may be determined by using a subtraction operation 230. The output of the subtraction operation 230 is a detail layer 1 image 240. The detail layer 1 image 240 includes the differences between the input frame 224 and the output 204 of the non-linear smoothing filter λ1202.
To determine the amount of smoothing applied to the input frame 224 with respect to the output 212 of the non-linear smoothing filter λ2210 may be determined by using a subtraction operation 232. The output of the subtraction operation 232 is a detail layer 2 image 242. The detail layer 2 image 242 includes the differences between the output 204 of the non-linear smoothing filter λ1202 and the output 212 of the non-linear smoothing filter λ2210.
To determine the amount of smoothing applied to the input frame 224 with respect to the output 222 of the non-linear smoothing filter λ3220 may be determined by using a subtraction operation 234. The output of the subtraction operation 234 is a detail layer 3 image 244. The detail layer 3 image 244 includes the differences between the output 222 of the non-linear smoothing filter λ3220 and the output 212 of the non-linear smoothing filter λ2210.
Additional comparisons may be included relative to different images, combinations of images, filtered and non-filtered images, as desired. As it may be observed, each of the different subtraction operations 230, 232, 235, identifies a different amount of noise that was removed as a result of one or more non-linear smoothing filters. Thus, the technique indicates different amounts of noise that exists as a result of different amounts of applied smoothing. In this manner, the system has available to it a set of output images indicating the effects of the different amounts of smoothing applied thereto, without having to pre-process the image content to determine a particular smoothing filter to be applied at the exclusion of other smoothing filters. In addition, the arrangement of a plurality of different non-linear smoothing filters, each using the same input frame, is suitable for parallel processing. Also, the arrangement of a plurality of different subtraction operations, each using a different sets of input, are suitable for parallel processing, and at least partially temporally overlapping processing for the same input frame. Additional subtraction operations (or otherwise) may be included, if desired.
The detail layer 1 image 240 may be provided to a non-linear smoothing filter σ1250. The non-linear smoothing filter al 250 reduces the noise in the resulting detail layer 1 based upon the image content, noise levels, etc. The output of the non-linear smoothing filter σ1250 is a smoothed layer 1260.
The detail layer 2 image 242 may be provided to a non-linear smoothing filter σ2252. The non-linear smoothing filter σ2252 reduces the noise in the resulting detail layer 2 based upon the image content, noise levels, etc. The output of the non-linear smoothing filter σ2252 is a smoothed layer 2262.
The detail layer 3 image 244 may be provided to a non-linear smoothing filter σ3254. The non-linear smoothing filter σ3254 reduces the noise in the resulting detail layer 3 based upon the image content, noise levels, etc. The output of the non-linear smoothing filter σ3254 is a smoothed layer 3264.
The non-linear smoothing filters 250, 252, and 254 are preferably different filters suitable for the corresponding non-linear smoothing filters 202, 210, and 220. In some cases, the non-linear smoothing filters 250, 252, and 254 are the same filter. The non-linear smoothing filters an are suitable for parallel processing, where the same input image may be at least partially be temporally processed in parallel. Additional non-linear smoothing filters may be included, if desired.
The output of the non-linear smoothing filters 250, 252, 254 may be provided to an adjustment process 270. The adjustment process 270 may be, for example, a gamma correction. The adjustment process 270 may result in a noise reduction because primarily (or substantially) only the residual noise content is being adjusted. For example, the adjustment process 270 may selectively not amplify smaller values to the same extent as larger values. In addition, adjustment of the residual noise facilitates selective limiting the amount of the enhancement so that a suitable multiplier can be limited for stronger details to reduce the effects of clipping. The adjustment process 270 may provide the same adjustment to each of the outputs of the non-linear smoothing filters 250, 252, and 254 or it may be different for one or more of such outputs.
The output of the adjustment process 270 may be corresponding adjusted images 272, 274, 276 for each of the input images 260, 262, 264. With each of the adjusted images 272, 274, 276 being a portion of the original input frame 200 they may be summed together to provide the noise related image content that was subtracted from the original input frame 200. In addition, the output of the non-linear smoothing filter λ1204 is also added to the output of the adjustment process 270 to provide an enhanced output image 280.
Any suitable filer or a selection of different filters may be used for the non-linear smoothing filters 202, 210, 220. For example, the non-linear smoothing filters may be a bilateral filter, a weighted least square filter, and/or a guided filter. In general, the filters preferably reduce the modification of image content near an edge relative to areas that are not as near to such an edge. In this manner, the output of the adjustment process 270 near an edge is small or near zero since there is limited need for changes in regions proximate an edge. Thus for regions near an edge the output of the adjustment process 270 is zero or otherwise near zero and therefore the adjustment process 270 will not significantly boost such edges. Without excessively boosted edges the overshooting problem near edges will not substantially occur.
The preferred implementation of the system uses a 1-dimensional operation on the image content.
Referring to
Preferably, the edge preserving filter 410 is implemented as follows:
filter[x]=filter[x]/(weight_left+weight_right+4), where the weights are calculated as:
Preferably, filter[x] may be the output of the process at location x. Preferably, input [x] is the input to the process at location x.
LUT(x) may be 2 when x is less than a threshold, 1 when x is less than 2*Threshold (but not less than said Threshold) and 0 if x is greater than or equal to 2*Threshold.
Preferably in one embodiment the enhancement process 430 is computed a multiplication process. In another embodiment the enhancement process 430 is computed as a multiplication and a clip
Preferably, the combining process 440 may be in addition to the output of the enhancement process 430, which may include additional enhancements, where the combining process 440 includes an additional clipping operation.
The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.
Number | Name | Date | Kind |
---|---|---|---|
7529425 | Kitamura et al. | May 2009 | B2 |
7898583 | Wakahara et al. | Mar 2011 | B2 |
RE43360 | Demos | May 2012 | E |
8265409 | Katagiri et al. | Sep 2012 | B2 |
8369640 | Ishiga | Feb 2013 | B2 |
8374457 | Wang et al. | Feb 2013 | B1 |
20010007478 | Kim | Jul 2001 | A1 |
20070071355 | Imai | Mar 2007 | A1 |
20080019605 | Yea et al. | Jan 2008 | A1 |
20080239090 | Ohwaki et al. | Oct 2008 | A1 |
20090245679 | Ohwaki et al. | Oct 2009 | A1 |
20110135217 | Su et al. | Jun 2011 | A1 |
20120062797 | Segall | Mar 2012 | A1 |
20120093432 | Akiyama | Apr 2012 | A1 |
20140241646 | Seo et al. | Aug 2014 | A1 |
Entry |
---|
E. Wharton, K. Panetta. and S. Agaian, “Edge preserving image enhancement using anisotropic diffusion”, Proc. SPIE, vol. 6812, 2008. |
Z. Farbman et al., “Edge-Preserving Decompositions for Multi-Scale Tone and Detail Manipulation,” ACM Transactions on Graphics 27(3) (Proc. ACM SIGGRAPH 2008), Aug. 2008, 10 pgs. |
K. He et al., “Guided Image Filtering,” The 11th European Conference on Computer Vision (ECCV), 2010, 14 pgs. |
C. Tomasi et al., “Bilateral Filtering for Gray and Color Images,” IEEE International Conference on Computer Vision, 1998, 8 pgs. |
H. Takeda et al., “Kernel Regression for Image Processing and Reconstructions,” IEEE Transactions on Image Processing, vol. 16, No. 2, Feb. 2007, pp. 349-366. |
A. Buades et al., “A Review of Image Denoising Algorithms, with a New One,” Multiscale Modeling and Simulation (SIAM Interdisciplinary journal), vol. 4, No. 2, 2005, pp. 490-530. |
Number | Date | Country | |
---|---|---|---|
20140241642 A1 | Aug 2014 | US |