Patent Application
20120274818
This application is based upon and claims the benefit of priority from prior Japanese Patent Application No. 2011-099750, filed Apr. 27, 2011, the entire contents of which are incorporated herein by reference.
Embodiments described herein relate generally to an image processing apparatus and an image processing method which handle stereo images and multiple images.
Various image processing methods of finding or trying to find the correspondence between pixels in stereo images or multiple images have been proposed.
As for stereo images or multiple images, a method of removing imaging-device-generated noise in image signals cannot be said to have been established sufficiently. Methods of removing noise efficiently and of alleviating a bad effect on image quality are particularly far from having been established sufficiently.
A general architecture that implements the various features of the embodiments will now be described with reference to the drawings. The drawings and the associated descriptions are provided to illustrate the embodiments and not to limit the scope of the invention.
Various embodiments will be described hereinafter with reference to the accompanying drawings. In general, according to one embodiment, an image processing apparatus comprises: a calculation module configured to find the correspondence between an arbitrary pixel of a first image input as a first image signal and an arbitrary pixel of a second image input as a second image signal acquired at the same time as the first image signal about the same subject; a determination module configured to compare the arbitrary pixel of the first image with the arbitrary pixel of the second image based on the calculation result of the calculation module and to determine whether the individual pixels are noise pixels or non-noise pixels differing from noise pixels; and a noise removal module configured to correct noise pixels based on the determination result of the determination module.
Embodiments will now be described hereinafter in detail with reference to the accompanying drawings.
An image processing apparatus 11 shown in
In many cases, the image processing apparatus 11 includes a parallax information creation module 31 and a parallax image creation module 33 in a subsequent stage (in a downstream) of the noise removal module 17. The parallax information creation module 31 and parallax image creation module 33 convert an input image into a right viewpoint output image signal and a left viewpoint output image signal which a display device can display and output the converted signals to image signal output terminals or output interfaces. The image processing apparatus 11 and the display device may be integrally formed to enable stereo images to be displayed (e.g., like a “portable information terminal device” integral with a display device).
The input image includes right-eye data (image output) and left-eye data (image output) produced by a stereo camera 21 which includes at least two cameras or two lenses separated right and left a specific distance apart and a camera.
The input image may be a multiple image obtained by preparing three or more cameras or a single camera with three or more lenses and taking an image with the cameras or the single camera. The input image may be either a still image or a moving image (a non-static image). When the input image is a moving image, the individual still images constituting a moving image have only to have been taken at the same time that enables a comparison to be made at the corresponding pixel calculation module 13. That is, the input image has only to be such that two or more images have been taken from the same subject at the same instant and may not be restricted by an imaging method. As long as the input images are two or more images taken from the same subject, they can be applied even if they differ in size (magnification).
In addition, the input image is not limited to the output from a camera. For instance, it may be an image signal input to an input terminal of a television device (television receiver/image display device) or a recorder (video reproducing device). In this case, noise is removed (as described later), taking into account that noise components produced by a signal process, such as compression, are superimposed on all the images to be compared, which enables the same effect to be expected.
The noise determination module 15 will be explained in detail with reference to
If having determined that some or all of the images (one image when there are two images) have noise, the noise determination module 15 forms a noise mask that shows noise information so as to enable the noise removal module 17 in a subsequent stage to use the information. In this proposition, suppose, for example, a “bright point component” in a black image or a monochromatic image of a specific concentration, a “high-concentration or chromatic color component” in a white image or in a low-concentration area lower than a specific concentration”, a “sporadic component” differing from peripheral components in a creation process (or formation condition), or multiple components of them are treated as noise. Therefore, when a noise mask is formed, even if, for example, hair or the like has adhered to the lens on a camera, it is desirable that the noise determination module 15 should have a “(non-noise) dictionary” which is referred to in order to determine that what has adhered to the lens is not noise or is non-noise.
The noise removal module 17 will be explained in detail with reference to
As described above, the individual images (input images) from which noise has been removed by the noise removal module 17 on the basis of the determination result of the noise determination module 15 are output as an output right-eye image and an output left-eye image, while the positional relationship at the time of inputting is being kept. That is, the input image from the right-eye camera (or for the right eye) is output as an output right-eye image to a subsequent stage and the input image from the left-eye camera (or for the left eye) is output as an output left-eye image to a subsequent stage [24].
Hereinafter, n (a non-zero integer) is used for a pixel in an arbitrary position (x, y) of a base image and [SAD|x2, y2] is determined as the sum of absolute differences (SAD) from a (4-valued) rectangle represented by (x−n, y−n), (x−n, y+n), (x+n, y−n), (x+n, y+n) and from (x2−n, y2−n), (x2−n, y2+n), (x2+n, y2−n), (x2+n, y2+n) for x2, y2 satisfying “0≦x2<reference image width”, “0≦y2<reference image height”[32]. Then, “SAD minimum value” and “(X, Y)” that minimizes [SAD|X, Y] are calculated, i.e., “calculating a reference image and an SAD for window with a center pixel of each of pixel of the base image (SAD calculations for a window centering on each pixel of the standard image with reference to the reference image)” are executed [33].
If the calculate “SAD minimum value” is less than threshold value [34-YES], “(X, Y)” is determined to be the corresponding pixel on the reference image for the base image (x, y), i.e., “determining a corresponding pixel for the reference is the minimum SAD image (SAD minimum pixel is corresponding pixel on reference image) [35]. If the “SAD minimum value” is greater than or equal to threshold value t [34-NO], it is determined that there is no corresponding pixel [36].
The above corresponding point calculation is done for all the pixels (x, y) on the reference image that satisfy the expressions “0≦x<base image width”, “0≦y<base image height”.
Next,
Specifically, the left-eye image (reference image), the right-eye image (base image), and pixel correspondence information found from
Let a noise value be the sum of absolute difference between (x, y) and each of surrounding pixels (x−1, y), (x+1, y), (x, y−1), and (x, y+1).
Of noise value A (base image) of pixel (x, y) and noise value B (reference image) of (X, Y), the greater one of the noise values is determined to be a single noise pixel ([single (noise)]) and the smaller one is determined to be a non-noise pixel ([non (noise)] [115].
If both noise value A and noise value B are greater than threshold value T1, they are determined to be both noise pixels ([both (noise)] [133-YES]. Both noise value A and noise value B are less than threshold value T2, they are determined to be both non-noise pixels ([both non (noise)] [120].
Hereinafter, the noise determination is made for all the pixels (x, y) and pixels (X, Y) between which the aforementioned correspondence relationship holds and a noise mask is formed for each of the left-eye image and right-eye image and output [119]. Even if it has been determined that they are both non-noise pixels ([both non (noise)]), noise masks are formed because they are identified as being pixels determined by the corresponding pixel calculation module 13 to have no corresponding pixel (pixels failing to find any corresponding pixel).
Specifically, as shown in
Hereinafter, each of noise value A and noise value B is compared with threshold value T1 [113]. If A>T1 and B>T1 [113-YES], it is determined that A and B are both noise pixels ([both (noise)]) [114] and a noise mask is set [119].
Each of noise value A and noise value B is compared with threshold value T1 [113]. If A≦T1 and B≦T1 [113-NO], each of noise value A and noise value B is compared with threshold value T2 [115].
If the relationship between each of noise value A and noise value B and threshold value T2 satisfies the expressions A<T2 and B<T2 [115-YES], both A and B are non-noise.
If the relationship between each of noise value A and noise value B and threshold value T2 satisfies the expressions A≦T2 or B≧T2 (excluding A<T2 and B<T2), at least one is a noise image and the rest are non-noise images [115-NO].
Hereinafter, if the relationship between each of noise value A and noise value B and threshold value T2 satisfies the expressions A≦T2 or B≧T2 ([115-NO]), it is determined whether noise value A and noise value B satisfy the expression A<B [116]. If A<B [116-YES], it is determined that the left-eye pixel (pixel value B) is noise ([single (noise, left eye)][117]. Therefore, if A≦B [116-NO], the right-eye pixel (pixel value A) is noise ([single (noise), right eye]) [118].
Therefore, a noise mask is set for each of both noise pixels ([both (noise)]), noise ([single (noise, left eye)]), and noise ([single (noise), right eye])) [119].
Threshold value T2 is a value set by referring to the dispersion of pixel values or the like. The relationship between threshold value T2 and threshold value T1 satisfies the expression T1>T2 or T1<T2. Threshold value T2 is used to determine filter strength. The relationship may satisfy the equation T1=T2.
Next,
First, with the pixel correspondence information obtained from a left-eye image, a right-eye image, and
The pixel values of all the single noise pixels ([single (noise)]) are replaced as described above, obtaining left-eye and right-eye images from which noise has been removed.
In the embodiment, the corresponding pixel calculation module may be based on any method, provided that the method is generally called a stereo matching method.
In addition, the noise value at the noise determination module represents a value indicating the amount of shift from a surrounding pixel. A plurality of absolute difference values with neighborhood pixels excluding the surrounding four pixels used in the explanation of
If, the number of input images is three or more (multiple image), the correspondence relationship is found in the same manner. Of the three input images, a combination of arbitrary two images is input. All the combinations are processed in the same manner, enabling three or more images to be handled.
As described above, with the embodiment, the corresponding points of the same subject (two or more images taken from the same photographic subject at the same instant) are compared to obtain noise information originating in an imaging device, which enables noise (originating in the imaging device) to be distinguished.
In addition, noise can be removed with high accuracy by using corresponding values of the remaining one (the other) of the images as alternative pixels to noise.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2011-099750 | Apr 2011 | JP | national |
