Patent Grant
7773136
This application is based upon and claims the benefit of priority from both the prior Japanese Patent Application No. 2006-279293, filed Oct. 12, 2006 and the prior Japanese Patent Application No. 2006-231384, filed Aug. 28, 2006, the entire contents of which are incorporated herein by references.
1. Field of the Invention
The present invention relates to an image pickup apparatus for taking color images and an image pickup method.
2. Description of the Related Art
Digital still cameras and digital movie cameras capable of taking color images are now in wide use. Image pickup devices mounted on such image pickup apparatuses have photographic sensitivity to infrared light, too. Thus, when infrared light is contained in incoming light, not only visible light component but also error components due to the infrared light are contained in an output signal thereof and therefore the color reproducibility deteriorates.
On the other hand, a method is available where an infrared light cut filter is used for an optical system in order to remove the infrared light contained in the incident light. Nevertheless, the infrared light cut filter is rather costly and it also has to be placed within the optical system, thus imposing a constraint on a design of the optical system. Also, it is difficult for the infrared light cut filter to purely remove the infrared light alone and there are cases where the input of the visible light is also attenuated. In such a case, the sensitivity of the image pickup devices deteriorates and therefore the color reproducibility drops.
An image pickup apparatus according to one embodiment of the present invention comprises: a color filter of multiple colors which transmits an infrared light component; an image pickup device which receives an incident light by way of the color filter; and a control unit which switches, according to a predetermined condition, between an addition processing where a correcting infrared light component for correcting an infrared component contained in an output signal is added to the output signal of each color component in the image pickup device and a subtraction processing where the infrared component contained in the output signal is removed from the output signal of each color component.
Embodiments will now be described by way of examples only, with reference to the accompanying drawings which are meant to be exemplary, not limiting and wherein like elements are numbered alike in several Figures in which:
The invention will now be described by reference to the preferred embodiments. This does not intend to limit the scope of the present invention, but to exemplify the invention.
Firstly, a description is given of representative modes of carrying out the present invention before explaining preferred embodiments thereof. An image pickup apparatus according to one embodiment of the present invention comprises: a color filter of multiple colors which transmits an infrared light component; an image pickup device which receives an incident light by way of the color filter; and a control unit which switches, according to a predetermined condition, between an addition processing where a correcting infrared light component for correcting an infrared component contained in an output signal is added to the output signal of each color component in the image pickup device and a subtraction processing where the infrared component contained in the output signal is removed from the output signal of each color component. In the addition processing (hereinafter referred to as “adding processing” also), an infrared light component for correction may be added in such a manner that infrared light components contained in the output signals are equal to each other. “Infrared light components are equal” includes also a case where the infrared light components are practically equal. If the difference in integration values of spectral sensitivity in the range of wavelength, which can be regarded as the infrared light component, contained in the output signal of each color component is 1/100 or less of each integration value, the infrared light components may be regarded as being equal. For example, when the difference in integration values for the component whose wavelength in the spectral sensitivity characteristics is 700 nm or more in the output signal of each color component is 1/100 or less, the infrared light components may be regarded as being equal.
According to this embodiment, in order to correct an infrared component contained in the output signal of each color component, an infrared light component for correction is added or subtracted according to a predetermined condition. Thereby, the color reproducibility can be enhanced by adjusting the sensitivity.
The control unit may detect the brightness of a captured pixel from the output signal of each color component and switch between the addition processing and the subtraction processing according to the brightness thereof. When the “brightness of a pixel” exceeds a predetermined threshold, the subtraction processing may be selected; and when it does not exceed the predetermined threshold, the addition processing may be selected. According to this structure and operation, the increase in noise components due to the correction can be suppressed while the degradation in sensitivity is being suppressed.
The control unit may switch between the addition processing and the subtraction processing according to a relation between a visible light component contained in the output signal of each color component and the infrared light component received by the image pickup device. When the “visible light component” is larger than the “infrared light component”, the subtraction processing may be selected; and when smaller, the addition processing may be selected. The “infrared light component” may be received through an infrared light transmitting filter. According to this structure and operation, the increase in noise components due to the correction can be suppressed while the degradation in sensitivity is being suppressed.
When a luminance value obtained from the output value of each color component exceeds a predetermined threshold value and is greater than the infrared component contained in the output signal received by image pickup device, the control unit may select the subtraction processing. According to this structure and operation, the increase in noise components due to the correction can be suppressed and further accuracy is ensured while the degradation in sensitivity is being suppressed.
A white balance may be achieved by increasing or decreasing the output signal of each color component, and a first infrared light component for correction may be added to the output signal of each color component in such a manner that infrared light components contained in the output signals are equal to each other. A white balance may be properly adjusted by increasing or decreasing the output signal of each color component, and a second infrared light component for correction may be subtracted from the output signal of each color component in such a manner that infrared light components contained in the output signals are equal to each other. According to this structure and operation, a shift of the infrared light component due to the adjustment of white balance after the correction of the infrared light component can be avoided.
The infrared light component received by the image pickup device by way of the infrared transmitting filter may be multiplied by a coefficient to equalize the infrared light components contained in the output signal of each color component, and the infrared light component for correction may be obtained for each color component and may be added to the output signal of each color component. The infrared light component received by the image pickup device by way of the infrared transmitting filter may be multiplied by a coefficient to remove the infrared light components contained in the output signal of each color component, and the infrared light component for correction may be obtained for each color component and may be subtracted from the output signal of each color component.
Another embodiment of the present invention relates to an image pickup method. This method is such that, for an output signal of each color component from an image pickup device which receives an incident light by way of a color filter of multiple colors that transmits an infrared light component, the addition processing and the subtraction processing are switched therebetween according to a predetermined condition, wherein the addition processing adds a correcting infrared light component to correct an infrared light component contained in the output signal and the subtraction processing removes the infrared light component contained in the output signal from the output signal of each color component.
According to this embodiment, in order to correct an infrared component contained in the output signal of each color component, an infrared light component for correction is added or subtracted according to a predetermined condition. Thereby, the color reproducibility can be enhanced by adjusting the sensitivity.
An image pickup method according to one embodiment of the present invention is such that, for an output signal of each color component from an image pickup device which receives an incident light by way of a color filter of multiple colors that transmits an infrared light component, an infrared light component for correction is added so that infrared light components contained in the output signals are equal to each other. “Infrared light components are equal” includes also a case where the infrared light components are practically equal. If the difference in integration values of spectral sensitivity in the range of wavelength, which can be regarded as the infrared light component, contained in the output signal of each color component is 1/100 or less of each integration value, then the infrared light components may be regarded as being equal. For example, when the difference in integration values for the component whose wavelength in the spectral sensitivity characteristics is 700 nm or more in the output signal of each color component is less than or equal to 1/100, the infrared light components may be regarded as being equal.
According to this embodiment, the infrared light component for correction is added to the output signal of each color that contains an infrared light component in a manner such that the infrared light components contained in the outputs are each equal. Hence, the color reproduction can be achieved, in a highly sensitive manner, without using the infrared light cut filter.
A white balance may be achieved by increasing or decreasing the output signal of each color component, and an infrared light component for correction may be added to the output signal of each color component in such a manner that infrared light components contained in the output signals are each equal. According to this structure and operation, a shift of the infrared light component due to the adjustment of white balance after the correction of the infrared light component can be avoided. The infrared light component received by the image pickup device by way of the infrared transmitting filter may be multiplied by a coefficient to equalize the infrared light components contained in the output signal of each color component, and the infrared light component for correction may be obtained for each color component and may be added to the output signal of each color component. According to this structure and operation, the sensitivity to the incident light in an infrared light region of the image pickup devices can be practically equalized to one another (See
Another embodiment of the present invention relates to an image pickup apparatus. This apparatus comprises: a color filter of multiple colors which transmits an infrared light component; an image pickup device which receives an incident light by way of the color filter; and a control unit which adds an infrared light component for correction, to an output signal of each color component of the image pickup device so that infrared light components contained in the output signals are mutually equal.
According to this embodiment, the correcting infrared light component is added to an output signal of each color that contains an infrared light component in a manner such that the infrared light components contained in the output signals are each equal. Hence, the color can be reproduced in a highly sensitive manner without using the infrared light cut filter.
It is to be noted that any arbitrary combination of the above-described structural components and the expressions according to the present invention changed among a method, an apparatus, a system and so forth are all effective as and encompassed by the present embodiments.
If it is structured by complementary color filters, the incident light is separated into yellow (Ye), cyan (Cy) and magenta (Mg). Or, the incident light is separated into yellow (Ye), cyan (Cy) and green (Gr). Or, the incident light is separated into yellow (Ye), cyan (Cy), magenta (Mg) and green (Gr). Since the color filter 10 is not equipped with the infrared light cut filter, the color filter 10 transmits an infrared light component in addition to the visible light component, as described above.
The infrared light transmitting filter 20 transmits the infrared light component and supplies it to the image pickup device 30. The image pickup device 30 is comprised of a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. A plate of the image sensor may be provided per color so as to combine images of each color. Or, the structure may be such that an incident light coming from the color filter 10 arranged in the Bayer array is received and an interpolation operation is done using the outputs of surrounding pixels to produce a color image.
The image pickup device 30 has a region that receives the light transmitted through the infrared light transmitting filter 20, in addition to a region that receives a plurality of color components transmitted through the color filter 10. The image pickup devices 30 supplies an image signal of multiple colors generated by subjecting the received color components to a photoelectric conversion and a signal generated by subjecting the received infrared components to the photoelectric conversion (this signal will be hereinafter denoted by “IR signal”), to the control unit 40.
The control unit 40 includes an infrared light component adjustment unit 42, a white-balance adjustment unit 44, a switching unit 46, a condition determining unit 48 and a condition table 50. In terms of hardware, the structure of the control unit 40 can be realized by any DSP, memory and other LSIs. In terms of software, it can be realized by memory-loaded programs and the like, but drawn and described herein are function blocks that are realized in cooperation with those. Hence, it is understood by those skilled in the art that these function blocks can be realized in a variety of forms such as by hardware only, software only or the combination thereof. Note that
The infrared light component adjustment unit 42 adds infrared light components for correction (hereinafter referred to as “correcting infrared light components” also) to the image signals of multiple colors supplied from the image pickup device 30, respectively. Hereinafter, this processing will be referred to as “adding processing”. Alternatively, the infrared light component adjustment unit 42 subtracts the correcting infrared light components from the image signals of multiple colors supplied from the image pickup device 30, respectively. This processing will be referred to as “subtraction processing”. Whether the adding processing, the subtraction processing or no adjustment of the infrared light components is to be performed is specified from the switching unit 46. The infrared light component adjustment unit 42 generates an infrared light component by multiplying an IR signal by a predetermined coefficient. The predetermined coefficient differs between the case of adding processing and the case of subtraction processing. Also, at least one of a plurality of correcting infrared light components to be used in the adding processing and those to be used in the subtraction processing may be generated. When generating the correcting infrared light components, the infrared light component adjustment unit 42 generates the infrared light components in a manner such that the infrared light component contained in an image signal of each color is practically equal to each other.
The white-balance adjustment unit 44 carries out correction in a manner that even in a state of light source where the color temperature differs, the white color is accurately produced in white. Accordingly, the white-balance adjustment unit 44 adjusts the sensitivity of the image pickup device 30 to the incident light of each color. More specifically, the size of a red R image signal, a green G image signal and a blue B image signal is so increased or decreased that the ratio of each image signal used to generate the white color by combining them is adjusted. Note that the white-balance adjustment unit 44 is not the essential component in the present embodiment, and a mode of carrying out the present invention where no white balance is performed at all is also effective.
According to a decision result by the condition determining unit 48, the switching unit 46 specifies to the infrared light component adjustment unit 42 whether the adding processing, the subtraction processing or no adjustment of the infrared light components is to be performed. Also, according the above decision result, which correcting infrared light component is to be used is specified if one is to be selected from a plurality of correction infrared light components.
The condition determining unit 48 generates a signal for use in condition determination from at least one of the image signal of multiple colors and the IR signal supplied from the image pickup device 30. For example, after the red R image signal, the green G image signal and the blue B image signal have been received from the image pickup device 30, an RGB space is converted to a YUV space. Here, Y indicates an luminance signal, U a differential signal in blue and V a differential signal in red. The condition determining unit 48 generates the luminance signal Y as a signal for use in condition determination. As a result, the brightness of captured pixels and images can be detected with ease.
Since the color filter 10 does not remove the infrared light component, not only the visible light components but also infrared light components are contained in the red R image signal, the green G image signal and the blue B image signal received from the image pickup device 30. The condition determining unit 48 can use a signal indicating a relation between the visible light components that contain an infrared light entered through the color filter 10 and the infrared light components obtained from the IR signal, as a signal for use in condition determination. For example, such a signal indicating the relation may be a signal indicating a magnitude relation between the above-described visible light components and the above-described infrared light components, a signal indicating the difference therebetween, or a signal indicating a ratio thereof.
Based on the signal for use in condition determination, the condition determining unit 48 selects a type of infrared light component adjustment processing by referring to the condition table 50, and specifies the selected type to the switching unit 46. The condition table 50 describes the above signals for use in determination and the types of infrared light component adjustment processing in a manner that associates the signals with the types of processing. Specific examples of the condition table 50 will be described later.
The display unit 60 displays an image generated based on the image signals which has been corrected by the control unit 40. That is, the display unit 60 displays the image generated based on the signals where the correcting infrared light components are added respectively to the image signals of multiple colors supplied from the image pickup device 30. Alternatively, the display unit 60 displays the image generated based on the signals where the correcting infrared light components are subtracted respectively from the image signals of multiple colors supplied from the image pickup device 30.
Prior to describing an operation of the image pickup apparatus 100 according to the present embodiment, an example of adding processing is first explained. In this example, a green G transmitting filter, a blue B transmitting filter and an infrared IR transmitting filter are used.
R2=α*R+β*IR (Equation 1)
G2=γ*G+δ*IR (Equation 2)
B2=ε*B+ζ*IR (Equation 3)
Each coefficient (α to ζ) in the above Equations 1 to 3 are determined in a manner such that the values of infrared light components contained in the output signals of the respective colors are practically equal to one another while the white balance is achieved. The coefficients α, γ and ε are coefficients by which to properly adjust the white balance, whereas the coefficients β, δ and ζ are coefficients by which to practically equalize the values of infrared light components contained in the respective output signals. It is to be noted here that coefficients β, δ and ζ are positive values or zero. These coefficients can be derived through simulation or experiments carried out by a designer.
In the examples of
R2=1.3*R (Equation 4)
G2=G+0.65*IR (Equation 5)
B2=1.55*B (Equation 6)
In
A description is next given of subtraction processing. Similar to the adding processing, the subtraction processing uses the Equations 1 to 3. In the case of subtraction processing, the infrared light component is subtracted from at least one of the output signals of the respective colors in order to remove the infrared light components contained in the output values of the respective colors. Accordingly, the coefficients β, δ and ζ are negative values or zero. Of course, the white balance may be adjusted together with the subtraction processing. In order to practically equalize the infrared light components contained in the output signals of the respective colors, the infrared light component may be subtracted from at least one of the output signals of the respective colors. In this case, infrared light components remain in each output signal.
In either the adding processing or the subtraction processing, the combination of the coefficients β, ι and ζ is not necessarily one fixed way. Basically, the combination where the added value or the subtracted value is minimum is used. A combination where the added value is larger may be used when the luminance value is extremely small and therefore it is desired that the sensitivity be set high.
Now, a first exemplary embodiment of the present embodiment will be described. The first exemplary embodiment illustrates an example where the processing is switched between the adding processing and the subtraction processing according to the value of a luminance signal Y.
In the condition table 50a shown in
After having decided on either one of the adding processing and the subtraction processing, the condition determining unit 48 may determine the degree of values added or subtracted, according to a difference between the value of the luminance value Y and the predetermined threshold value. For example, in a case where a plurality of kinds of combinations of coefficients β, δ and ζ in the above Equations 1 to 3 are prepared and set in the adding processing, the smaller the value of the luminance signal Y is than the predetermined threshold value, a combination of the larger coefficients β, δ and ζ will be selected. Also, in a case where a plurality of kinds of combinations of coefficients β, δ and ζ are prepared and set in the subtraction processing in a similar manner, the larger the value of the luminance signal Y is than the predetermined threshold value, a combination of the larger coefficients β, δ and ζ will be selected.
If a difference between the value of the luminance signal Y and the predetermined threshold value is smaller than a preset value, the condition determining unit 48 may select a state where neither of the adding processing and the subtraction processing is performed.
According to the above-described first exemplary embodiment, the color reproducibility can be enhanced by adjusting the sensitivity. That is, while the sensitivity is enhanced by adding the infrared light components to the pixels captured at a dark place, the color balance can be adjusted. Also, the infrared light components are subtracted from the pixels captured at a bright place, so that the color balance can be adjusted by suppressing the increase in noise components as compared with the case of adding the infrared light components.
The degree of infrared light components to be added or subtracted is varied according to the decision result, so that the sensitivity can be set within a desired range. If the white balance is also adjusted at the time when the infrared light components are corrected, the following situation can be avoided. That is, such a situation is that the adjustment of the infrared light components is invalidated in a case when the white balance is adjusted in a post-image processing.
A description is next given of a second exemplary embodiment of the present embodiment. This is an example where whether the adding processing or the subtraction processing is to performed is switched therebetween according to the magnitude relation between the visible light components and the infrared light components entered via the color filters 10.
In the condition table 50b shown in
Note that after having determined either one of the adding processing and the subtraction processing, the condition determining unit 48 may determine the degree of values added or subtracted, according to a difference between the value of the above luminance value Y and the above infrared light component, in a similar manner to the first exemplary embodiment. Similarly to the first exemplary embodiment, if a difference between the value of the visible light component contained in the luminance signal Y and a predetermined threshold value is smaller than a preset value, the condition determining unit 48 may select a state where neither of the adding processing and the subtraction processing is performed.
According to the second exemplary embodiment as described above, the color reproducibility can be enhanced by adjusting the sensitivity. That is, when the visible light component is smaller than the infrared light component, adding the infrared light component can adjust the color balance while the sensitivity is enhanced. Also, when the visible light component is larger than the infrared light component, the infrared light component is subtracted, so that the color balance can be adjusted.
A description is next given of a third exemplary of the present embodiment. The third exemplary embodiment is an example where the condition described in the first exemplary embodiment is combined together with that in the second exemplary embodiment and the determination is made using the combined conditions.
If the visible light component contained in this luminance signal Y is larger than the infrared light component contained therein (Y of S14), the subtraction processing will be selected (S18). If the visible light component contained in this luminance signal Y is less than or equal to the infrared light component contained therein (N of S14), the adding processing will be selected (S16).
After having decided on either one of the adding processing and the subtraction processing, the condition determining unit 48 may determine the degree of values added or subtracted, according to a difference between the absolute value of the visible light component contained in the luminance value Y and the predetermined threshold value, similarly to the first exemplary embodiment. Similarly to the first exemplary embodiment, if the difference between the absolute value of the visible light component contained in the luminance signal Y and the predetermined threshold value is smaller than a preset value, the condition determining unit 48 may select a state where neither of the adding processing and the subtraction processing is performed.
According to the third exemplary embodiment as described above, images can be picked up with high sensitivity no matter whether the image is picked up at bright or dark place and no matter whether the infrared light component is large or small. Also, the color reproducibility can be enhanced without using the infrared light cut filter.
The present invention has been described based on some embodiments. These embodiments are merely exemplary, and it is understood by those skilled in the art that various modifications to the combination of each component and each process thereof are possible and that such modifications are also within the scope of the present invention.
In second exemplary embodiment, if the difference between or the ratio of the visible light component entered by way of the color filter 10 and the infrared light component obtained from the IR signal is used as a signal for use in condition determination, this difference or ratio needs to be compared against a predetermined threshold value. Such a threshold value can also be set to a value derived through simulation or experiments carried out by a designer.
The image pickup apparatus 100 may perform the adding processing only, without performing the subtraction processing at all. In such a case, there is no need to provide the switching unit 46, the condition determining unit 48 and the condition table 50 in the control unit 40. In this modification, too, the following advantageous effects are achieved. The color reproduction can be achieved, in a highly sensitive manner, without using an infrared light cut filter. That is, an infrared light component is added to the image signal of each color so as to realize the high sensibility. Also, the processing is so performed as to equalize the infrared light component contained in the image signal of each color. Hence, the image signal of each color is added with a certain fixed increment of infrared light component. As a result, the balance of the visible light components can be maintained and therefore the color reproducibility can be enhanced. If the white balance is also adjusted at the time when the infrared light components are corrected, the following situation can be avoided. That is, such a situation is that the adjustment of the infrared light components is invalidated in a case when the white balance is adjusted in a post-image processing.
In the above embodiment, an example has been shown where the red R transmitting filter, the green G transmitting filter, the blue B transmitting filter and the infrared IR transmitting filter are used. In this regard, a yellow Ye transmitting filter, a cyan Cy transmitting filter and a magenta Mg transmitting filter and an infrared light IR transmitting filter may be used.
While the preferred embodiments of the present invention have been described using specific terms, such description is for illustrative purposes only, and it is to be understood that changes and variations may be further made without departing from the spirit or scope of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2006-231384 | Aug 2006 | JP | national |
| 2006-279293 | Oct 2006 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6211521 | Bawolek et al. | Apr 2001 | B1 |
| 6657663 | Morris | Dec 2003 | B2 |
| 6759646 | Acharya et al. | Jul 2004 | B1 |
| 7170046 | Higashitsutsumi | Jan 2007 | B2 |
| 7460160 | Hershey et al. | Dec 2008 | B2 |
| 20060188155 | Matsuyama | Aug 2006 | A1 |
| 20070070224 | Sasaki | Mar 2007 | A1 |
| 20070146512 | Suzuki et al. | Jun 2007 | A1 |
| 20070187794 | Fukuyoshi et al. | Aug 2007 | A1 |
| 20070201738 | Toda et al. | Aug 2007 | A1 |
| 20080068475 | Choe et al. | Mar 2008 | A1 |
| 20080088826 | Ohyama et al. | Apr 2008 | A1 |
| Number | Date | Country |
|---|---|---|
| 06-105319 | Apr 1994 | JP |
| 11341502 | Dec 1999 | JP |
| 2002-142228 | May 2002 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20080049115 A1 | Feb 2008 | US |
