Color image sensing device

BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to an arrangement of color filters for an image sensor in which pixels have the offset sampling structure.
2. Related Background Art
FIGS. 1, 2 and 3 show conventionally known examples of an arrangement of color filters for a color solid-state image sensing device. In FIG. 1, green light transmissive filters (hereinafter referred to as "Gr filters") are vertically arrayed in the stripe form. Red light transmissive filters (hereinafter referred to as "Rd filters") and blue light transmissive filters (hereinafter referred to as "Bl filters") are arrayed two rows apart vertically and one column apart horizontally between the Gr filters.
In FIG. 2, magenta light transmissive filters (hereinafter referred to as "Mg filters"), green light transmissive filters, cyan light transmissive filters (hereinafter referred to as "Cy filters"), and yellow light transmissive filters (hereinafter referred to as "Ye filters") are arranged in the sequence as shown in units of eight color filters consisted of horizontal two pixels and vertical four pixels.
FIG. 3 is concerned with a solid-state image sensor having the offset sampling structure disclosed in Japanese Patent Application No. 1-24433, for example. In this offset sampling structure, Rd, Gr and Bl filters are arranged in units of three color filters consisted of horizontal three pixels and vertical one pixel with an offset amount of 1.5 pixels in the horizontal direction between two rows.
FIGS. 4, 5 and 6 are characteristic diagrams of the first quadrant as obtained by expressing color light carriers in the color filter arrangements of FIGS. 1, 2 and 3 using the two-dimensional frequency plane (f.sub.H, f.sub.V), respectively. Assuming that the horizontal pixel pitch is P.sub.H and the vertical pixel pitch is P.sub.V, each diagram represents an area of 0.ltoreq.f.sub.H .ltoreq.1/P.sub.H and 0.ltoreq.f.sub.V .ltoreq.1/2P.sub.V. In any diagram, the arrow indicates a carrier of each color, the arrow length indicates a magnitude of the carrier, and the arrow direction indicates a phase relationship.
In FIG. 4, color light carriers occur at (1/2P.sub.H, 0), (1/P.sub.H, 0), (0, 1/4P.sub.V), (1/2P.sub.H, 1/4P.sub.V) and (1/P.sub.H, 1/4P.sub.V) other than (0, 0). Among then, (0, 0) and (1/P.sub.H, 0) represent the carriers which occur for achromatic light and cause turn-back distortions. The remaining carriers are perfectly canceled out and disappeared for achromatic light, but not disappeared for chromatic light and cause turn-back distortions.
Likewise, in FIG. 5, color light carriers occur at (1/2P.sub.H, 0), (1/P.sub.H, 0), (1/2P.sub.H, 1/4P.sub.V), (0, 1/2P.sub.V), (1/2P.sub.H, 1/2P.sub.V) and (1/P.sub.H, 1/2P.sub.V) other than (0, 0). Among then, (0, 0) and (1/P.sub.H, 0) represent the carriers which occur for achromatic light and others represent the carriers which are disappeared for chromatic light.
In FIG. 6, assuming that the horizontal offset amount of the solid-state image sensor is P.sub.H /2, color light carriers occur at (2/3P.sub.H, 0), (1/3P.sub.H, 1/2P.sub.V) and (1/P.sub.H, 1/2P.sub.V) Among them, (0, 0) and (1/P.sub.H, 1/2P.sub.V) represent the carriers which occur for achromatic light and others represent the carriers which are disappeared for chromatic light.
It is generally known that the above offset sampling structure has a feature as follows. With the rectangular sampling structure as shown in FIGS. 1 and 2, since the carrier occurs for achromatic light at the frequency corresponding to the horizontal position of (1/P.sub.H, 0), f.sub.H =1/2P.sub.H becomes the Nyquist's frequency and the frequency component thereabove cannot be obtained. Thus, the horizontal resolution obtainable with this structure is maximally f.sub.H =1/2P.sub.H. On the other hand, with the offset sampling structure as shown in FIG. 3, the carrier does not occur for achromatic light at the frequency corresponding to the horizontal position of (1/P.sub.H, 0) and f.sub.H =1/P.sub.H can become the Nyquist's frequency. Therefore, although the color filter arrangement of FIG. 3 has the same sampling pitch as the rectangular sampling structure of FIGS. 1 and 2, it can realize the horizontal resolution twice the above cases, i.e., f.sub.H =1/P.sub.H.
However, because objects to be usually photographed are not always achromatic but colored in general cases, the color light carriers at all the positions shown in FIGS. 4, 5 and 6 generate turn-back distortions, so that some scenes may be awkward or hard to see. This necessitates use of an optical low-pass filter or the like to cut the detrimental color light carriers, which leads to a reduction in the resolution.
In the color solid-state image sensing device with the offset sampling structure shown in FIGS. 3 and 6, for example, the color light carrier occurs at the position of (2/3P.sub.H, 0) and, therefore, an optical low-pass filter capable of cutting off the frequency component above f.sub.H =2/3P.sub.H in the horizontal direction is required. This means that while the horizontal resolution up to f.sub.H =1/P.sub.H could be intrinsically obtained for achromatic light, the practically obtainable horizontal resolution is only 2/3of that, i.e., f.sub.H =2/3P.sub.H.
SUMMARY OF THE INVENTION
The present invention has been made to solve the problem as mentioned above, and its object is to provide a color image sensing device which has good resolution and causes less moires, and which can produce the horizontal resolution up to f.sub.H =1/P.sub.H even with the sampling structure having the horizontal pixel pitch of P.sub.H and the vertical pixel pitch of P.sub.V.
To achieve the above object, a color image sensing device according to one embodiment of the present invention is constituted as follows.
Specifically, the color image sensing device comprises an image sensor having pixels of the offset sampling structure in which the horizontal pitch is P.sub.H, the vertical pitch is P.sub.V and the horizontal offset amount is P.sub.H /2, and a color filter array comprising three or four types of color filters provided in one-to-one relation to the pixels of the image sensor, the color filter array having the offset sampling structure in which those types of color filters are each arranged with the horizontal pitch of 2P.sub.H, the vertical pitch of 2P.sub.V and the horizontal offset amount of P.sub.H.
With the color image sensing device thus constituted, the horizontal resolution up to f.sub.H =1/P.sub.H is obtained for achromatic light and the horizontal and vertical resolutions are less reduced for chromatic light.
Another embodiment of the present invention is to provide an image sensing device in which the image sensor using a color filter array of the offset sampling structure according to the above embodiment is combined with an optical low-pass filter which is effective in suppressing occurrence of color moires.
To achieve the above object, a color image sensing device according to still another embodiment of the present invention is given by (1) or (2) below.
(1) A color image sensing device featured in comprising the following components a, b and c:
a. an image sensor having pixels of the offset sampling structure in which the horizontal pitch is P.sub.H, the vertical pitch is P.sub.V and the horizontal offset amount is P.sub.H /2;
b. a color filter array comprising four types of color filters provided in one-to-one relation to the pixels of the above image sensor, the color filter array having the offset sampling structure in which those types of color filters are each arranged with the horizontal pitch of 2P.sub.H, the vertical pitch of 2P.sub.V and the horizontal offset amount of P.sub.H ; and
c. conversion means for converting outputs of the pixels associated with the four types of color filters into RGB signals through matrix operation of 3 row.times.4 column, the matrix being set such that, for each row, the sum of coefficients of first two columns is equal to the sum of coefficients of the remaining two columns.
(2) A color image sensing device according to (1) and including an optical low-pass filter which is disposed in an image sensing optical system and divides an incident light beam into twos spaced through a distance D in a direction turned by an angle .theta. counterclockwise relative to the scanning direction of the image sensor or in an direction turned by an angle .theta. clockwise relative to the reversed scanning direction of the image sensor, the optical low-pass filter meeting the conditions below; ##EQU1## where 0.ltoreq..theta..ltoreq..pi./2.
With the color image sensing devices of above (1) and (2), the resolution up to f.sub.H =1/P.sub.H is obtained and moires are less generated. To achieve the above object, a color image sensing device according to still another embodiment of the present invention is constituted as follows. Specifically, the color image sensing device comprises the following components a, b and c:
a. an image sensor having the offset sampling structure in which the horizontal pixel pitch is P.sub.H, the vertical pixel pitch is P.sub.V and the horizontal pixel offset amount is P.sub.H /2;
b. a color filter array comprising three or more types of color filters, provided for the above image sensor, and having the offset sampling structure in which those types of color filters are each arranged with the horizontal pitch of 2P.sub.H, the vertical pitch of 2P.sub.V and the horizontal offset amount of P.sub.H ; and
c. an optical low-pass filter disposed in an image sensing optical system and comprising a first optical member for dividing an incident light beam into twos spaced through a distance P.sub.1 in a direction turned by +45.degree. or -45.degree. counterclockwise relative to the horizontal scanning direction of the image sensor and a second optical member for dividing an incident light beam into twos spaced through a distance P.sub.2 in a direction turned by 90.degree. relative to the beam dividing direction of the first optical member, the optical low-pass filter meeting the conditions below; ##EQU2##
With the color image sensing device thus constituted, color carriers generated by the color filter array are removed by the optical low-pass filter and the occurrence of color moires is suppressed.
To achieve the above object, a color image sensing device according to still another embodiment of the present invention is constituted as follows.
Specifically, the color image sensing device comprises the following components a, b and c:
a. an image sensor having the offset sampling structure in which the horizontal pixel pitch is P.sub.H, the vertical pixel pitch is P.sub.V and the horizontal pixel offset amount is P.sub.H /2;
b. a color filter array comprising three or more types of color filters, provided for the above image sensor, and having the offset sampling structure in which those types of color filters are each arranged with the horizontal pitch of 2P.sub.H, the vertical pitch of 2P.sub.V and the horizontal offset amount of P.sub.H ; and
c. an optical low-pass filter disposed in an image sensing optical system and comprising a first optical member for dividing an incident light beam into twos spaced through a distance P.sub.1 in a direction turned by +45.degree. or -45.degree. counterclockwise relative to the horizontal scanning direction of the image sensor, a second optical member for dividing an incident light beam into twos spaced through a distance P.sub.2 in a direction parallel to the horizontal scanning direction and a third optical member for dividing an incident light beam into twos spaced through a distance P.sub.3 in a direction turned by 90.degree. relative to the beam dividing direction of the first optical member, the optical low-pass filter meeting the conditions below; ##EQU3##
With the color image sensing device thus constituted, color carriers generated by the color filter array are removed by the optical low-pass filter and the occurrence of color moires is suppressed.
Other objects and features of the present invention will be apparent from the following detailed description and the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a diagram showing one example of arrangement of color filters;
FIG. 2 is a diagram showing another example of arrangement of color filters;
FIG. 3 is a diagram showing still another example of arrangement of color filters;
FIG. 4 is a characteristic diagram of color light carriers in the color filter arrangement of FIG. 1;
FIG. 5 is a characteristic diagram of color light carriers in the color filter arrangement of FIG. 2;
FIG. 6 is a characteristic diagram of color light carriers in the color filter arrangement of FIG. 3;
FIG. 7 is a diagram showing an arrangement of color filters used in a first embodiment of the present invention;
FIG. 8 is a characteristic diagram of color light carriers in the color filter arrangement of FIG. 7;
FIG. 9 is a block diagram of the first embodiment;
FIG. 10 is a diagram for explaining characteristics of the first embodiment;
FIG. 11 is a diagram showing an arrangement of color filters used in a second embodiment of the present invention;
FIG. 12 is a characteristic diagram of color light carriers in the color filter arrangement of FIG. 11;
FIG. 13 is a block diagram of the second embodiment;
FIG. 14 is a diagram showing layout of an optical low-pass filter for use in a third embodiment of the present invention;
FIG. 15 is a diagram for explaining the construction of the optical low-pass filter in the third embodiment;
FIG. 16 is a characteristic diagram of the optical low-pass filter in the third embodiment;
FIG. 17 is a diagram showing a modification of the optical low-pass filter in the third embodiment;
FIG. 18 is a characteristic diagram of the optical low-pass filter shown in FIG. 17;
FIG. 19 is a diagram showing two-dimensional frequency characteristics (in the second quadrant) of color light carriers in the color filter arrangement of FIG. 7;
FIG. 20 is a diagram for explaining characteristics of a fourth embodiment;
FIG. 21 is a diagram for explaining the construction of the optical low-pass filter used in a fifth embodiment of the present invention;
FIG. 22 is a characteristic diagram of the optical low-pass filter in the fifth embodiment;
FIG. 23 is a characteristic diagram of a modification of the optical low-pass filter in the fifth embodiment;
FIG. 24 is a diagram for explaining characteristics of a sixth embodiment of the present invention;
FIGS. 25 and 26 are diagrams for explaining the offset sampling structure;
FIG. 27 is a diagram showing two-dimensional frequency characteristics of an ideal optical low-pass filter;
FIG. 28 is a diagram showing two-dimensional frequency characteristics of an optical low-pass filter used in a seventh embodiment of the present invention;
FIG. 29 is a diagram showing the construction of primary parts of the seventh embodiment;
FIGS. 30A to 30C are diagrams for explaining operation of the optical low-pass filter in the seventh embodiment;
FIG. 31 is a diagram showing the construction of primary parts of an eighth embodiment of the present invention;
FIG. 32 is a diagram showing the construction of primary parts of a ninth embodiment of the present invention;
FIG. 33 is a diagram showing the construction of primary parts of a tenth embodiment of the present invention;
FIGS. 34A to 34C are diagrams for explaining operation of the optical low-pass filter in the seventh embodiment;
FIG. 35 is a diagram showing the construction of primary parts of an eleventh embodiment of the present invention;
FIG. 36 is a block diagram of a twelfth embodiment;
FIG. 37 is a block diagram of an adapted filter process for a luminance signal in the twelfth embodiment;
FIGS. 38 and 39 are diagrams showing the construction of a filter used in the twelfth embodiment;
FIGS. 40 and 41 are diagrams showing image patterns which are liable to generate false colors;
FIG. 42 is a block diagram of a thirteenth embodiment;
FIG. 43 is a block diagram showing a matrix selection unit in the thirteenth embodiment;
FIG. 44 is a table for explaining operation of the thirteenth embodiment; and
FIG. 45 is a table for explaining operation of the thirteenth embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Hereinafter, the present invention will be described in detail in connection with preferred embodiments.
FIG. 7 shows an arrangement of color filters in a color image sensing device according to a first embodiment of the present invention. As will be seen from FIG. 7, a solid-state image sensor has the offset sampling structure in which the horizontal pixel pitch is P.sub.H, the vertical pixel pitch is P.sub.V and the horizontal pixel offset amount is P.sub.H /2. The image sensor is associated with a color filter array of the type that Magenta, green, cyan and yellow light transmissive filters Mg, Gr, Cy, Ye are arranged at positions corresponding to pixels in one-to-one relation.
In the offset sampling structure, those color filters are each arranged with the horizontal pitch of 2P.sub.H, the vertical pitch of 2P.sub.V and the horizontal offset amount of P.sub.H.
FIG. 8 is a characteristic diagram representing color light carriers in the above color filter arrangement using the two-dimensional frequency plane (f.sub.H, f.sub.V) over an area of 0.ltoreq.f.sub.H .ltoreq.1P.sub.H and 0.ltoreq.f.sub.V 11/2P.sub.V in the first quadrant. In FIG. 8, the color light carriers occur at (1/P.sub.H, 0), (1/2P.sub.H, 1/4P.sub.V), (0, 1/2P.sub.V) and (1/P.sub.H, 1/2P.sub.V) other than (0, 0). Among them, (0, 0) and (1/P.sub.H, 0) represent the carriers which occur for achromatic light and the remaining carriers are disappeared for achromatic light.
As will be apparent from FIG. 8, no color light carriers occur until the position of (1/P.sub.H, 0) in the horizontal direction. Accordingly, the frequency component up to f.sub.H =1/P.sub.H can be obtained. In other words, the color image sensing device of this embodiment can produce the horizontal resolution as high as f.sub.H =1/P.sub.H which is 1.5 times that obtainable with the color image sensing device having the sampling structure of the prior art.
The color light carrier nearest to the origin is given by (1/2P.sub.H, 1/4P.sub.V), but it is sufficiently spaced from the origin and is also disappeared for achromatic light. Therefore, that carrier will not cause a significant turn-back distortion for both the horizontal and vertical frequency components.
Thus, with the color image sensing device of this embodiment, high resolution is obtained and moires are less caused.
Signal processing in the color image sensing device which has the color filter arrangement shown in FIG. 1 will be next described. FIG. 9 is a block diagram for the signal processing.
A CCD sensor 1 as a solid-state image sensor is associated with a color filter array 19 comprising four types of color filters shown in FIG. 7. A video signal read out of the sensor 1 pixel by pixel is subjected to gain adjustment by an AGC 2 and, thereafter, to A/D conversion by an A/D (analog-to-digital) converter 3 at the timing synchronized to the read-out clock. The A/D converter 3 preferably has linear characteristics for the purpose of color processing to be carried out later, and the A/D conversion is desirably performed using 8 or more bits from the stand-point of errors in quantization.
A resulting luminance signal is passed through an interpolation filter 4 for two-dimensionally interpolate the offset sampling structure and a high-pass filter (HPF) 5 for detection of the higher-frequency luminance component thereof. The detected component is added in an adder 6 with the lower-frequency luminance component Y.sub.L obtained by the later-described technique and, thereafter, a summation signal is subjected to D/A (digital-to-analog) conversion by a D/A converter 7 and then outputted therefrom.
On the other hand, the output of the A/D converter 3 is also applied to four interpolation filters 8, 9, 10 and 11 which respectively output color signals Mg, Cy, Ye and Gr at the synchronized timing.
Note that by controlling the sequence of horizontal lines applied to the interpolation filters 8, 9, 10 and 11 in accordance with the clock, it is possible to obtain both an output signal based on interlace scan and an output signal based on non-interlace scan.
The synchronized color signals from the interpolation filters 8 to 11 are inputted to an RGB conversion unit 12 for conversion into three signals R, G, B. This conversion is performed through matrix operation below. ##EQU4## Here, the matrix A is a matrix of 3 row and 4 column optimized to make spectroscopic characteristics Mg(.lambda.), Gr(.lambda.), Cy(.lambda.), Ye(.lambda.) of the sensor 1 for Mg, Gr, Cy, Ye approach ideal spectroscopic characteristics Mg(.lambda.), Gr(.lambda.), Cy(.lambda.) for RGB specified in the NTSC system. Let it be now assumed that:
A=(a.sub.ij) (2)
(i=1, 2, 3; j=1, 2, 3, 4)
If the gain is adjusted to be constant for a monochromatic object, the base band components of Mg, Gr, Cy, Ye can be expressed below by using some function .alpha.(f.sub.H, f.sub.V) on the frequency space:
Mg(f.sub.H, f.sub.V)=Gr(f.sub.H, f.sub.V)=Ye(f.sub.H, f.sub.V)=Cy(f.sub.H, f.sub.V)=.alpha.(f.sub.H, f.sub.V) (3)
It is here supposed that the base band components of respective colors are sufficiently limited in band by two-dimensional filters such as the interpolation filters 8 to 11. In this case, the carrier components of the color signals at the position (1/2P.sub.H, 1/4P.sub.V) in FIG. 8 are expressed below:
Mg(f.sub.H, f.sub.V)=Cy(f.sub.H, f.sub.V)=.alpha.(f.sub.H, f.sub.V),
Gr(f.sub.H, f.sub.V)=Ye(f.sub.H, f.sub.V)=.alpha.(f.sub.H, f.sub.V)(4)
Therefore, the carrier components of the RGB signals at that position are expressed below from Equations (1), (2), (3) and (4). ##EQU5## At this time, if for each row of the matrix A the sum of coefficients of the first and third columns is equal to the sum of coefficients of the second and fourth columns, i.e., if the relationship of:
a.sub.i1 +a.sub.i3 =a.sub.i2 +a.sub.i4
(i=1, 2, 3) (6)
holds, the carrier components of the RGB signals at the point (1/2P.sub.H, 1/4P.sub.V) are reduced and, as result, there occur no carrier components of the color signals at that point. Likewise, the carrier components of the color signals at the point (-1/2P.sub.H, -1/4P.sub.V) in the third quadrant, symmetrical to the point (1/2P.sub.H, 1/4P.sub.V) about the origin in FIG. 8, can also be disappeared by making the relationship of Equation (6) satisfied.
FIG. 10 shows the positions at which the carriers of the luminance signal and the color signals occur near the origin. Because of disappearance of the carriers at (+1/2P.sub.H, 1/4P.sub.V) (double signs being effective for the same order), turn-back distortions to the base bands become smaller than the case of not meeting the relationship of Equation (6).
As explained above, the RGB conversion unit shown in FIG. 9 converts the color signals Mg, Gr, Cy, Ye into the RGB signals while reducing turn-back distortions.
Then, in a white balance unit 13, the RGB signals are converted from R, G, B into .alpha.R, G, .beta.B based on color temperature information obtained by a white balance sensor 18, thereby making white balance.
After that, in a .gamma. conversion unit 14, the RGB signals are subjected to .gamma. conversion through table conversion. In a color difference matrix unit 113, the following conversion in conformity with the NTSC system is performed for producing the aforesaid lower-frequency luminance component Y.sub.L and color difference signals R-Y, B-Y; ##EQU6## The color difference signals R-Y, B-Y are subjected to D/A conversion by subsequent D/A converters 16, 17, respectively, and then outputted therefrom. The lower-frequency luminance component Y.sub.L is added with the higher-frequency luminance component detected by the high-pass filter 5 as mentioned above, following which the summation signal is subjected to D/A conversion by a D/A converter 7 and then outputted therefrom.
In this embodiment, the above signal processing system may be hard-wired according to the block diagram, or it may be implemented in the form of software using a DSP (digital signal processor) or the like.
A second embodiment of the present invention will be next described.
FIG. 11 shows an arrangement of color filters making up a color filter array used in a color solid-state image sensing device according to a second embodiment of the present invention. As with the first embodiment, a solid-state image sensor of this embodiment has the offset sampling structure in which the horizontal pixel pitch is P.sub.H, the vertical pixel pitch is P.sub.V and the horizontal pixel offset amount is P.sub.H /2. Color filters, i.e., red, green and blue light transmissive filters Rd, Gr, Bl are arranged at positions corresponding to pixels in one-to-one relation. In the color filter array, as shown in FIG. 10, the color filters Rd and Bl are each arranged with the offset sampling structure having the horizontal pitch of 2P.sub.H, the vertical pitch of 2P.sub.V and the horizontal offset amount of P.sub.H. On the other hand, while the color filter Gr is arranged in the form of oblique stripes, this arrangement can be regarded as that two arrays of the offset sampling structure having the horizontal pitch of 2P.sub.H, the vertical pitch of 2P.sub.V and the horizontal offset amount of P.sub.H are superposed with shifts of P.sub.H /2 in the horizontal direction and P.sub.V in the vertical direction relative each other. FIG. 12 is a characteristic diagram representing color light carriers in the above color filter arrangement using the two-dimensional frequency plane (f.sub.H, f.sub.V) over an area of 0<f.sub.H <1/P.sub.H and 0<f.sub.V <1/2P.sub.V in the first quadrant.
In FIG. 12, the color light carriers occur at (1/P.sub.H, 0), (1/2P.sub.H, 1/4P.sub.V), (0, 1/2P.sub.V) and (1/P.sub.H, 1/2P.sub.V) other than (0, 0). Among them, (0, 0) and (1/P.sub.H, 0) represent the carriers which occur for achromatic light and the remaining carriers are disappeared for achromatic light. This is similar to the first embodiment.
Accordingly, also in this embodiment, the horizontal resolution as high as f.sub.H =1/P.sub.H can be obtained. In addition, no significant turn-back distortion occurs for both the horizontal and vertical frequency components.
Signal processing in the color solid-state image sensing device which has the color filter arrangement shown in FIG. 11 will be next described by referring to FIG. 13.
A CCD sensor 201 is associated with a color filter array 201a comprising three types of color filters shown in FIG. 11. A video signal read out of the sensor 201 pixel by pixel is subjected to gain adjustment by an AGC 202 and, thereafter, gains of RGB signals are adjusted in a white balance unit 211 based on color temperature information obtained by a white balance sensor 220, thereby making white balance. After that, the RGB signals are subjected to .gamma. conversion in a .gamma. conversion unit 212, and then to A/D conversion by an A/D converter 203 at the timing synchronized to the read-out clock.
A resulting luminance signal is passed through an interpolation filter 225 for two-dimensionally interpolating the offset sampling structure and then a high-pass filter (HPF) 216 for detection of the higher-frequency luminance component thereof. The detected component is added in an adder 217 with the lower-frequency luminance component Y.sub.L obtained by the later-described technique and, thereafter, a summation signal is subjected to D/A conversion by a D/A converter 218 and then outputted therefrom.
On the other hand, the output of the A/D converter 203 is also applied to three interpolation filters 206, 207 and 208 which respectively output color signals R, G and B at the synchronized timing.
Note that by controlling the sequence of horizontal lines applied to the interpolation filters 206, 207, 208 and 225 in accordance with the clock, it is possible to obtain both an output signal based on interlace scan and an output signal based on non-interlace scan.
The synchronized color signals from the interpolation filters 206, 207 and 208 are inputted to a color difference matrix unit 213 for conversion in conformity with the NTSC according to the equation (1), to thereby produce the lower-frequency luminance component Y.sub.L and color difference signals R-Y, B-Y. The color difference signals R-Y, B-Y are subjected to D/A conversion by subsequent D/A converters 214, 215, respectively, and then outputted therefrom. The lower-frequency luminance component Y.sub.L is added with the higher-frequency luminance component detected by the high-pass filter 216 as mentioned above, following which the summation signal is subjected to D/A conversion by a D/A converter 218 and then outputted therefrom.
In this embodiment, too, the above signal processing system may be hard-wired according to the block diagram, or it may be implemented in the form of software using a DSP or the like.
While the above embodiments have been explained in connection with still picture recording, the present invention is not limited thereto and can also be practiced in motion picture recording such as represented by video cameras.
With this embodiment, as explained above, in spite of using the sampling structure in which the horizontal pixel pitch is P.sub.H and the vertical pixel pitch is P.sub.V, no detrimental carriers occur over a wide range along the horizontal frequency axis, and the horizontal resolution as high as the frequency of f.sub.H =1/P.sub.H. It is thus possible to provide the color image sensing device which has high resolution and causes fewer moires.
Meanwhile, as stated before, in the color image sensing device of the first embodiment there occur carrier components responsible for turn-back distortions at such positions as shown in FIG. 10. In order to minimize the turn-back distortions in that type color image sensing device, an optical low-pass filter 50 is required to be located forwardly of the sensor 1, as shown in FIG. 14, to thereby trap positions in the vicinity of the carriers (.-+.1/2P.sub.H, .+-.1/4P.sub.V) (double signs being effective for the same order), which are closest to the origin, for suppressing the turn-back distortions.
A color image sensing device comprising the image sensing optical system of the first embodiment and such as additional optical low-pass filter will be described below as a third embodiment of the present invention. The optical low-pass filter 50 functions to, as shown in FIG. 15, divide an incident light beam into twos spaced through a distance D in a direction turned by an angle .theta. clockwise relative to the reversed scanning direction, while meeting the conditions below: ##EQU7## where
0.ltoreq..theta..ltoreq..pi./2 (8)
If the value of D exceeds a lower limit of Inequality (7), the turn-back distortions would be increased and, if it exceeds an upper limit of Inequality (7), the resolution would be decreased. Thus, any case would lead to the unsatisfactory result.
In this embodiment, .theta. and D are set as follows: ##EQU8## By so setting, the optical low-pass filter 50 traps such positions as indicated by dot lines in FIG. 16. Therefore, the carriers of the color signals are located at not only (.-+.1/2P.sub.H, .+-.1/4P.sub.V) (double signs being effective for the same order), but also (.+-.1/P.sub.H, 0) and (0, .+-.1/2P.sub.V), making it possible to minimize the turn-back distortions caused by the color signals and thus provide the satisfactory image quality.
The exact same characteristics as the above can also be obtained in the case of setting the beam dividing direction of the optical low-pass filter 50 in a direction symmetrical to the direction shown in FIG. 15 about a point, i.e., in a direction turned by an angle .theta. clockwise relative to the scanning direction. The optical low-pass filter 50 can be made of a double refracting plate using a uniaxial crystal such as quartz. Other than this example, the filter 50 may be in any form, including a prism, so long as it has a property to divide the incident light beam into twos.
Furthermore, as shown in FIG. 17, there can also be employed, as a modification of the third embodiment, an optical low-pass filter 80 in combination of an optical low-pass filter 81 which is made of a double refracting plate and divides an incident light beam into twos spaced through a distance D below in a direction turned by an angle 45.degree. clockwise relative to the reversed scanning direction; ##EQU9## and an optical low-pass filter 82 which is made of a double refracting plate for dividing an incident light beam into twos spaced through a distance P.sub.H /2 parallel to the scanning direction. In this case, spatial frequency characteristics of the optical low-pass filter 80 trap such positions as indicated by dot lines in FIG. 18. Therefore, not only the carriers of the color signals located at (.-+.1/2P.sub.H, .+-.1/4P.sub.V) (double signs being effective for the same order) are trapped, but also the carriers of the color signals located at (.+-.1/P.sub.H, 0) and (0, .+-.1/2P.sub.V) are trapped in their vicinity, making it possible to achieve a sufficient degree of suppression. In addition, since the carriers of the luminance signal located at (.+-.1/P.sub.H, .+-.1/2P.sub.V) (double signs being effective regardless of the order) are further trapped, there can be obtained images in which the turn-back distortions are satisfactorily suppressed.
A fourth embodiment of the present invention will be next described. This embodiment is different from the first embodiment only in the matrix for use with the RGB conversion unit.
FIG. 19 is a characteristic diagram representing color light carriers in the color solid-state image sensor of FIG. 7 using the two-dimensional frequency plane (f.sub.H, f.sub.V) in the second quadrant. In FIG. 19, the color light carriers occur at (-1/P.sub.H, 0), (-1/2P.sub.H, 1/4P.sub.V), (0, 1/2P.sub.V) and (-1/P.sub.H, 1/2P.sub.V) other than (0, 0). Among them, (0, 0) and (-1/P.sub.H, 1/2P.sub.V) represent the carriers which occur for achromatic light and the remaining carriers which occur for chromatic light. The above carrier positions are symmetrical to those shown in FIG. 8 about the f.sub.V axis except for carrier signs of the respective colors only at (-1/2P.sub.H, 1/4P.sub.V).
When converting the color signals Mg, Gr, Cy, Ye into the RGB signals in the RGB conversion unit 110 shown in FIG. 9, the carrier components of the color signals at the position (-1/2P.sub.H, 1/4P.sub.V) in FIG. 19 are expressed below using Equation (3):
Mg(f.sub.H, f.sub.V)=Ye(f.sub.H, f.sub.V)=.alpha.(f.sub.H, f.sub.V), Gr(f.sub.H, f.sub.V)=Cy(f.sub.H, f.sub.V)=.alpha.(f.sub.H, f.sub.V)(10)
Therefore, the carrier components of the RGB signals at that position are expressed below from Equations (1), (2), (3) and (10): ##EQU10## At this time, if for each row of the matrix A the sum of coefficients of the first and fourth columns is equal to the sum of coefficients of the second and third columns, i.e., if the relationship of:
a.sub.i1 +a.sub.i4 =a.sub.i2 +a.sub.i3
(i=1, 2, 3) (12)
holds, the carrier components of the RGB signals at the point (-1/2P.sub.H, 1/4P.sub.V) are disappeared and, as a result, there occur no carrier components of the color signals at that point. Likewise, the carrier components of the color signals at the point (1/2P.sub.H, -1/4P.sub.V) in the fourth quadrant, symmetrical to the point (-1/2P.sub.H, 1/4P.sub.V) about the origin in FIG. 19, can also disappear by satisfying the relationship of Equation (12).
FIG. 20 shows the positions at which the carriers of the luminance signal and the color signals occur near the origin.
In order to minimize the turn-back distortions in that type of color image sensing device, like the first embodiment, the optical low-pass filter 50 is required to be located forwardly of the sensor 1, as shown in FIG. 14, to thereby trap positions in the vicinity of the carriers (.+-.1/2P.sub.H, .+-.1/4P.sub.V) (double sings being effective for the same order), which are closest to the origin, for suppressing the turn-back distortions.
A color image sensing device comprising the image sensing optical system of the fourth embodiment and such as additional optical low-pass filter will be described below as a fifth embodiment of the present invention. The optical low-pass filter 50 functions to, as shown in FIG. 21, divide an incident light beam into twos spaced through a distance D in a direction turned by an angle .theta. counter-clockwise relative to the scanning direction, while meeting the conditions below: ##EQU11## where
0.ltoreq..theta..ltoreq..pi./2 (8)
If the value of D exceeds a lower limit of Inequality (7), the turn-back distortions would be increased and, if it exceeds an upper limit of Inequality (7), the resolution would be decreased. Thus, exceeding the limits would lead to the unsatisfactory result in any case.
In this embodiment, .theta. and D are set as follows: ##EQU12## By so setting, the optical low-pass filter 50 traps such positions as indicated by dot lines in FIG. 22. Therefore, the carriers of the color signals located at not only (.+-.1/2P.sub.H, .+-.1/4P.sub.V) (double signs being effective for the same order), but also (.+-.1/P.sub.H, 0) and (0, .+-.1/2P.sub.V), making it possible to minimize the turn-back distortions caused by the color signals and thus provide the satisfactory image quality.
The exactly same characteristics as the above can also be obtained in the case the setting the beam dividing direction of the optical low-pass filter 50 in a direction symmetrical to the direction shown in FIG. 21 about a point, i.e., in a direction turned by an angle .theta. counter-clockwise relative to the reversed scanning direction. The optical low-pass filter 50 can be made of a double refracting plate using an uniaxial crystal such as quartz. Other than this example, the filter 50 may be in any form, including a prism, so long as it has a property to divide the incident light beam into twos. In order to provide the same effect as the optical low-pass filter shown in FIG. 17, it is only required to use the optical low-pass filter 80 in combination with the optical low-pass filter 81 modified such that it is made of a double refracting plate for dividing an incident light beam into twos spaced through a distance D below in a direction turned by an angle 45.degree. counter-clockwise relative to the scanning direction: ##EQU13## and the optical low-pass filter 82 which is made of a double refracting plate for dividing an incident light beam into twos spaced through a distance P.sub.H /2 parallel to the scanning direction. In this case, spatial frequency characteristics of the optical low-pass filter 80 trap such positions as indicated by dot lines in FIG. 23. Therefore, not only the carriers of the color signals located at (.+-.1/2P.sub.H, .+-.1/4P.sub.V) (double signs being effective for the same order) are trapped, but also the carriers of the color signals located at (.+-.1/P.sub.H, 0) and (0, .+-.1/2P.sub.V) are trapped in their vicinity making it possible to achieve a sufficient degree of suppression. In addition, since the carriers of the luminance signal located at (.+-.1/P.sub.H, .+-.1/2P.sub.V) (double signs being effective regardless of the order) are further trapped, there can be obtained images in which the turn-back distortions are satisfactorily suppressed.
A sixth embodiment of the present invention will be next described.
The carrier components of the color signals at the positions (1/P.sub.H, 0) and (0, 1/2P.sub.V) in FIG. 8 are expressed below using Equation (3):
Mg(f.sub.H, f.sub.V)=Gr(f.sub.H, f.sub.V)=.alpha.(f.sub.H, f.sub.V),
Cy(f.sub.H, f.sub.V)=Ye(f.sub.H, f.sub.V)=.alpha.(f.sub.H, f.sub.V)(13)
Therefore, after conversion into the RGB signals by the RGB conversion unit 12 shown in FIG. 9, the carrier components of the RGB signals at those positions are expressed below from Equations (1), (2), (3) and (13): ##EQU14## At this time, if for each row of the matrix A the sum of coefficients of the first and second columns is equal to the sum of coefficients of the third and fourth columns, i.e., if the relationship of;
a.sub.il a.sub.i2 =a.sub.i3 +a.sub.i4
(i=1, 2, 3) (15)
holds, the carrier components of the RGB signals at the points (1/P.sub.H, 0) and (0, 1/2P.sub.V) disappear and, as a result, there occur no carrier components of the color signals at that point. Likewise, the carrier components of the color signals at the points (-1/P.sub.H, 0) and (0, -1/2P.sub.V), symmetrical to the point (1/2P.sub.H, 0) and (0, 1/2P.sub.V) about the origin in FIG. 8, can also disappear by satisfying the relationship of Equation (15) satisfied.
FIG. 24 shows the positions at which the carriers of the luminance signal and the color signals occur near the origin.
In order to minimize the turn-back distortions in that type color image sensing device, it is only required to combine the optical low-pass filter used in the third embodiment which has been dividing characteristics as shown in FIG. 15, with the optical low-pass filter used in the fifth embodiment which has beam dividing characteristics as shown in FIG. 21. In this case, the dividing distance in each of the optical low-pass filters is to meet the conditions given by Equation (7).
With this embodiment, as explained above, the horizontal resolution as high as the frequency of f.sub.H .ltoreq.1/P.sub.H, making it possible to provide the color image sensing device which has high resolution and causes fewer moires.
A seventh embodiment of the present invention will be next described.
This embodiment is intended to provide a color image sensing device in which the image sensor using the color filter arrangement of such offset sampling structure as shown in FIG. 7 or 11 is combined with an optical low-pass filter effective in suppressing color moires.
In other words, this embodiment has been made by carefully analyzing the sampling structure used in the color filter arrangement shown in FIG. 7 or 11 from the two-dimensional aspect and, as a result, deriving conditions of the optical low-pass filter optimum for preventing color moires.
First, the offset sampling structure is considered in which the horizontal and vertical pitches are P.sub.H and P.sub.V, respectively, and the horizontal offset amount is P.sub.H /2, as shown in FIG. 25. It is known that the above sampling structure corresponds to, in terms of the two-dimensional frequency plane, the offset structure in which the horizontal and vertical pitches are 2/P.sub.H and 1/P.sub.V, respectively, and the horizontal offset amount is 1/P.sub.H 2 as shown in FIG. 26. How to derive the latter from the former is described in, for example, a book by Takahiko Suisaka, "Digital Signal Processing of Image", Nikkan Kogyo Shinbunsha (1985), p. 317.
Accordingly, assuming that the luminance signal is obtained based on the so-called switch-Y method, by which the color signal from the color filter for each pixel is regarded equivalently as the luminance signal, in the image sensor using the color filter arrangement as shown in FIG. 7 or 11, the sampling structure of the luminance signal is represented by marks in FIG. 27. With regards to the color signals, it is found that Mg in FIG. 7, for example, also has the offset sampling structure in which the horizontal and vertical pitches are 2P.sub.H and 2P.sub.V, respectively, and the horizontal offset amount is P.sub.H. Therefore, the sampling structure of the color signals is represented by mark .quadrature. in FIG. 27.
As a result, an optical low-pass filter optimum for preventing color moires should be such that its frequency characteristics become zero or minimum near four sides of a rhombus indicated by solid lines in FIG. 27.
Thus, in view of the foregoing, this embodiment is directed to a color image sensing device which includes an optical filter optimum for the image sensor using the color filter arrangement as shown in FIG. 7 or 11.
The seventh embodiment will be described below in more detail. In the following description, the term "directions" indicates two opposite directions different from each other by 180.degree. and the term "direction" indicates one direction.
While ideal characteristics are rhombic as mentioned above, when constituting the optical low-pass filter by using a double refracting plate or the like in practice, it is easier to make frequency characteristics zero along four straight lines.sub.1, l.sub.1 ', l.sub.2, l.sub.2 ' passing the color carrier frequencies (1/2P.sub.H, 1/4PV), which are closest to the origin, the making angles 45.degree. with respect to the fx axis, as indicated by solid lines in FIG. 28. For this purpose, the optical low-pass filter is required to comprise a first optical member for dividing an incident light beam into twos spaced through a distance P.sub.1 in directions turned by an angle+45.degree. or -45.degree. counter-clockwise relative to the horizontal scanning direction, and a second optical member for dividing an incident light beam into twos spaced through a distance P.sub.2 in directions making 90.degree. relative to the beam dividing directions of the first optical member. In this case, the two-dimensional MTF value representing frequency characteristics of the optical low-pass filter is given by: ##EQU15##
Thus, the MTF value changes in the form of a cosine function in directions making .+-.45.degree. relative to the horizontal and in the form of a square of cosine as a result of multiplication of two terms in the horizontal and vertical directions. Consequently, the turn-back distortions are satisfactorily suppressed in the horizontal, vertical and oblique directions.
To make the optical low-pass filter exhibit sufficient performance, it is desirable that the four straight lines shown in FIG. 28 pass near the points (.+-.1/2P.sub.H, .+-.1/4P.sub.V). For this purpose, since the distances from the origin to the four straight lines are given by;
l.sub.1, l.sub.1 ':1/2P.sub.1
l.sub.2, l.sub.2 ':1/2P.sub.2
values of these distances are required to fall within a range of .+-.25% about h.sub.0 below: ##EQU16## Here, h.sub.0 represents the distance from the origin to the four straight lines l.sub.1, l.sub.1 ', l.sub.2, l.sub.2 ' when these lines pass the points (.+-.1/2P.sub.H, .+-.1/4P.sub.V). To this end, the separation (or division) widths P.sub.1, P.sub.2 are required to meet the following conditions: ##EQU17##
In each equation, if the value exceeds an upper limit, the quantity of color moires would be increased and, if it exceeds a lower limit, the resolution of the entire system comprising the optical low-pass filter and the image sensor would be decreased. Thus, exceeding the limits would lead to the unsatisfactory result in any case.
FIG. 29 shows the construction of primary parts of the color image sensing device implementing the seventh embodiment of the present invention. In FIG. 29, denotes by 21 is an optical low-pass filter and 22 is a color image sensor having the offset sampling structure, the image sensor being associated with a color filter array 22a which has the color filter arrangement as shown in FIG. 7 or 11.
The optical low-pass filter 21 comprises a first optical member 23 consisted of double refracting plates 25, 26 and a second optical member 24 made of a double refracting plate 27. As shown, the separating directions of the double refracting plates 25, 26, 27 are turned by 90.degree., 0.degree., 45.degree. counter-clockwise relative to the horizontal scanning direction and the magnitudes of the separation widths are ##EQU18## respectively. FIGS. 30A to 30C show how the light beam is separated successively by the optical low-pass filter 21 of FIG. 29. As shown in FIG. 30A, a light beam incident upon the double refracting plate 25 emerges therefrom as two linearly polarized beams, i.e., an ordinary ray 38 and an extraordinary ray 39, which have the equal intensity and are polarized in the directions indicated by solid lines in the drawing. Then, those two rays enter the double refracting plate 26 having the optical axis inclined by 90.degree. relative to the optical axis of the double refracting plate 25. At this time, since the light beam 39 becomes an ordinary ray for the double refracting plate 26, it goes straight forward through the plate 26 and emerges therefrom as is. On the other hand, since the light beam 38 becomes an extraordinary ray for the double refracting plate 26, it is refracted and emerges therefrom at a position denoted by 30. The above process is illustrated in FIG. 3B. From the first optical member 23 comprising the double refracting plates 25, 26, there are eventually obtained two light beams 39, 30 separated in directions turned by -45.degree. counter-clockwise relative to the horizontal scanning direction with the separation width of P.sub.1, as shown. Since those light beams 39, 30 are respectively polarized by 90.degree. and 0.degree. counter-clockwise relative to the horizontal scanning direction, they are separated by the double refracting plate 27 into fours which have the equal intensity, as illustrated in FIG. 30C. The separation width effected by the double refracting plate 27 is P.sub.2. The values of P.sub.1 and P.sub.2 are set to meet above Equations (2) and (3), respectively, for achieving the desired performance.
FIG. 31 shows the construction of primary parts of an eighth embodiment of the present invention. A color image sensor 42 and a color filter array 42a associated therewith are constituted in the same manner as the seventh embodiment. An optical low-pass filter 90 comprises a first optical member 92 consisted of a phase plate 94 for converting linearly polarized light into circularly polarized light and a double refracting plate 95, and a second optical member 91 made of a double refracting plate 93. The first optical member 92 and the second optical member 91 are not necessarily arranged in the above order from the side of the image sensor 42, and may be arranged in the reversed order. Two linearly polarized beams, i.e., an ordinary ray and an extraordinary ray, separated by the double refracting plate 93 with the separation width of P.sub.2 are converted by the phase plate 94 into two circularly polarized beams which are then separated by the double refracting plate 95, having the separation width of P.sub.1, into fours of the equal intensity similarly to the case of in FIG. 30C.
FIG. 32 shows the construction of primary parts of a ninth embodiment of the present invention. An optical low-pass filter 100 in this embodiment uses a double refracting plate 96 which has the separation width sufficiently smaller than the two double refracting plates 93, 95 and also has the separating directions turned by 0.degree. counterclockwise relative to the horizontal scanning direction, in place of the phase plate 94 in the optical low-pass filter 90 of FIG. 31. Assuming that the separation width of the double refracting plate 96 is d.sub.3, there emerge from the optical low-pass filter 100a total of eight light beams resulted from double-shifting of the four light beams in FIG. 30C by d.sub.3 in the directions of 45.degree.. Because of d.sub.3 being sufficiently small, however, the resulting frequency characteristics are almost the same as those obtained in FIG. 30C. With the ninth embodiment, dependency of the frequency characteristics on wavelength is reduced as compared with the eighth embodiment using the phase plate 94 and the thickness of the entire device is reduced for more compact size.
In the foregoing embodiments, the optical member is constituted by a single double refracting plate, a plurality of double refracting plates, or a combination of a phase plate therewith. However, the optical member for use in the present invention is not limited to those embodiments and may be in any form, e.g., a prism placed in the optical system, so long as it has a capability to divide the light beam into twos.
With the seventh to ninth embodiments, as explained above, there can be obtained an optical low-pass filter which is combined with a color single-plate image sensor having the offset sampling structure to effectively prevent the occurrence of color moires.
A tenth embodiment of the present invention will be next described. This tenth embodiment is directed to a color image sensing device which includes an optimum optical filter as with the above seventh, eighth and ninth embodiments.
The tenth embodiment will be explained below in more detail. In the following description, the term "directions" indicates two opposite directions different from each other by 180.degree. and the term "direction" indicates one direction.
While ideal characteristics are rhombic as mentioned above, when constituting the optical low-pass filter by using a double refracting plate or the like in practice, it is easy to make frequency characteristics zero along not only four straight lines l.sub.1, l.sub.1 ', l.sub.3, l.sub.3 ' passing the color carrier frequencies (.+-.1/2P.sub.H, .+-.1/4P.sub.V), which are closest to the origin, and making angles .+-.45.degree. with respect to the f.sub.x axis, but also straight lines l.sub.2, l.sub.2 ' parallel to the f.sub.y axis, as indicated by solid lines in FIG. 28. For this purpose, the optical low-pass filter is required to comprise a first optical member for dividing an incident light beam into twos spaced through a distance P.sub.1 in directions turned by an angle .+-.45.degree. or -45.degree. counterclockwise relative to the horizontal scanning direction, a second optical member for dividing an incident light beam into twos spaced through a distance P.sub.2 in directions parallel to the horizontal scanning direction, and a third optical member for dividing an incident light beam into twos spaced through a distance P.sub.3 in directions making 90.degree. relative to the beam dividing directions of the first optical member. In this case, the two-dimensional MTF value representing frequency characteristics of the optical low-pass filter is given by: ##EQU19##
Accordingly, the MTF value can trap the vicinity of (.+-.1/2P.sub.H, .+-.1/4P.sub.V) where the color signal carriers occur, without deteriorating the horizontal resolution, and also trap the vicinity of (.+-.1/P.sub.H, .+-.1/2P.sub.V) where the luminance signal carriers occur. It is thus possible to suppress all the carriers of the color and luminance signals which are nearest to the origin.
To make the optical low-pass filter exhibit sufficient performance, since the distance from the origin to the four straight lines shown in FIG. 28 are given by;
l.sub.1, l.sub.1 ':1/2P.sub.1
l.sub.3, l.sub.3 ':1/2P.sub.3 ;
values of these distances are required to fall within a range of +25% about h.sub.0 below; ##EQU20## and the distance from the origin to the two straight lines;
1/2P.sub.2
is required to fall within a range of .+-.25% about 1/P.sub.H. Here, the value of h.sub.0 represents the distance from the origin to the four straight lines l.sub.1, l.sub.1 ', l.sub.2, l.sub.2 ' when these lines pass the points (.+-.1/2P.sub.H, .+-.1/4P.sub.V).
For that purpose, the separation (or division) widths P.sub.1, P.sub.2, P.sub.3 are required to meet the following conditions: ##EQU21##
In each equation, if the value exceeds an upper limit, the quantity of color moires caused by the carriers of the luminance and color signals would be increased and, if it exceeds a lower limit, the resolution of the entire system comprising the optical low-pass filter and the image sensor would be decreased. Thus, exceeding the limits would lead to the unsatisfactory result in any case.
FIG. 33 shows the construction of primary parts of the color image sensing device implementing the tenth embodiment of the present invention. In FIG. 33, denoted by 51 is an optical low-pass filter and 52 is a color solid-state image sensor having the offset sampling structure, the image sensor being associated with a color filter array 52a which has the color filter arrangement as shown in FIG. 7 or FIG. 11.
The optical low-pass filter 51 comprises first, second and third optical members 53, 54, 55 made of double refracting plates 56, 57, 58, respectively. As shown, the separating directions of the double refracting plates 56, 57, 58 are turned by 45.degree., 0.degree., -45.degree. counterclockwise relative to the horizontal scanning direction and the magnitudes of the separation widths are ##EQU22## respectively.
FIGS. 34A to 34C show how the light beam is separated successively by the optical low-pass filter 1 of FIG. 35. As shown in FIG. 34A, an incident light beam is divided by the double refracting plate 56 in directions turned by 45.degree. counterclockwise relative to the horizontal scanning direction to emerge therefrom as two linearly polarized beams, i.e., an ordinary ray 69 and an extraordinary ray 60, which have the equal intensity and are polarized in the directions indicated by solid lines in the drawing. Then, those two rays 69, 60 emerging from the double refracting plate 56 are divided by the double refracting plate 57 in parallel to the horizontal scanning direction into fours 69, 60, 61, 62, as shown in FIG. 34B. The polarizing directions are indicated by solid arrows in the drawing. Subsequently, as shown in FIG. 34C, the light beams 69, 60, 61, 62 emerging from the double refracting plate 57 are divided in directions turned by -45.degree. counterclockwise relative to the horizontal scanning direction into eights 69, 60, 61, 62, 63, 64, 65, 66. Eventually, those eight light beams emerge from the optical low-pass filter 51. In this case, the two-dimensional MTF value representing frequency characteristics of the optical low-pass filter is given by: ##EQU23## Accordingly, it becomes possible to satisfactorily suppress the occurrence of moires caused by the carriers of the luminance and color signals as seen from FIG. 28.
FIG. 35 shows the construction of primary parts of a color image sensing device according to an eleventh embodiment of the present invention. A color image sensor 120 and a color filter array 120a associated therewith are constituted in the same manner as the tenth embodiment. An optical low-pass filter 110 comprises a first optical member 111 consisted of a double refracting plate 114 and a phase plate 115 for converting linearly polarized light into circularly polarized light, a second optical member 112 consisted of a double refracting plate 116 and a phase plate 117, and a third optical member 113 made of a double refracting plate 118. The double refracting plate 114 divides an incident light beam into twos, i.e., an ordinary ray and an extraordinary ray, spaced through a distance P.sub.1 in directions turned by 45.degree. counterclockwise relative to the horizontal scanning direction. Those two rays emerging from the double refracting plate 114 are converted by the phase plate 115 from linearly polarized light into circularly polarized light. The light beams emerging from the phase plate 115 are divided by the double refracting plate 116 through a distance P.sub.2 in directions parallel to the horizontal scanning direction, and then converted by the phase plate 117 from linearly polarized light into circularly polarized light. The light beams emerging from the phase plate 117 are divided by the double refracting plate 118 through a distance P.sub.3 in directions turned by -45.degree. counterclockwise relative to the horizontal scanning direction. Eventually, a total of eight light beams emerge from the optical low-pass filter 90 similarly to the case shown in FIG. 34C. Therefore, the two-dimensional MTF value of the optical low-pass filter 90 is expressed by above Equation
(1) and thus can provide the same frequency characteristics as the tenth embodiment.
In the tenth and eleventh embodiments, the first, second and third optical members are not necessarily arranged in the above order from the object side, and may be arranged in any order so long as the optical low-pass filter has frequency characteristics meeting Equation (1). Furthermore, in those embodiments, the optical member is constituted by a single double refracting plate or a combination of a phase plate therewith. However, the optical member for use in the present invention is not limited to those embodiments and may be in any form, e.g., a prism placed in the optical system, so long as it has a capability to divide the light beam into twos.
With the tenth and eleventh embodiments, as explained above, there can be obtained an optical low-pass filter which is combined with a color single-plate image sensor having the offset sampling structure to effectively prevent the occurrence of color moires.
Meanwhile, in the case of the aforementioned color image sensing device of the first embodiment as shown in FIG. 9, i.e., when the signal from the sensor 1 is simply subjected to low-pass filtering by the interpolation filter 4 to produce the luminance signal, there occurs the following phenomenon.
Generally, responses of four color filters Mg, Cy, Ye, Gr to some constant object are a little different from one another. Therefore, even if luminance of the object is constant, the arrangement structure of FIG. 7 is reflected in the luminance signal resulted from the signal processing through the system of FIG. 9 and thus appears in the form of checker patterns.
When the four color filters are arranged as shown in FIG. 7, characteristics of the color light carriers are given as shown in FIG. 8. The above phenomenon can be regarded as a phenomenon caused by the DC component turning back to the carrier position indicated by P in FIG. 8. Preventing that phenomenon in the simplest manner requires to carry out signal processing such that response of the carrier at the point P in FIG. 8 becomes zero. For example, by setting a luminance signal Y at the position of Gr.sup.(1) in FIG. 7 as follows; ##EQU24## a horizontal low-pass filter for trapping the horizontal frequency component 1/2P.sub.H in FIG. 8 is realized. Accordingly, the point P is also trapped and the checker patterns are removed.
Also, by adding information data on every two pixels spaced from each other by two pixels in the vertical direction, a vertical low-pass filter for trapping the vertical frequency component 1/4P.sub.V can be realized to remove the checker patterns.
However, if the above horizontal or vertical processing is performed, the horizontal or vertical band would be so limited as to deteriorate the resolution.
A twelfth embodiment of the present invention is intended to solve the foregoing problem and is directed to an image signal processor which can remove the checker patterns in the image sensing device with the filter as shown in FIG. 7, without reducing the resolution.
To this end, the image signal processor of this embodiment is constituted as follows:
(1) an image signal processor comprising decision means for determining whether or not a constant region exhibiting small luminance change in a predetermined direction is present within an image, processing means for carrying out low-pass filtering of a luminance signal in the predetermined direction, and selection means for selecting the processing means to carry out the low-pass filtering when the decision means determines the presence of the constant region; or
(2) an image signal processor in which the decision means makes a decision based on absolute values of outputs of horizontal and vertical band-pass filters for the luminance signal.
With the image signal processors of (1) and (2), when the constant region exhibiting small luminance changes in a predetermined direction is present within an image, low-pass filtering of the luminance signal is carried out in the predetermined direction.
With the image signal processors of (2), the division of the constant region exhibiting small luminance changes is made based on absolute values of outputs of horizontal and vertical band-pass filters for the luminance signal.
The twelfth embodiment will be described below in more detail. FIG. 36 is a block diagram of a signal processor for a video camera which implements the twelfth embodiment.
A sensor 301 is associated with the color filter array shown in FIG. 7 for reading two rows of the array in the zigzag form during one horizontal scan period (1H) as seen from the drawing. Accordingly, all the pixels are read out once for one field. Assuming now that the read-out clock has frequency of f.sub.s, an output of the sensor 301 is changed in the order of Mg, Cy, Gr, Ye for each clock. The output signal is applied to an analog processing unit 302 for signal processing such as AGC (automatic gain control), and then subjected to A/D (analog-to-digital) conversion by an A/D converter 303. The A/D converter used desirably has 10 or more bit precision.
The resulting digital signal is inputted to a later-described adaptive filter processing unit 305 for a luminance signal to create a high-frequency luminance signal free of checker patterns as mentioned above. The resulting signal is applied to a luminance processing unit 306 for standard video signal processing such as .gamma. conversion and blanking, and then subjected to D/A (digital-to-analog) conversion by a D/A converter 307. The resulting analog signal is added in a synch signal adding unit 308 with a synch signal for a standard TV signal.
A switch 304 is operated to change over an input signal for each half of the clock f.sub.s so that a signal Mg-Gr is outputted to S1 and a signal Cy-Ye is outputted to S2.
Those signals changed over to S1 and S2 are inputted to the switches 309 and 310, respectively, so that they are separated into signals Mg, Gr, Cy, Ye for respective signals. Since these color signals each have the offset structure as shown in FIG. 7, lacked information is interpolated by two-dimensional interpolation filters 311, 312, 313, 314.
In a matrix operation unit 315, primary color signals R, G, B are obtained from the four interpolated color signals Mg, Gr, Cy, Ye through matrix operation below: ##EQU25## Here, the matrix A is a matrix of 3 row and 4 column optimized to make spectroscopic characteristics Mg(.lambda.), Gr(.lambda.), Cy(.lambda.), Ye(.lambda.) of the sensor 201 for Mg, Gr, Cy, Ye approach ideal spectroscopic characteristics Mg(.lambda.), Gr(.lambda.), Cy(.lambda.) for RGB specified in the NTSC system.
The primary color signals R, G, B thus obtained are applied to a color processing unit 316 for signal processing such as white balance (WB) and .gamma. conversion, following which they are inputted to a color difference matrix operation unit 317. The color difference matrix operation unit 317 creates two color difference signals R-Y, B-Y which are subjected to D/A conversion by D/A converters 318, 319, respectively.
After that, the color difference signals are subjected to orthogonal modulation by an encoder 320 for mixing in a mixing unit 321 with the signal, comprising the luminance signal Y and the synch signal S, and the mixed result is outputted as the standard TV signal.
FIG. 37 shows the construction of the adaptive filter processing unit 305 for the luminance signal. The inputted luminance signal Y is the result of reading the pixels in the zigzag form in the order of Mg, Cy, Gr and Ye. Since carriers are present at positions corresponding to the frequency (1/2P.sub.H) equal to 1/4 of the clock f.sub.s in the horizontal direction and 240 TV lines (1/4P.sub.V) in the vertical direction, as mentioned before, it is required to determine the magnitudes of frequency components at those carrier positions.
A horizontal band-pass filter 401 comprises, as shown in the lower half of FIG. 38, a digital filter with five taps of (-1/2, 0, 1, 0, -1/2). A vertical band-pass filter 402 comprises, as shown in the lower half of FIG. 39, a digital filter with three taps of (-1/2, 1, -1/2) by using a 1H (horizontal scan period) memory.
A decision unit 403 outputs a select signal S to a switch 407 depending on absolute values of outputs of the horizontal band-pass filter 401 and the vertical band-pass filter 402.
In response to the select signal S from the decision unit 403, the switch 407 is able to select one of a through signal output 404, an output of a horizontal low-pass filter 405, and an output of a vertical low-pass filter 405. The horizontal low-pass filter 405 can be constituted by, as shown in the upper half of FIG. 38, sharing delays and taps with the horizontal band-pass filter 401. Here, the horizontal low-pass filter 405 comprises a low-pass filter with five taps of (1/2, 0, 1, 0, 1/2) to make response zero along the line T.sub.H in FIG. 8, thereby enabling to remove the checker patterns. Also, the vertical low-pass filter 406 can be constituted by, as shown in the upper half of FIG. 39, sharing the 1H memory and taps with the vertical band-pass filter 401. Here, the vertical low-pass filter 406 comprises a low-pass filter with three taps of (1/2, 1, 1/2) to make response zero along the line T.sub.V in FIG. 8, thereby enabling to remove the checker patterns. Operation of the decision unit 403 can be controlled in accordance with the following table:
______________________________________Case Selection______________________________________1 A .andgate. B S12 .sup.-- A .andgate. B S23 A .andgate. .sup.-- B S34 .sup.-- A .andgate. .sup.-- B S2 or S3______________________________________
It is here assumed that A represents an event that an absolute value of the output of the horizontal band-pass filter 401 is larger than a certain value, and B represents an event that an absolute value of the output of the vertical band-pass filter 402 is larger than a certain value. A, B respectively represent reversed events of A, B.
For example, in Case 1 where the object structure is largely changed in both the horizontal and vertical directions, there occur A and B. In this case, the checker patterns will not be awkward even if they are present. Conversely, if the signals are subjected to the low-pass filtering, the image would be blurred. Therefore, the switch S11 is selected.
As another example, in Case 2 where the object has the vertical component, the switch S12 is selected to effect the low-pass filtering in the horizontal direction. By so doing, the checker patterns can be removed without impairing information on the object.
In the above embodiment, the outputs of the horizontal and vertical band-pass filters are combined with each other to determine whether or not the object structure is abruptly changed in the portion of interest. Alternatively, it is also possible to determine changes in the object structure in oblique directions by using oblique band-pass filters and effect the oblique low-pass filtering if the changes are small.
With this embodiment, as described above, the checker patterns can be prevented without deteriorating the resolution.
Meanwhile, in the first embodiment, the color signals are processed after conversion of the four complementary color signals Mg, Cy, Ye, Gr, having been interpolated, into the primary color signals R, G, B through the matrix operation, as shown in FIG. 9. This color processing technique is called a matrix method or primary color separating method and described in the specification of Japanese Patent Application No. 63-281456 filed by the present applicant and the article "Simultaneous RGB Processing in Image Mixing CCD Camera" by Nishimura, etc., Technical Report of Television Society of Japan, TEBS 89-9 ED 89-13, Feb. 1989.
In the RGB conversion unit 12 of FIG. 9, the four complementary color signals Mg, Cy, Ye, Gr are converted into the primary color signals R, G, B through the matrix operation below: ##EQU26##
On this occasion, the oblique pattern of bright and dark areas are considered as shown in FIG. 40. The fat lines represent bright areas and the thin lines represent dark areas. That pattern corresponds to a wave at the point shown in FIG. 8.
For the wave in the above pattern, as will be seen from FIG. 40, response to Mg and Cy is large, while response to Gr and Ye is small. Accordingly, if the signals R, G, B are calculated in accordance with Equation (1), a false color would be caused in spite of an achromatic object.
Further, for the pattern as shown in FIG. 41, response to Mg and Ye is large, while response to Gr and Cy is small, thus resulting in the similar problem.
A thirteenth embodiment of the present invention is made to solve the above problem and is directed to an image signal processor which can suppress the occurrence of false colors and realize high image quality. The image signal processor comprises image pattern detecting means for detecting an image pattern which tends to cause a false color, color signal processing means for making color signal processing adapted for the image pattern, and selection means for selecting the color signal processing means depending on an output of the image pattern detecting means to make the color signal processing.
The color signal processing means may carry out the process for converting color signals into achromatic signals.
With the image signal processor thus arranged, upon detection of the image pattern which tends to cause a false color, the color signal processing adapted for the detected image pattern is carried out.
The thirteenth embodiment will be described below in more detail. FIG. 42 is a block diagram of a signal processor for a video camera which implements the thirteenth embodiment.
A sensor 501 is associated with the color filter array shown in FIG. 7 for reading two rows of the array in the zigzag form during one horizontal scan period (1H) as seen from the drawing. Accordingly, all the pixels are read out once for one field. Assuming now that the read-out clock has frequency of fs, an output of the sensor 501 is changed in the order of Mg, Cy, Gr, Ye for each clock. The output signal is applied to an analog processing unit 502 for signal processing such as AGC (automatic gain control), and then subjected to A/D (analog-to-digital) conversion by an A/D converter 503. The A/D converter used desirably has 10 or more bit precision.
The resulting digital signal is inputted to an interpolation filter 504 and then to a luminance processing unit 505 for standard video signal processing such as .gamma. conversion and blanking, followed by D/A (digital-to-analog) conversion by a D/A converter 506. Further, the resulting analog signal is added in a period signal adding unit (not shown) with a synch signal for a standard TV signal.
On the other hand, the output of the A/D converter 503 is separated into color signals Mg, Gr, Cy, Ye and then interpolated two-dimensionally in interpolation filters 507 to 510. This process is described in the specification of Japanese Patent Application No. 63-281456 filed by the present applicant, for example, and hence will not be explained further herein. RGB conversion units 512, 513, 514 convert the four color signals Mg, Gr, Cy, Ye into three primary color signals R, G, B in accordance with above Equation (1) using matrix coefficient different from one another.
A matrix selection unit 511 determines, as described later, which one of the RGB conversion units should be used to minimize the occurrence of false colors, and the decision result is outputted to a switch 515. The switch 515 selects one set of RGB outputs from the RGB conversion units 512, 513, 514 depending on the decision result of the matrix selection unit 511.
Differences among the RGB conversion units 512, 513, 514 will be next explained.
The RGB conversion unit 512 performs the matrix operation for to minimize the occurrence of false colors for the bright and dark patterns shown in FIG. 40. Specifically, for the pattern of FIG. 40 where it is assumed that Mg and Cy are bright, while Gr and Ye are dark, R is expressed below in accordance with above Equation (1):
R=a.sub.11 Mg+a.sub.12 Gr+a.sub.13 Cy+a.sub.14 Ye (2)
As explained in the above-cited reference, too, a false color Rm of the red color for the pattern of FIG. 40 is expressed by multiplying +1 to the terms of Mg and Cy and -1 to the terms of Gr and Ye as follows:
Rm=a.sub.11 Mg-a.sub.12 Gr+a.sub.123 Cy-a.sub.14 Ye (3)
When Rm becomes zero, the occurrence of false colors is minimized. Thus:
a.sub.11 Mg+a.sub.13 Cy=a.sub.12 Gr+a.sub.14 Ye (4)
Since the above discussion is similarly applied to other colors, the conditions to be met are expressed below:
a.sub.k1 Mg+a.sub.k3 Cy=a.sub.k2 Gr+a.sub.k4 Ye (5)
A method of determining the matrix coefficients, which are also optimized in color reproduction under the restricting conditions of (5), is described in the above-cited Japanese Patent Application No. 63-281456. As a result of executing the matrix operation using the thus-determined coefficients in the RGB conversion unit 512, the output of the unit 512 is modified such that the occurrence of false colors is minimized for the pattern of FIG. 40.
Likewise, the conditions of minimizing the occurrence of false colors for the pattern of FIG. 41 is given by:
a.sub.k1 Mg+a.sub.k4 Ye=a.sub.k2 Gr+a.sub.k3 Cy (6)
As a result of executing the matrix operation, which is optimized in color reproduction under the restricting conditions of (6), in the RGB conversion unit 513, the output of the unit 513 is modified such that the occurrence of false colors is minimized for the pattern of FIG. 41.
The RGB conversion unit 514 performs the operation of converting the color signals into achromatic signals when both the patterns of FIGS. 40 and 41 are overlapped with each other in the color matrix. Means for conversion into achromatic signals is implemented by utilizing the following relationship, for example:
a.sub.11 =a.sub.21 =a.sub.31 (1=1, 2, 3, 4) (7)
When both the patterns of FIGS. 40 and 41 are overlapped with each other, this corresponds to an edge region of the object. In such a region, the image quality will be less deteriorated even if the color signals are converted into achromatic signals.
FIG. 43 shows the construction of the matrix selection unit 511. The signal from the sensor 501 through zigzag reading of the pixels is inputted to the matrix selection unit 511 while repeating the cycle of Mg, Cy, Gr and Ye in this order. Accordingly, the result of adding the input signal and the signal which is obtained by delaying the input signal one data in a delay 601, in an adder 602 is given by repetition of Mg+Cy/Cy+Gr/Gr+Ye/Ye+Mg. The result of taking the difference between the above summation signal and the signal which is obtained by delaying the summation signal two data in delay 603 and 604, in a subtracter 605 is given by repetition of (Mg+Cy)-(Gr+Ye)/(Cy+Gr)-(Ye+Mg)/(Gr+Ye)-(Mg+ Cy)/(Ye+Mg)-(Cy+Gr). If the pattern component of the object as shown in FIG. 41 is large, an absolute value of (Cy+Gr)-(Ye+Mg) is increased. If the pattern component of FIG. 40 is large, an absolute value of (Mg+Cy)-(Gr+Ye) is increased. Accordingly, by changing over the data on a data-by-data basis by a switch 606, the output is separated such that .+-.[(Mg+Cy)-(Gr+Ye)] is led to an upper absolute value detecting unit 607 and .+-.[(Cy+Gr)-(Ye+Mg)] is led to a lower absolute value detecting unit 608. The absolute value detecting units 607, 608 take absolute values of those outputs, which are then compared with present thresholds Th1, Th2 in comparators 609, 610, thereby producing logical outputs P, Q. P is 1 if .gtoreq.Th1 and 0 if <Th1, while Q is 1 if .gtoreq.Th2 and 0 if <Th2. A decision logic 611 determines the matrix to be used by using the logic table shown in FIG. 44.
The conversion of the color signals into the achromatic signals is not always required to be performed in the RGB conversion unit 514, and may be achieved by using the so-called chroma-killer process to make the color difference signals R-Y, B-Y zero in the color signal processing unit 516.
Further, it is also possible to input a horizontal address X and a vertical address Y of the pixel to the switch 515 and, if the outputs of P and Q are zero, to switch over M1 and M2 alternately as shown in FIG. 45.
The color filters of the sensor 501 are not limited to Mg, Cy, Gr, Ye and may have any form, including pure such as R, G.sub.1, G.sub.2, B, so long as four colors are arranged in the offset structure.
In addition, this embodiment treats the two image patterns which tend to cause false colors, and includes the three color signal processing means 512 to 514 adapted for those image patterns. However, this embodiment is not limited to such an arrangement and may be implemented in any form including one or more image patterns which tend to cause false colors, and one or more color signal processing means adapted for the respective image patterns.
With this embodiment, as explained above, there can be obtained an image signal processor which has high resolution and can realize high image quality, while reducing the occurrence of false colors, even for such an image pattern tending to cause false colors.

Number	Date	Country
2-290819	Oct 1990	JPX
2-290820	Oct 1990	JPX
2-295212	Nov 1990	JPX
2-297071	Nov 1990	JPX
2-297072	Nov 1990	JPX
2-299028	Nov 1990	JPX

Number	Name	Date
4754323	Kaji et al.	Jun 1988
4974066	Tsuji	Nov 1990
5034805	Ishizaka	Jul 1991

Color image sensing device

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

US Classifications

Field of Search

US

International Classifications

Abstract

Description

Claims

Priority Claims (6)

US Referenced Citations (3)

Foreign Referenced Citations (1)