Patent Application
20150002710
1. Field of the Invention
Apparatus and methods consistent with the present invention relate to an imaging apparatus and a method of improving the sensitivity of the same, and more particularly, to an imaging apparatus, which can improve image data sensitivity under low illumination conditions and prevent noise, and a method of improving the sensitivity of the imaging apparatus.
2. Description of the Related Art
Imaging apparatuses, such as cameras or camcorders, convert light of an image into an electric signal, using an imaging device such as a complementary metal-oxide semiconductor (CMOS) or a charge-coupled device (CCD).
Ideally, imaging devices produce an electric signal in proportion to the amount of incident light. However, various kinds of noise are generated when light is converted into an electric signal. Such noise includes dark current noise, kTC noise, and fixed pattern noise.
Dark current noise, which is thermal noise proportional to the temperature, is a major factor in image quality degradation under low illumination conditions. kTC noise is generated by various switching pulses that are used to drive a CMOS or a CCD camera. Fixed pattern noise results from non-uniformity caused by manufacturing variations between pixels in an imaging device, such as a CMOS or a CCD. Fixed pattern noise includes a white spot defect, a black spot defect, a line defect, a banded defect, and a sensitivity speck. Such noise is added to charges which are photoelectrically converted and accumulated by the imaging device, thereby degrading image quality.
Under high illumination conditions with a great amount of light, since noise is relatively small compared to photoelectrically converted and accumulated charges, image quality degradation is negligible. However, under low illumination conditions, fixed pattern noise, dark current noise, and kTC noise become greater than photoelectrically converted and accumulated charges.
In order to make photoelectrically converted and accumulated charges larger than noise under low illumination conditions, an imaging device having a large pixel pitch may be used, or the exposure time of an imaging device may be increased. However, the imaging device having the large pixel pitch is expensive and the size of the imaging device should be increased to provide the same resolution.
The present invention provides an imaging apparatus, which can preserve resolution and can also expand the dynamic range of an image signal under low illumination conditions by increasing the power of an output image signal in comparison to the power of noise, and a method of improving the sensitivity of the imaging apparatus.
The present invention also provides an imaging apparatus, which can improve sensitivity by preventing noise boost-up, and a method of improving the sensitivity of the imaging apparatus.
According to an aspect of the present invention, there is provided an imaging apparatus comprising: a pixel binning unit pixel binning input image data to a given pixel size; a gain determining unit determining a pixel binning gain based on the input image data or the brightness of the input image data; and a calculating unit calculating output image data based on the pixel binned input image data and the pixel binning gain.
The pixel binning unit may preserve the resolution of the input image data.
The gain determining unit may determine the pixel binning gain as a given maximum gain when the brightness of the input image data is less than a first threshold, and determine the pixel binning gain as a gain, which linearly decreases from the given maximum gain as the brightness of the input image data increases, when the brightness of the input image data is greater than the first threshold.
The calculating unit may calculate the output image data by multiplying the pixel binned input image data by the pixel binning gain.
The imaging apparatus may further comprise a temporal expansion unit expanding the dynamic range of the output image data based on current frame data and previous frame data of the output image data.
The gain determining unit may determine a data merge gain of the temporal expansion unit and provide the data merge gain to the temporal expansion unit.
According to another aspect of the present invention, there is provided an imaging apparatus comprising: a pixel binning unit pixel binning input image data to a given pixel size; a high pass filtering unit filtering high frequency components in a plurality of directions of the input image data; a resolution preserving factor determining unit determining a resolution preserving factor based on the high frequency components; and a calculating unit calculating output image data based on the pixel binned input image data, the high frequency components, and the resolution preserving factor.
The high pass filtering unit may comprise: a horizontal high pass filter filtering a first high frequency component in a horizontal direction of the input image data; and a vertical high pass filter filtering a second high frequency component in a vertical direction of the input image data.
The high pass filtering unit may further comprise a diagonal high pass filter filtering a third high frequency component in a diagonal direction of the input image data.
The resolution preserving factor determining unit may: obtain a maximum absolute value among absolute values of a difference between the first and second high frequency components, a difference between the second and third high frequency components, and a difference between the first and third high frequency components; determine the resolution preserving factor as a given minimum factor when the maximum absolute value is less than or equal to a second threshold; determine the resolution preserving factor as a given maximum factor when the maximum absolute value is greater than or less than a third threshold; and determine the resolution preserving factor as a gain, which linearly increases as the maximum absolute value increases between the minimum factor and the maximum factor, when the maximum absolute value is between the second threshold and the third threshold.
The calculating unit may calculate the output image data by multiplying a sum of the high frequency components by the resolution preserving factor and adding the pixel binned input image data to the multiplication result.
The imaging apparatus may further comprise a temporal expansion unit expanding the dynamic range of the output image data based on current frame data and previous frame data of the output image data.
According to another aspect of the present invention, there is provided a method of improving the sensitivity of an imaging apparatus, the method comprising: pixel binning input image data to a given pixel size; determining a pixel binning gain based on the input data or the brightness of the input image data; and calculating output image data based on the pixel binned input image data and the pixel binning gain.
According to another aspect of the present invention, there is provided a method of improving the sensitivity of an imaging apparatus, the method comprising: pixel binning input image data to a given pixel size; filtering high frequency components in a plurality of directions of the input image data; determining a resolution preserving factor based on the high frequency components; and calculating output image data based on the pixel binned input image data, the high frequency components, and the resolution preserving factor.
According to another aspect of the present invention, there is provided a computer-readable recording medium having embodied thereon a program for implementing a method of improving the sensitivity of an imaging apparatus, wherein the method comprises: pixel binning input image data to a given pixel size; determining a pixel binning gain based on the input image data or the brightness of the input image data; and calculating output image data based on the pixel binned input image data and the pixel binning gain.
According to another aspect of the present invention, there is provided a computer-readable recording medium having embodied thereon a program for implementing a method of improving the sensitivity of an imaging apparatus, wherein the method comprises: pixel binning input image data to a given pixel size; filtering high frequency components in a plurality of directions of the input image data; determining a resolution preserving factor based on the high frequency components; and calculating output image data based on the pixel binned input image data, the high frequency components, and the resolution preserving factor.
The above and other aspects of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings, in which:
The present invention will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown.
Referring to
The pixel binning unit 120 pixel bins input image data 110 to a predetermined pixel size, for example, 2×2 or 3×3. A plurality of pixel data adjacent to one pixel are combined into one pixel data. Since pixel binning is a process of combining data of a plurality of pixels into data of one pixel, sensitivity can be improved but resolution is reduced under low illumination conditions.
However, the pixel binning unit 120 of the imaging apparatus of
Likewise,
In this manner, input image data for all pixels ‘a’, ‘b’, ‘c’, . . . ‘p’ can be obtained. Accordingly, the pixel binning of
Referring to
The calculating unit 140 calculates the output image data 150 based on the pixel binned image data output from the pixel binning unit 120 and the pixel binning gain output from the gain determining unit 130. Although the calculating unit 140 is shown to calculate the output image data 150 by multiplying the output of the pixel binning unit 120 by the pixel binning gain in
According to the imaging apparatus of
Referring to
Since the pixel binning unit 120 and the calculating unit 140 are the same as those of
The temporal expansion unit 250 expands the dynamic range of output image data 260 based on current frame data and previous frame data of image data output from the calculating unit 140.
The gain determining unit 230 determines a pixel binning gain, and also determines a data merge gain, which is a ratio at which the current frame data and the previous frame data merge with each other, and provides the determined data merge gain to the temporal expansion unit 250. The ratio at which the current frame data and the previous frame data are merged with each other may be varied depending on a motion between frames.
The gain determining unit 230 and the temporal expansion unit 250 will be explained later in detail with reference to
Referring to
The pixel binning unit 320 pixel bins input image data to a predetermined pixel size. The pixel binning unit 320 can preserve the resolution of input image data 310.
The high pass filtering unit 330 filters high frequency components in a plurality of directions of the input image data 310. The filtering of the high frequency components in the plurality of directions comprises judging whether a high frequency component of a pixel is generated by an image or a noise. In general, when there is a high frequency component in a certain direction, existence of an edge in the direction can be detected. Accordingly, if all high frequency components filtered in horizontal, vertical, and diagonal directions of one pixel have high values, the one pixel may be detected as a noise.
The resolution preserving factor determining unit 340 determines a resolution preserving factor based on the high frequency components. If the high pass filtering unit 330 judges that a high frequency component of a pixel is generated by an image, the resolution preserving factor determining unit 340 determines the resolution preserving factor to maintain the high frequency component, whereas if the high pass filtering unit 330 judges that the high frequency component of the pixel is generated by a noise, the resolution preserving factor determining unit 340 determines the resolution preserving factor not to maintain the high frequency component. The resolution preserving factor determining unit 340 will be explained later with reference to
The calculating unit 350 calculates output image data 360 based on the pixel binned input image data, the high frequency components, and the resolution preserving factor.
Referring to
The pixel binning unit 415 pixel bins input image data 410 to a predetermined pixel size. The pixel binning unit 415 preserves the resolution of input image data 410.
The high pass filtering unit 420 includes a horizontal high pass filter 425, a vertical high pass filter 430, and a diagonal high pass filter 435.
The horizontal high pass filter 425 filters a first high frequency component ‘H_hpf’ in a horizontal direction of the input image data 410. The vertical high pass filter 430 filters a second high frequency component ‘V_hpf’ in a vertical direction of the input image data 410. The diagonal high pass filter 435 filters a third high frequency component ‘D_hpf’ in a diagonal direction of the input image data 410.
In modifications, the high pass filtering unit 420 may include only two filtering units, e.g., the horizontal high pass filter 425 and the vertical high pass filter 430, or may include four or more filtering units.
The resolution preserving factor determining unit 440 calculates the absolute value |H_hpf−V_hpf| of a difference between the first and second high frequency components H_hpf and V_hpf, the absolute value |V_hpf−D_hpf| of a difference between the second and third high frequency components V_hpf and D_hpf, and the absolute value |H_hpf−D_hpf| of a difference between the first and third high frequency components H_hpf−D_hpf. Next, the resolution preserving factor determining unit 440 obtains a maximum Diff_Max, which is the largest of the three absolute values. That is, the maximum Diff_Max is given by ‘Diff_Max=max(|H_hpf−V_hpf|, |V_hpf−D_hpf|, |H_hpf−D_hpf|)’.
Next, the resolution preserving factor determining unit 440 determines a resolution preserving factor based on the obtained maximum Diff_Max.
Referring to
Referring to
The gain determining unit 445 also determines a data merge gain, which is a ratio at which current frame data and previous frame data are merged with each other, from first output image data 475, and provides the determined data merge gain to a temporal expansion unit 480. Accordingly, both spatial and temporal gains can be adjusted.
The calculating unit 450 includes a first multiplier 455, a second multiplier 465, a first adder 460, and a second adder 470.
The first multiplier 455 multiplies input image data Data_BI pixel binned by the pixel binning unit 415 by a pixel binning gain Expansion_gain_S determined by the gain determining unit 445.
The first adder 460 calculates a sum SHF (=H_hpf+V_hpf+D_hpf) of high frequency components.
The second multiplier 465 multiplies the sum SHF of the high frequency components by a resolution preserving factor RP_factor determined by the resolution preserving factor determining unit 440.
The second adder 470 adds an output of the first multiplier 455 to an output of the second multiplier 465.
That is, first output image data Data_Out_S 475, which is an output of the calculating unit 450, is given by ‘Data_Out_S=Data_BI*Expansion_gain_S+RP_factor*SHF’.
The first output image data 475 may be input to the temporal expansion unit 480 again.
The temporal expansion unit 810 includes a motion detector 820, a data merger 830, a third multiplier 840, and a third adder 850.
The motion detector 820 detects a motion between current frame data and previous frame data.
The data merger 830 merges current frame data with previous frame data of first output image data 475 at a predetermined ratio based on the motion detected by the motion detector 820.
Referring to
The SAD, which is a sum (in blocks) of differences between current frame data (or the brightness of the current frame data) and previous frame data (or the brightness of the previous frame data), can be used to judge the degree of motion as well. That is, it is judged that the degree of motion increases as the SAD increases, and the degree of motion decreases as the SAD decreases.
In
In short, an output Data_merge of the data merger 830 may be defined by ‘Data_merge=Curr_gain*Data_curr+Prev_gain*Data_prev’.
Referring to
The third multiplier 840 multiplies the data merge gain Expansion_gain_T by the output Data_merge of the data merger 830.
The third adder 850 adds an output of the third multiplier 840 to the current frame data Data_curr.
As a result, the second output image data Data_Out may be defined by ‘Data_Out=Data_curr+Expansion_gain_T*Data_merge’.
In operation 910, input image data is pixel binned to a predetermined pixel size. The pixel binning can preserve the resolution of the input image data.
In operation 920, a pixel binning gain is determined based on the input image data or the brightness of the input image data. Since the pixel binning gain has already been described with reference to
In operation 930, output image data is calculated based on the pixel binned input image data and the pixel binning gain.
In operation 1010, input image data is pixel binned to a predetermined pixel size.
In operation 1020, high frequency components in a plurality of directions of the input image data are filtered. For example, a first high frequency component in a horizontal direction of the input image data, a second high frequency component in a vertical direction of the input image data, and a third high frequency component in a diagonal direction of the input image data may be filtered.
In operation 1030, a resolution preserving factor is determined based on the high frequency components. Since the resolution preserving factor has already been explained with reference to
In operation 1040, output image data is calculated based on the pixel binned input image data, the high frequency components, and the resolution preserving factor.
In operation 1110, input image data is pixel binned to a predetermined pixel size.
In operation 1120, a pixel binning gain is calculated based on the input image data or the brightness of the input image data.
In operation 1130, high frequency components in a plurality of directions of the input image data are filtered.
In operation 1140, a resolution preserving factor is determined based on the high frequency components.
In operation 1150, output image data is calculated based on the pixel binned input image data, the pixel binning gain, the high frequency components, and the resolution preserving factor.
In operation 1160, the dynamic range of the output image data is expanded based on current frame data and previous frame data of the output image data.
The present invention may be embodied as computer readable codes on a computer readable recording medium. The computer readable recording medium is any data storage device that can store data which can be thereafter read by a computer system.
Examples of the computer readable recording medium include read-only memories (ROMs), random-access memories (RAMs), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer readable recording medium can be dispersively installed in a computer system connected to a network, and stored and executed as a computer readable code in a distributed computing environment.
As described above, the imaging apparatus and the method of improving the sensitivity of the imaging apparatus according to the exemplary embodiments of the present invention can expand the dynamic range of an image signal under low illumination conditions while preserving the resolution of input image data.
Furthermore, the imaging apparatus and the method of improving the sensitivity of the imaging apparatus according to the exemplary embodiments of the present invention can increase the signal to noise ratio under low illumination conditions.
While the present invention has been particularly shown and described with reference to the exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2007-0069212 | Jul 2007 | KR | national |
This is a divisional of U.S. patent application Ser. No. 12/020,597, filed Jan. 28, 2008, which claims priority from Korean Patent Application No. 10-2007-0069212, filed on Jul. 10, 2007, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12020597 | Jan 2008 | US |
| Child | 14486645 | US |
