Patent Application
20060268127
This application is based on Japanese Patent Application No. 2005-158931 filed on May 31, 2005, in Japanese Patent Office, the entire content of which is hereby incorporated by reference.
The present invention relates to an image pickup apparatus and more particularly to an image pickup apparatus having two image pickup modes of normal image pickup mode and wide dynamic range image pickup mode and two recording formats of unprocessed image recording format and processed image recording format.
Conventionally, a logarithmic conversion type image pickup device in which a logarithmic conversion circuit having a MOSFET is added to a solid-state image pickup device composed of photoelectric conversion devices such as photodiodes arranged in a matrix shape is known, the logarithmic conversion type image pickup device in which the sub-threshold characteristic of the MOSFET is used, thus an electric signal is logarithmically converted for an amount of incident light by the output characteristic of the solid-state image pickup (for example, refer to Patent Document 1).
Further, a logarithmic conversion type image pickup device in which a specific reset voltage is given to the MOSFET, thus the intrinsic output characteristic of the solid-state image pickup device, that is, a linear characteristic state that an electric signal is linearly converted and outputted according to the amount of incident light and the logarithmic characteristic state aforementioned can be switched automatically is known (for example, refer to Patent Document 2). Furthermore, an image pickup apparatus in which the linear characteristic state can be automatically switched to the logarithmic characteristic state, and the reset time of the MOSFET is adjusted, thus the potential state of the MOSFET can be adjusted is disclosed (for example, refer to Patent Document 3).
On the other hand, a digital camera, which is a typical example of the image pickup apparatus, regarding recording of picked-up images, having a mode for recording processed images (images subject to the image processing) and a mode for recording unprocessed images (images not subject to the image processing) is known (for example, refer to Non-Patent Document 1). Processed images include, for example, images of JPEG, bit map, and TIFF and unprocessed images include RAW images.
Patent Document 1: Japanese Laid-Open Patent Application HEI11-298798
Patent Document 2: Japanese Laid-Open Patent Application 2002-77733
Patent Document 3: Japanese Laid-Open Patent Application 2002-300476
Non-Patent Document 1: α-7DIGITAL Catalog, Konica Minolta Photo-Imaging Co., Ltd.
(http://konicaminolta.jp/products/consumer/digital_came ra/catalogue/pdf/a-7digital.pdf) page 9/9, Column of “Recording Format” of “Recording Section”, Retrieved on May 10, 2005
However, when the image pickup devices disclosed in Patent Documents 1 to 3 are used in the linear characteristic state, output in proportion to the electric charge generated in the photoelectric conversion device is obtained, so that there is an advantage that even a photographic subject at low brightness can obtain an image signal having a high contrast (a high gradation), while there is a disadvantage that the dynamic range is narrow. On the other hand, when the image pickup devices disclosed in Patent Documents 1 to 3 are used in the logarithmic characteristic state, output converted natural-logarithmically for the amount of incident light is obtained, so that there is an advantage that a wide dynamic range can be ensured, while an image signal is compressed logarithmically, so that there is a disadvantage that the contrast is low.
Further, when the image pickup devices disclosed in Patent Documents 1 to 3 are used in the state of coexistence of the linear characteristic state and logarithmic characteristic state, the wide dynamic range can be ensured, though there is a disadvantage that the contrast of the part in the logarithmic characteristic state is low.
Furthermore, depending on recording of those images as processed images or recording thereof as unprocessed images, the contents of image processing differ greatly.
The present invention was developed in view of the foregoing and is intended to provide an image pickup apparatus, when picking up an image using an image pickup device having a photoelectric conversion characteristic composed of a linear characteristic region and a logarithmic characteristic region, for performing an appropriate image process according to the image pickup mode and recording mode, thereby always obtaining optimum images. In view of forgoing, one embodiment of an image pickup apparatus according to one aspect of the present invention is an image pickup apparatus, comprising:
an image pickup device which takes an image of a photographic subject and outputs an electric signal corresponding to the image, the image pickup device being capable of operating a normal image pickup mode and a wide dynamic range image pickup mode with a wider dynamic range than the normal image pickup mode;
an image processing section which processes the electric signal outputted from the image pickup device, the image processing section being capable of a normal image processing as a versatile image processing and a wide dynamic range image processing as an image processing specific for images picked up in the wide dynamic range image pickup mode;
an image recording section which includes an image recording media, the image recording section for recording an image signal processed by the image processing section in the image recording media;
an recording format selection section for selecting one mode from an unprocessed image recording format and a processed image recording format as a recording format in the image recording section;
an image pickup mode selection section for selecting one mode from the normal image pickup mode or the wide dynamic range image pickup mode; and
an image pickup control section which controls a process executed in the image processing section based on the image pickup mode selected by the mode selection section and the recording format selected by the recording format selection section.
According to another aspect of the present invention, another embodiment of an image pickup apparatus is an image pickup apparatus, comprising:
an image pickup device which takes an image of a photographic subject and outputs an electric signal corresponding to the image, the image pickup device being capable of operating a normal image pickup mode and a wide dynamic range image pickup mode with a wider dynamic range than the normal image pickup mode;
an image processing section which processes the electric signal outputted from the image pickup device, the image processing section being capable of a normal image processing as a versatile image processing and a wide dynamic range image processing as an image processing specific for images picked up in the wide dynamic range image pickup mode;
an image recording section which includes an image recording media, the image recording section for recording an image signal processed by the image processing section in the image recording media;
an recording format selection section for selecting one mode from an unprocessed image recording format and a processed image recording format as a recording format in the image recording section;
an image pickup mode selection section for selecting one mode from the normal image pickup mode or the wide dynamic range image pickup mode; and
an image pickup control section which controls a process executed in the image processing section based on the image pickup mode selected by the mode selection section and the recording format selected by the recording format selection section,
wherein the image pickup control section makes the image processing section execute the normal image processing when the image pickup mode selection section selects the normal image processing mode, and the image recording format selection section selects the processed image recording format, and the image pickup control section makes the image processing section execute the normal image processing and the wide dynamic range image processing when the image pickup mode selection section selects the wide dynamic range image pickup mode, and the recording format selection section selects the processed image recording format.
Hereinafter, the embodiment of the present invention will be explained with reference to the accompanying drawings.
An exchange lens 20 is attached onto the front of a body 10.
On the top of the body 10, a release button 101 which is an operation member for picking up images is installed and on the lower part of the release button 101 inside the body 10, a two-step switch composing an AF/AE switch 101a operating at the first step of pressing of the release button 101 and a release switch 101b operating at the second step of pressing of the release button is arranged. Further, on the top of the body 10, a flash light 102 is built in and a mode setting dial 112 for setting the operation mode of the digital camera 1 is arranged.
On the back of the body 10, a power switch 111 for turning on or off the power source for the digital camera 1, a jog dial 115 composed of five switches upward, downward, left, right, and central for executing respective settings in the respective modes of the digital camera 1, a finder eye lens 121a, and an image monitor 131 for displaying recorded images are arranged.
A cameral control unit 150 which is a control unit of the digital camera 1 is composed of a central processing unit (CPU) 151, a work memory 152, and a storage unit 152, and reads a program stored in the storage unit 153 into the work memory 152, and intensively controls each unit of the digital camera 1 according to the concerned program.
Further, the cameral control unit 150 receives input signals from the power switch 111, mode setting dial 112, jog dial 115, AF/AE switch 101a, and release switch 101b or the like, communicates with a beam measurement module 121b on an optical finder 121, thereby controls a beam measurement operation, communicates with an AF module 144, thereby controls a focusing operation, drives a reflex mirror 141 and a sub-mirror 142 via a mirror drive unit 143, controls a shutter 145 via a shutter drive unit 146, controls a flash light 102 via a flash control unit 147, communicates with an image pickup control section 161, thereby controls an image pickup operation, displays picked-up image data and various information on an image monitor 131, and displays various information on an in-finder display unit 132.
Furthermore, the camera control unit 150 communicates with a BL communication unit (on the body side) 172 installed on a mount (on the body side) 171 for functioning as a communication unit between the body 10 and the exchange lens 20, a lens control unit 241 for controlling the focus and zoom of a lens 211 via a BL communication unit (on the lens side) installed on a mount (on the lens side) 271 and via a lens interface 251 of the exchange lens 20, a stop control unit 222 for controlling a stop 221, and a lens information storage unit 231 for storing intrinsic information of the exchange lens 20, thereby controls the whole exchange lens 20.
An image focused by the lens 211 of the exchange lens 20 is photoelectrically converted by an image pickup device 162, then is amplified by an amplifier 163, is converted to digital data by an analog-digital (A-D) converter 164, is converted to digital image data subject to a predetermined image process by an image processing section 165, is stored once in an image memory 181, and then is finally stored in a memory card 182.
The image pickup device 162 has a photoelectric conversion characteristic composed of a linear characteristic region and a logarithmic characteristic region and can freely set a range of both characteristic regions (dynamic range). Switching of both characteristics and the range of both characteristic regions (dynamic range) are controlled by the image pickup control section 161. It will be described later in detail.
A series of image pickup operations aforementioned is controlled by the image pickup control section 161 under the control of the camera control unit 150. The cameral control unit 150, image pickup control section 161, image memory 181, and memory card 182 function as an image recording section of the embodiment of the present invention. Further, the camera control unit 150, mode setting dial 112, and jog dial 115 function as a recording format selection section for selecting unprocessed image recording format and processed image recording format of the image recording section of the embodiment of the present invention and also function as an image pickup mode selection section for selecting the normal image pickup mode and wide dynamic range image pickup mode.
Further, digital image data stored in the memory card 182 is read into a personal computer (PC) or a portable information terminal (PDA) which are not drawn, thereby can be subject a further image process by the software of the PC or PDA. By doing this, an image process having a higher performance can be performed by the PC and the like having a more powerful processing capacity than the CPU 151 loaded in the digital camera 1 and a higher image quality can be realized.
Next, the relationship between the image pickup mode and recording mode and the image process of the digital camera 1 which is an example of the image pickup apparatus relating to the embodiment of the present invention will be explained by referring to FIGS. 3 to 7. FIGS. 3 to 7 are flow charts showing the flow of the image pickup operation of the embodiment of the present invention, and
Further, in this example, the image pickup mode means the normal image pickup mode and wide dynamic range image pickup mode, and the normal image pickup mode means a mode for picking up images only by the linear characteristic of the image pickup device 162, and the wide dynamic range image pickup mode means a mode for picking up images using both linear characteristic and logarithmic characteristic of the image pickup device or using only the logarithmic characteristic.
Further, in this example, the recording mode means the unprocessed image recording mode and processed image recording mode and the unprocessed image recording mode means a mode for recording images not subject to the image process (in this example, it will be explained as RAW image recording), in which images generally are not often subject to image compression but may be subject to image compression such that RAW image data is subject to a reversible compression process. The processed image recording mode means a mode for recording images subject to the image process (in this example, it will be explained as JPEG image recording). Generally, images are often subject to image compression, though images may not be subject to image compression such as bit map.
Further, in this example, the image process includes a predetermined image process and a wide dynamic range image processing.
In
At Step S121, it is confirmed whether the image pickup mode in the camera mode of the digital camera 1 is set at the wide dynamic range image pickup mode or not. When it is set at the wide dynamic range image pickup mode (YES at Step S121), the process goes to Step S131.
At Step S131, it is confirmed whether execution of unprocessed image recording in the wide dynamic range image pickup mode is set or not. When the execution of unprocessed image recording is set (YES at Step S131), the process goes to S132, and the subroutine “wide D image pickup/unprocessed recording mode” shown in
At Step S131, when the execution of unprocessed image recording is not set (NO at Step S131), the process goes to S141, and the subroutine “wide D image pickup/processed recording mode” shown in
When the image pickup mode is not set at the wide dynamic range image pickup mode at Step S121 (NO at Step S121), it is confirmed at Step S151 whether it is set at the normal image pickup mode or not. When it is set at the normal image pickup mode (YES at Step S151), the process goes to Step S161. When it is set at any image pickup mode (for example, the continuous image pickup mode, etc.) other than the normal image pickup mode (NO at Step S151), the process moves to the control in the mode corresponding to the setting. The explanation thereof will be omitted.
It is confirmed at Step S161 whether the execution of unprocessed image recording is set in the normal image pickup mode or not. When the execution of unprocessed image recording is set (YES at Step S161), the process goes to S162, and the subroutine “normal image pickup/unprocessed recording mode” shown in
When the execution of unprocessed image recording is not set at Step S161 (NO at Step S161), the process goes to S171, and the subroutine “normal image pickup/processed recording mode” shown in
At Step S301, the release button 101 is operated and it is confirmed whether the AF/AE switch 101a is turned on or not. Until it is turned on, the process waits at Step S301. When it is turned on (YES at Step S301), the AF operation is performed at Step S311 and focusing is executed. At Step S312, beam measurement is executed by the beam measurement module 121b and at Step S313, from the beam measurement result, the exposure, that is, the stop value of the stop 221 of the exchange lens 20 and the speed of the shutter 145 are set.
At Step S314, the dynamic range is set in accordance with the exposure set at Step S313. At Step S321, the release button 101 is operated and it is confirmed whether the release switch 101b is turned on or not. Until it is turned on, the flow of operations at Step S301 to Step S321 is repeated. When it is turned on (YES at Step S321), at Step S322, under the condition of the stop value and shutter speed which are set at Step S313 and the dynamic range set at Step S314, the wide dynamic range image pickup is executed, and at Step S323, the picked-up unprocessed image (here, RAW data) and ancillary information of the image (a flag indicating image pickup in the “wide D image pickup/unprocessed recording mode”, etc.) are recorded once in the image memory 181, thereafter at Step S324, the unprocessed image recorded in the image memory 181 and the ancillary information thereof are recorded in the memory card 182, and the process returns to the main routine.
In the “wide D image pickup/processed recording mode”, at Step S411, for the wide dynamic range image picked up at Step S322, the heightening contrast processing which is an example of the wide dynamic range image processing of the embodiment of the present invention is executed (it will be described later in detail), and at Step S412, a series of image processes which will be described later and image compression (here, JPEG compression) are executed, and at Step S423, the processed image processed at Step S412 and ancillary information of the image (a flag indicating image pickup in the “wide D image pickup/processed recording mode”, etc.) are recorded once in the image memory 181, and at Step S424, the processed image recorded in the image memory 181 and the ancillary information thereof are recorded in the memory card 182, and the process returns to the main routine.
In the “normal image pickup/unprocessed recording mode”, at Step S522, under the condition of the stop value and shutter speed which are set at Step S313 and the dynamic range set at Step S314, the normal image pickup, that is, the image pickup of operating the image pickup device by the linear characteristic is executed.
In the “normal image pickup/processed recording mode”, at Step S522, under the condition of the stop value and shutter speed which are set at Step S313 and the dynamic range set at Step S314, the normal image pickup, that is, the image pickup of operating the image pickup device 162 by the linear characteristic is executed.
As mentioned above, an image pickup apparatus for appropriately selecting and applying two kinds of image processes of a predetermined image process and wide dynamic range image processing in accordance with the image pickup mode and recording mode of the image pickup apparatus, thereby always obtaining optimum images can be provided.
Next, an example of the image pickup device having a photoelectric conversion characteristic composed of a linear characteristic region and a logarithmic characteristic region of the embodiment of the present invention will be explained by referring to FIGS. 8 to 10.
The pixels 162a are composed of a photodiode PD, transistors T1 to T6 as a MOSFET (metal oxide semiconductor field effect transistor), and a capacitor C as an integral capacitor. The transistors T1 to T6 use a P-channel MOSFET in this example. φVD, φV, φVPS, φRST, φS, and RSB indicate signals (voltages) for the transistors and capacitor C and GND indicates grounding.
The photodiode PD is a photoelectric conversion unit and outputs an optical current IPD according to an amount of incident light from a photographic subject.
The transistor T1 is a switch used to take out a pixel variation signal indicating an error component between pixels caused by manufacturing variations of the transistor T2 and is generally kept on so that the optical current IPD flows between the transistor T2 and the photodiode PD. When taking out the pixel variation signal, the transistor T1 is turned off, and the optical current IPD of the photodiode PD is interrupted, and only the pixel variation signal is taken out.
The transistor T2 is connected to a gate and a drain and operates so as to generate a voltage for applying a linear conversion or a logarithmic conversion to the optical current IPD in the gate thereof using a sub-threshold characteristic (a characteristic for flowing a minute current called a sub-threshold current when the gate voltage is not higher than the threshold value) of the MOSFET.
Concretely, when a photographic subject to be picked up is dark, that is, when the amount of incident light entering the photodiode PD is small, since the gate potential of the transistor T2 is higher than the source potential of the concerned transistor, the transistor T2 is in the so-called cutoff state, and the sub-threshold current does not flow in the transistor T2, and the optical current IPD generated by the photodiode PD flows in a parasitic capacity CPD of the photodiode PD, thus the optical current IPD is integrated, and a voltage according to the integral electric charge is generated.
At this time, the transistor T1 is turned on, so that a voltage according to the optical current IPD integrated by the parasitic capacity CPD is generated in the gates of the transistors T2 and T3 as a voltage VG. Due to the voltage VG, a current flows through the transistor T3 and an electric charge in proportion to the voltage VG is accumulated in the capacitor C (the transistorT3 and capacitor C compose an integral circuit). And, at a connection node a between the transistor T3 and the capacitor C, that is, output VOUT, a voltage linearly proportional to the integral value of the optical current IPD appears. This is an operation in the linear characteristic region.
On the other hand, when the photographic subject to be picked up is bright and the amount of incident light entering the photodiode PD is large, the gate potential of the transistor T2 is not higher than the source potential of the concerned transistor, and the transistor T2 is operated in the sub-threshold area, so that the sub-threshold current flows in the transistor T2, and a voltage VG for converting natural-logarithmically the optical current IPD is generated in the gates of the transistors T2 and T3. And, due to the voltage VG, a current flows through the transistor T3 and an electric charge equivalent to a value obtained by natural-logarithmically converting the integral value of the optical current IPD is accumulated in the capacitor C. By doing this, at the connection node a (output VOUT) between the transistor T3 and the capacitor C, a voltage proportional to the value obtained by natural-logarithmically converting the integral value of the optical current IPD is generated. This is an operation of the concerned image pickup device 162 in the logarithmic characteristic region.
As mentioned above, whether the operation of the pixels 162 is in the linear characteristic region or the logarithmic characteristic region depends on the relation of the magnitude between the gate potential VG of the transistor T2 and the source potential φVPS thereof. The gate potential VG depends on the optical current IPD as mentioned above, so that by controlling the source potential φVPS, the switching point (inflection point) between the linear characteristic and the logarithmic characteristic can be controlled.
As mentioned above, according to the brightness of the photographic subject, that is, the amount of incident light, for each pixel, a voltage linearly or natural-logarithmically proportional to the integral value of the optical current IPD is generated.
The transistor T4 is a transistor for resetting the capacitor C and operates as a switch to be turned on or off according to the φRST impressed to the gate of the transistor T4. When the transistor T4 is turned on, a reset voltage RSB is impressed to the capacitor C and the accumulated charge is returned to the state before starting the integration.
The transistor T5 composes a source follower amplifier circuit, executes current amplification for the output VOUT aforementioned, and functions so as to lower the output impedance.
The transistor T6 is a transistor for reading a signal and operates as a switch to be turned on or off according to the voltage φV impressed to the gate. The source gate of the transistor T6 is connected to the vertical signal line 162g and when the transistor T6 is turned on, derives the photoelectric conversion output VP which is amplified in current by the transistor T5 and is reduced in impedance to the vertical signal line 162g.
Next, the constitution of the image processing section 165 in the circuit block diagram shown in
As mentioned above, the image signal converted photoelectrically by the image pickup device 162 is amplified by the amplifier 163, is converted to digital data by the analog-digital (A-D) converter 164, is converted to digital image data subject to a predetermined image process by the image processing section 165, is recorded once in the image memory 181, and then is recorded finally in the memory card 182. Digital data 501 inputted to the image processing section 165 is input to a white balance processing unit 502, and WB output 502a subject to a white balance process is input directly to a switching unit 504 and also input to a heightening contrast processing section 503, and HC output 503a subject a heightening contrast processing is also input to the switching unit 504.
The switching unit 504, under the control of the image pickup control section 161, switches selectively either of the WB output 502a and HC output 503a and inputs it to a color interpolation/color correction unit 505. The image signal subject to color interpolation/color correction is γ-corrected by a γ correction unit 506, is converted from RGB data to YCrCb data by a color space conversion unit 507, is image-compressed by a JPEG compression unit 508, and is input to an output unit 509 as JPEG compressed data 508a. On the other hand, the digital data 501 inputted to the image processing section 165 is directly input to the output unit 509 as RAW data, and either of the JPEG compressed data 508a and RAW data 501 is selectively switched by the output unit 509 under the control of the image pickup control section 161, and it is output to the image pickup control section 161 as digital image data 510.
Next, the operation of each unit of the image processing section 165 shown in
The white balance process in such a case will be explained, for example, by referring to
Therefore, in this example, firstly, the photoelectric conversion characteristic 513b of the logarithmic characteristic of red in the area 512b is calculated and converted to a linear characteristic 513d and is connected to the linear characteristic 513a of the photographic subject brightness area 512a (513a+513d). In an actual conversion, a lookup table (LUT) and the like is used to shorten the time necessary for calculation. Thereafter, the linear characteristic with the photographic subject brightness areas 512a and 512b connected to each other is multiplied by a correction value, thus the photoelectric conversion characteristic (513a+513d) of the linear characteristic region is made coincide with the photoelectric conversion characteristic 514a of the reference color of green. The logarithmic characteristic 513c of the photographic subject brightness area 512c is added with a correction value (a negative correction value is added in this example) and is made coincide with the photoelectric conversion characteristic 514c of the reference color of green. By doing this, the photoelectric conversion characteristic of red is made coincide with the photoelectric conversion characteristic of green.
Further, in the example shown in
As mentioned above, the white balance processing unit 502 calculates the photoelectric conversion characteristic of the color to be corrected to data of the same characteristic in accordance with the linear characteristic region and logarithmic characteristic region of the photoelectric conversion characteristic of the reference color and multiplies or adds for each of the linear characteristic region and logarithmic region so as to make the photoelectric conversion characteristics coincide with each other.
Next, the heightening contrast processing (gradation conversion process) by the heightening contrast processing section 503 shown in
The color element division unit 531 divides image data from the image pickup device 162, here, the WB output 502a (linear/logarithmic image) from the preceding white balance processing unit 502 into image data for each four color elements (R, Gr, Gb, B) in the G checkered RB line sequential arrangement having a Bayer system color filter structure, that is, obtains four kinds of color image data (R image, Gr image, Gb image, and B image) derived by dividing the four Bayer elements into each elements. Further, the image size of each color image is ½ of the original image size. Further, the four kinds of color images are respectively linear/logarithmic images including linear characteristic and logarithmic characteristic information.
The region division unit 532, from each of the four kinds of color images input from the color element division unit 531 (each color image is expressed as a basic image I), divides and extracts images in the logarithmic characteristic region (assumed as an image I1) and images in the linear characteristic region (an image I2).
The basic image I for each color image input from the color element division unit 531 to the region division unit 532 has, for example, a photoelectric conversion characteristic 550 shown in
y=α×x+b (where 0≦x≦Xth) (1-1)
y=α×log(x)+β(where Xth≦x) (1-2)
In the region division unit 532, as shown in Conditional Expressions (2-1) to (2-4) indicated below, the pixels composing the basic image I (here, to indicate a two-dimensional image, may be properly expressed as image I (x, y)) are divided into an region where the photoelectric conversion output is a predetermined value θ or larger and an region where it is smaller than θ (the basic image I is clipped (so-called clipping) by θ at the upper limit position and lower limit position of each characteristic region). The “θ” is properly referred to as a division parameter.
if (I(x, y)≧θ)
then
I1(x,y)=I(x,y) (2-1)
I2(x,y)=0(zero) (2-2)
else
I1(x,y)=0(zero) (2-3)
I2(x,y)=I(x,y) (2-4)
The conditional expressions, in short, indicate that in the image I (x, y), the image in the region where the photoelectric conversion output is θ or larger is the image I1 (image I1 (x, y)) and the image in the region where the photoelectric conversion output is smaller than θ is the image I2 (image I2 (x, y)).
However, in this example, as shown in
As mentioned above, the region division unit 532, when a basic image I having the photoelectric conversion characteristic 550 is input, from the concerned basic image I, performs the division extraction process of the image I1 (logarithmic characteristic image) shown in an region 554 and the image I2 (liner characteristic image) shown in an region 555 with the position of the division parameter θ (=Yth, the inflection point) bounded by. The setting information of the boundary parameter θ may be stored in the heightening contrast processing section 503 (for example, the region division unit 532).
On the other hand, the basic image I, according to the so-called Retinex theory, assuming the illumination component of the basic image I as L and the reflection factor component as R, is expressed by Formula (3-1) indicated below.
I=L×R (3-1)
However, Formula (3-1) is a formula for the basic image I as a linear characteristic region image and the basic image I as a logarithmic characteristic region image is expressed by Formula (4-1) indicated below.
Log(I)=Log(L)+Log(R) (4-1)
As shown in
I2=L1×R2 (3-2)
I1=Log(L1)+Log(R1) (4-2)
Generally, the illumination component is often lower in the frequency than the reflection factor component, so that the illumination component is approximated to the low frequency component of an image. A low frequency component extraction filter (low-pass filter) is assumed as F. F is a Gaussian filter, a median filter, an ε-filter which is an edge maintaining type low-pass filter, or a bilateral filter. Formula (4-2), when F is applied, since the high frequency component Log (R1) is removed, is changed to:
Log(L1)=F(I1) (5-1),
thus, the light can be obtained.
The first illumination component compression unit 534 performs the compression process for the illumination component image extracted by the first illumination component extraction unit 533. Namely, the first illumination component compression unit 534 performs a predetermined compression process for the illumination component Log (L1) extracted and outputs it as a logarithmic value Log (L1′) of a illumination component L1′ obtained by compressing the illumination component L1. Assuming the compressibility for heightening contrast realization processing (dynamic range compressibility) as c and the parameter for controlling the reproduction dynamic range as s, the Log (L1′) outputted from the first illumination component compression unit 534 is indicated by the following Formula (6-1).
Log (L1′) Log (L1)×c+Log (s) (6-1)
Assuming the image after heightening contrast realization processing for the image I1 as image I1′ and the reflection factor component of the image I1 as R1, Formula (4-1) is expressed by Formula (7-1) indicated below.
Log(I1′)=Log(L1′)+Log(R1) (7-1)
Therefore, the image I1′ is expressed by the following Formula (8-1) obtained by taking an inverse logarithm for both sides of Formula (7-1).
I1′=exp(Log(L1′)+Log(R1))=L1′×R1 (8-1)
The linear conversion unit 535 converts the logarithmic image which is divided and extracted by the region division unit 532 and is compressed by the first illumination component extraction unit 533 and the first illumination component compression unit 534 to a linear image. Concretely, the linear conversion unit 535 performs calculations by conversion of Formula (7-1) to Formula (8-1), thereby converts the logarithmic image Log (I1′) to the linear image I1′. When the logarithmic image Log (I1′) is converted to the linear image I1′ by the linear conversion unit 535 like this, it can be handled as an image having the same characteristic (linear characteristic) as that of the image I2′ which is a succeeding linear image (here, a composition process of the images I1′ and I2′ can be performed).
Further, as shown in
Among the images I1 and I2 extracted by the region division unit 532, the image I2 is subject to heightening contrast processing by the method indicated below by the second illumination component extraction unit 536 and second illumination component compression unit 537 and is input to the image composition unit 538 as an image I2′. It will be explained below.
The second illumination component extraction unit 536 extracts the illumination component L2 from the image L2 divided and extracted by the region division unit 532. The extraction process of the illumination component L2 from the concerned image L2 is expressed by Formula (9-1) indicated below.
L2=F(I2) (9-1)
F in Formula (9-1), similarly to the aforementioned, indicates a filter such as a Gaussian filter. On the other hand, the reflection factor component R2 of the image I2 is obtained from the relationship of R2=I2/L2 (refer to Formula (3-2)).
The second illumination component compression unit 537 performs a predetermined compression process for the illumination component L2 obtained by the second illumination component extraction unit 536 and outputs a illumination component L2′ obtained by compressing the concerned illumination component. Assuming the dynamic range compressibility as c and the parameter for controlling the reproduction dynamic range as s, the compression illumination component L2′ is given by Formula (10-1) indicated below.
L2′=exp (Log (L2)×c+Log (s)) (10-1)
The compression illumination component L2′ obtained by the second illumination component compression unit 537 is multiplied by the reflection factor component R2 by a multiplication unit 544 and as a result, the image I2′ after the heightening contrast processing for the image I2 is obtained. The reflection factor component R2 is obtained by dividing the illumination component L2 transmitted through a route F by the image I2 transmitted through a route E by a division unit 543.
The image composition unit 538 prepares a composite image of the linear image and logarithmic image after the heightening contrast processing aforementioned. Namely, the image composition unit 538 prepares a composite image O on the basis of the image I1′ obtained by the heightening contrast processing for the image I1 and the image I2′ obtained by the heightening contrast processing for the image I2 (O=I1′+I2′). The image I1′ (logarithmic characteristic image) and the image I2′ (linear characteristic image) are smoothly connected (composed) as indicated by a photoelectric conversion characteristic 562 (O) shown in
The color element composition unit 539 composes the image O obtained by the image composition unit 538, that is, four kinds of images O corresponding to the respective color image I (R image, Gr image, Gb image, and B image aforementioned) for each element of the four Bayer elements aforementioned and obtains the HC output 503a (linear image) having the image information of the original four elements. Further, the image size is returned from the ½ size of each image O to the HC output 503a of the equimultiple size. Further, the HC output 503a is an image including only the linear characteristic information (unified to image data of the linear characteristic).
The image pickup device having the linear characteristic and logarithmic characteristic is explained above, though in the case of an image pickup device having the logarithmic characteristic, it is desirable to perform only the process of image I1 (logarithmic character image) aforementioned. Further, various methods for obtaining a wide dynamic range image of the linear characteristic have been proposed, though for those images, it is desirable to perform only the process of I2 (linear characteristic image) aforementioned.
As mentioned above, when the heightening contrast processing (gradation conversion process) by the heightening contrast processing section 503 is performed, images having different photoelectric conversion characteristics can be converted to images having the same photoelectric conversion characteristic and contrast emphasis (contrast improvement) of a low-contrast part of a picked-up image can be realized. Namely, a process (process by the linear conversion unit 535) of extracting the base (illumination component) for each local space (the linear characteristic region and logarithmic characteristic region) of an image, compressing the concerned extracted base, and converting it to the same photoelectric conversion characteristic as that of the linear characteristic region together with the reflection factor component of the image is performed, thus the image data composed of the logarithmic character and linear character can be unified to image data of the linear characteristic, and in addition to it, contrast emphasis (improvement) in the logarithmic characteristic region can be realized.
In any way, by the heightening contrast processing by the heightening contrast processing section 503, picked-up image data is unified to the photoelectric conversion characteristic in the low-brightness area, thus all the image data can be processed as image data having the same characteristic, and the processing of data in the subsequent image process is simplified (the calculation can be simplified or the processing can be speeded up). Here, the subsequent processes by the color interpolation/color correction unit 505, γ correction unit 506, and color space conversion unit 507 can be performed efficiently using the conventional color processing method as it is. Next, the functions of the switching unit 504 of the image processing section 165 and subsequent units will be explained by referring to
The switching unit 504, according to the image pickup mode of the digital camera 1 explained in
The processes by the color interpolation/color correction unit 505, γ correction unit 506, and color space conversion unit 507 following the switching unit 504, as mentioned above, can use the conventional color processing method as it is, so that the explanation thereof will be omitted here. Further, for the JPEG compression unit 508 which is an example of image compression, since both WB output 502a and HC output 503a are image data of the linear characteristic, the known JPEG compression method can be used, so that the explanation thereof will be omitted.
In the output unit 509, the processes by the color interpolation/color correction unit 505, γ correction unit 506, and color space conversion unit 507 are performed for the aforementioned WB output 502a or HC output 503a and the processed image data 508a subject to JPEG compression and the unprocessed image data (RAW data) input to the image processing section 165 are input to it. From the output unit 509, according to an output control signal 161b from the image pickup control section 161 and in accordance with the image pickup mode and recording mode of the image pickup apparatus, either of the processed image data 508a and unprocessed image data (RAW data) 501 is output toward the image pickup control section 161 as digital image data 510, is stored once in the image memory 181 by the image pickup control section, and, then is finally stored in the memory card 182.
In the above explanation, the switching unit 504 switches and outputs the WB output 502a and HC output 503a according to the image pickup mode. A method for practically switching the WB output 502a and HC output 503a according to the image pickup mode without using the switching unit 504 will be explained below.
When the image pickup mode is the normal image pickup (image pickup in the linear characteristic) mode and the maximum value of the input range of the WB output 502a which is an input signal of the heightening contrast processing section 503 is set to a photoelectric conversion output value Yth at the switching point between the linear characteristic and the logarithmic characteristic, use of only the lower half (the processing series of the linear image I2) shown in
When the coefficient of the filter F (I2) of the second illumination component extraction unit 536 is set to 1 only for its own pixel and 0 (zero) for the others, from Formula (9-1), L2=I2 is held and the reflection factor component R2=I2/L2=I2/I2=1 is held.
Therefore, when c=1 is set in Formula (10-1), the output of the second illumination component compression unit 537 L2′=L2=I2 is held. Therefore, the composite image O==I2′=R2×L2′=1×I2=I2=I is held and the composite image O becomes equal to the input image I, that is, the process by the heightening contrast processing section 503 is not acted practically (made invalid).
When the process parameter of the heightening contrast processing section 503 is appropriately controlled in the normal image pickup (image pickup in the linear characteristic) mode like this, the switching unit 504 can be made unnecessary and it can contribute to realization of miniaturization and low cost of the image processing section 165. The circuit block diagram of the image processing section 165 when the switching unit 504 is not used is shown in
Furthermore, an image after the image process which is recorded in the image memory 181 or memory card 182 can be confirmed by the image monitor 131 loaded on the digital camera 1 or a monitor such as a PC or a PDA reading an image in the digital camera 1 and memory card 182. Further, it may be considered to confirm by printing it by a printer.
As a result of confirmation of the image, depending on the image pickup scene and condition, a case that the compressibility of the dynamic range or the dynamic range reproduction range is required to be changed or a case that although an image is picked up in the wide dynamic range image pickup mode, an image picked up in the normal image pickup mode is desirable occurs. According to the embodiment of the present invention, in such a case, when the parameter of the heightening contrast processing section 503 aforementioned is reset, a desired image can be obtained without picking up an image again. Hereinafter, one example of the method will be indicated by referring to
The dynamic range and contrast are affected greatly by the compressing method and compressibility of the illumination component and images are varied greatly with the setting thereof. The aforementioned process by the heightening contrast processing section 503, after all from Formulas (6-1) and (10-1), is equivalent to execution of the dynamic range compression given by Formula (11-1) indicated below for the illumination component L linearly converted.
L′=s×Lc (11-1)
Here, assuming the reproduction dynamic range capable of reproducing monitor and print as M and the reproduction dynamic range to be reproduced for monitor and print in an image as SI, the parameter s for controlling the reproduction dynamic range is expressed by Formula (11-2) indicated below from Formula (11-1) and Formula (11-1) is rewritten to Formula (11-3).
s=M/SIc (11-2)
L′=M×(L/SI)c (11-3)
When M=SI and c=1 are set in Formula (11-3), L601 shown in
Further, when c≦1 is held, L602 shown in
Furthermore, when the reproduction dynamic range S1 is enlarged overall the picked-up image (SI=S),
When SI=S and c=1 in Formula (11-3), L604 shown in
As mentioned above, when the parameter s for controlling the reproduction dynamic range and the dynamic range compressibility c are reset again for an image which is picked up and is recorded in the image memory 181 or the memory car 182, an image having a changed reproduction dynamic range, a changed contrast, and a different appearance can be obtained freely without picking up an image again.
According to the embodiment of the present invention, an image pickup apparatus for performing selectively two image processes of a predetermined image process and a wide dynamic range image processing in accordance with two image pickup modes of normal image pickup and wide dynamic range image pickup and two recording formats of unprocessed image recording format and processed image recording format, thereby always obtaining optimum images can be provided.
According to another side of the embodiment of the present invention, an image pickup apparatus, when the normal image pickup mode and processed image recording format are selected, for performing the normal image processing by the image processing section, thereby always obtaining optimum images can be provided.
According to still another side of the embodiment of the present invention, an image pickup apparatus, when the wide dynamic range image pickup mode and processed image recording format are selected, for performing the normal image processing and wide dynamic range image processing by the image processing section, thereby always obtaining optimum images can be provided.
According to a further side of the embodiment of the present invention, an image pickup apparatus for using an image pickup device having a linear characteristic region and a logarithmic characteristic region for controlling the range of the linear characteristic region and logarithmic characteristic region and controlling so as to have only the linear characteristic to select an normal image pickup mode and the linear characteristic region and logarithmic characteristic region to select a wide dynamic range image pickup mode, thereby always obtaining optimum images can be provided.
According to a still further side of the embodiment of the present invention, an image pickup apparatus for using a process of heightening contrast for an image picked up in a wide dynamic range image pickup mode as a wide dynamic range image processing, thereby always obtaining optimum images can be provided.
According to yet a further side of the embodiment of the present invention, an image pickup apparatus for using a process of converting an image picked up in a wide dynamic range image pickup mode to an image picked up in an normal image pickup mode as a wide dynamic range image processing, thereby always obtaining optimum images can be provided.
According to yet a further side of the embodiment of the present invention, an image pickup apparatus for switching an image processing path for performing a wide dynamic range image processing and an image processing path for not performing the wide dynamic range image processing in accordance with an image pickup mode, thereby always obtaining optimum images can be provided.
According to yet a further side of the embodiment of the present invention, an image pickup apparatus for setting the process parameter of a wide dynamic range image processing so as to make the process invalid in an normal image pickup mode, thereby always obtaining optimum images can be provided.
According to yet a further side of the embodiment of the present invention, an image pickup apparatus for compressing data as an normal image processing, thereby always obtaining optimum images can be provided.
According to yet a further side of the embodiment of the present invention, an image pickup apparatus, when unprocessed image recording format is selected, for performing neither an normal image processing nor a wide dynamic range image processing, thereby always obtaining optimum images can be provided.
As describe above, according to the embodiment of the present invention, an image pickup apparatus, when picking up an image using an image pickup device having a photoelectric conversion characteristic composed of a linear characteristic region and a logarithmic characteristic region, for performing an appropriate image process according to an image pickup mode and a recording mode, thereby always obtaining optimum images can be provided.
Further, while the present invention has been described by reference to specific embodiments, it should be understood that modifications and variations of the invention may be constructed without departing from the scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2005-158931 | May 2005 | JP | national |
