Patent Grant
10911731
The contents of the following Japanese and International patent applications are incorporated herein by reference:
The present invention relates to an image-capturing device.
An image-capturing system that allocates the visible three primary colors (RGB) to invisible three wavelength bands having mutually different central wavelengths, respectively, has been known.
[Patent Document 1] WO2007/083437
Because conversion from an invisible wavelength band into a visible wavelength band has been performed arbitrarily on a device-by-device basis, the color gamut in the visible color space could not be utilized sufficiently.
One aspect of the present invention provides an image-capturing device including: an image-capturing unit that is photosensitive to light in an invisible band; a generating unit that generates invisible wavelength information defined based on a sensitivity characteristic, which is an output characteristic of the image-capturing unit, to an object light flux wavelength for conversion of image-capturing data generated from an output signal of the image-capturing unit into visible color space image data; and a processing unit that relates the invisible wavelength information to the image-capturing data.
The summary clause does not necessarily describe all necessary features of the embodiments of the present invention. The present invention may also be a sub-combination of the features described above.
Hereinafter, (some) embodiment(s) of the present invention will be described. The embodiment(s) do(es) not limit the invention according to the claims, and all the combinations of the features described in the embodiment(s) are not necessarily essential to means provided by aspects of the invention.
As illustrated in the figure, the direction parallel with the optical axis 21 that points to the image sensor 100 is determined as the Z-axis positive direction, the direction that points into the paper surface on a plane orthogonal to the Z-axis is determined as the X-axis positive direction, and the direction that points upward on the paper surface is defined as the Y-axis positive direction. Regarding a relationship with the composition in photographing, the X-axis becomes the horizontal direction, and the Y-axis becomes the vertical direction. In some of the figures that follow, coordinate axes are displayed with the coordinate axes of
The taking lens 20 is configured with a plurality of optical lens groups, and forms an image of an object light flux from a scene in the proximity of its focal plane. The taking lens 20 may be a replaceable lens that can be attached to and detached from the digital camera 10. In this case, the camera body functions as an image-capturing device. For convenience of explanation, in
The image sensor 100 is arranged in the proximity of the focal plane of the taking lens 20. The image sensor 100 is an infrared image sensor photosensitive to light in an invisible band. In the present embodiment, as one example of it, the image sensor 100 is photosensitive to light in the range of 800 nm to 2000 nm in the near infrared band of 800 nm to 2500 nm. These near infrared band and range of photosensitivity are not limited to those in the present example.
The image sensor 100 includes a plurality of pixels arrayed two-dimensionally. Each one among the plurality of pixels includes a photoelectric converting unit, and a bandpass filter provided corresponding to the photoelectric converting unit. As described in detail below, in the present embodiment, there are three types of bandpass filter, and each one among the plurality of photoelectric converting units is provided with one of the bandpass filters.
The image sensor 100 is timing-controlled by the drive unit 204 to convert an object image formed on its light receiving surface into a pixel signal and output it to the A/D conversion circuit 202. The A/D conversion circuit 202 converts the pixel signal as an output signal output by the image sensor 100 into a digital signal. It outputs image-capturing data obtained by the digital conversion to the work memory 203.
The control unit 201 plays a role as a generating unit that generates infrared wavelength information as one example of invisible wavelength information for converting image-capturing data generated from the output signal of the image sensor 100 into visible color space image data. The visible color space shows a range of colors that a predetermined color coordinate system can handle. In other words, because it shows a range of colors that a color coordinate system can handle, the visible color space is in some cases referred to as a display system color space in the present specification. As described in detail below, the infrared wavelength information is defined based on the sensitivity characteristic which is the output characteristic of the image sensor 100 to an object light flux wavelength.
The image processing unit 205 performs various types of processing such as brightness correction processing on image-capturing data using the work memory 203 as a workspace. Also, the image processing unit 205 plays a role as a processing unit that relates infrared wavelength information as tag information to the image-capturing data that has undergone the various types of processing. An image-capturing file in which the image-capturing data and the infrared wavelength information are related to each other is recorded in a memory card 220 attached to the memory card IF 207.
The control unit 201 and the image processing unit 205 cooperate with each other to convert the image-capturing data into visible color space image data using the infrared wavelength information. Details of conversion are described below.
The image processing unit 205 can convert infrared wavelength band-based image-capturing data into visible wavelength band-based image data to generate the image data by allocating mutually different visible bands to respective wavelength bands of three types of bandpass filter.
The generated image data is converted into a display signal by the LCD drive circuit 210, and displayed on the display unit 209. As described in detail below, even objects that are difficult to color-discriminate in a visible wavelength band-based image, like water and oil for example, can be color-discriminated in an infrared wavelength band-based image. A menu screen for various types of setting is also displayed on the display unit 209. For example, a menu screen related to setting of a reference point described below is displayed. Also, the generated image data is recorded in the memory card 220 attached to the memory card IF 207.
The system memory 206 records a program to control the digital camera 10, various types of parameter and the like. In the present embodiment, an image-capturing parameter is stored. The image-capturing parameter includes information indicative of a light source characteristic, information indicative of transmittances of three types of bandpass filter, information indicative of the photosensitivity of the image sensor 100 and the like. The information indicative of transmittance may be stored as a table in which transmittances are respectively associated with wavelength bands of constant intervals, or may be stored as a function for calculating a transmittance corresponding to a wavelength band.
The manipulating unit 208 accepts a manipulation of a user to output a manipulation signal corresponding to the manipulation to the control unit 201. For example, if a menu screen related to setting of a reference point is displayed on the display unit 209, it outputs a manipulation signal related to the setting of the reference point to the control unit 201 according to the manipulation. A user can select a method of setting a reference point or the like through the manipulating unit 208.
Also, the manipulating unit 208 includes manipulating member such as a release switch, a cross-key or an OK key. The release switch is configured with a press-button that can sense manipulations at two steps in its pressing direction. The control unit 201 performs AF, AE and the like which are photographing preparatory operations by sensing SW1 which is a first step pressing, and performs an object image acquiring operation by the image sensor 100 by sensing SW2 which is a second step pressing. In the present embodiment, AF is performed so that an object image is focused in the infrared wavelength band.
The communicating unit 211 communicates with another device. The communicating unit 211 transmits an image-capturing file to the other device according to a manipulation by a user through the manipulating unit 208. Examples of the other device include a device provided with a display unit such as a personal computer, a smartphone or a tablet, and a server device on the Internet, and the like.
A series of photographing sequences is started upon the manipulating unit 208 accepting a manipulation by a user and outputting a manipulation signal to the control unit 201.
In the image sensor 100 as a whole, each one among a plurality of pixels arrayed two-dimensionally is provided discretely with any of the NIR1 filter, the NIR2 filter and the NIR3 filter. Accordingly, it can be said that the image sensor 100 detects an incoming object light flux while separating it into respective wavelength bands. In other words, the image sensor 100 performs photoelectric conversion while separating an object image formed on its light receiving surface into three wavelength bands that are in the infrared band and are mutually different.
As shown in (a) of
Each of the NIR1 filter, the NIR2 filter and the NIR3 filter is transmissive to the peak wavelengths of the other band filters. Specifically, the NIR1 filter is transmissive to the peak wavelength of the NIR2 filter. Likewise, the NIR3 filter is transmissive to the peak wavelength of the NIR2 filter. Also, the NIR2 filter is transmissive to the respective peak wavelengths of the NIR1 filter and the NIR3 filter.
As shown in (b) of
Here, the output characteristic of the image sensor 100 is calculated by multiplication of the transmittances of bandpass filters and the photosensitivity of the image sensor 100. In the present embodiment, the output characteristic of the image sensor 100 is defined as a range over which the image sensor 100 has a photosensitivity equal to or higher than a predetermined threshold (for example, 1%) in the photosensitivity calculated by multiplication.
As shown in (c) of
If the region 301 is defined, a coordinate (m, n) of a reference point RefP is set subsequently. As described in detail below, in the present embodiment, the coordinate (m, n) of the reference point RefP is set, as a default position, at a position corresponding to a reference white described below. If the lower limit value, the upper limit value and the reference point RefP are determined, a relationship between the reference point RefP and a wavelength at each point of the curved part of the region 301 is determined so as to set the coordinate (m, n) of the reference point RefP to (0.33, 0.33). That is, the coordinates of the peak wavelength λa, the peak wavelength kb and the peak wavelength λc are uniquely determined.
A region 302 is a region formed by linking, with straight lines, the respective points of the peak wavelength λa, the peak wavelength kb and the peak wavelength λc. The region 302 is larger than a region indicated by a region 303 described below.
The region 303 is a region formed by linking, with straight lines, a point AB on a straight line linking the peak wavelength λa and the reference point RefP, a point CD on a straight line linking the peak wavelength λb and the reference point RefP and a point EF on a straight line linking the peak wavelength λc and the reference point RefP. The region 303 indicates a color space that is actually reproducible. Although because the region 303 is a region of the infrared wavelength band, it is actually a region that cannot be visually recognized in colors, it is referred as a color space for convenience of explanation in the present specification. In the present specification, the region 303 is referred to as an image-capturing system color space as a space corresponding to the display system color space. Also, the region 303 is in some cases referred to as a virtual color space in a corresponding manner to the visible color space. In the image-capturing system color space, an infrared wavelength band passage of which each bandpass filter has allowed is expressed by a combination of numerical values. The region 303 is determined by the transmittance distribution of each bandpass filter.
If the coordinate (m, n) of the reference point RefP is set, a display resolution is determined. In other words, distances from the respective points of the peak wavelength λa, the peak wavelength kb and the peak wavelength λc to respective corresponding vertexes of the region 303 are determined. That is, the coordinate (a, b) of the point AB, the coordinate (c, d) of the point CD and the coordinate (e, f) of the point EF are determined. Thereby, the area of the region 303, that is, the color-saturation, is determined.
As mentioned above, the region 301 in (a) of
In the present embodiment, the coordinate (m, n) of the reference point RefP is associated with the coordinate (o, p) of the reference white RefW as a particular coordinate. The region 303 in (a) of
As illustrated in the figure, oil has a spectral intensity in a wide range between 900 nm and 1700 nm. In more detail, in the range of 900 nm to around 1350 nm, the spectral intensity of oil somewhat decreases in the range between around 1150 nm to around 1200 nm, but increases overall, and increases from around 0.1 to around 1. In the range of around 1350 nm to around 1700 nm, it temporarily increases in the range between around 1410 nm to around 1480 nm, but decreases overall, and decreases from around 1 to around 0.1.
The spectral intensity of water increases from a little less than 0.2 to around 1 in the range of 900 nm to around 1100 nm. In the range of around 1100 nm to around 1400 nm, it temporarily increases in the range of around 1150 nm to around 1270 nm, but decreases overall, and decreases from around 1 to around 0.2. In the range of around 1400 nm to around 1700 nm, it stays approximately flat, and somewhat decreases in the range of around 1650 nm to 1700.
Because the spectral intensity of oil ranges over a relatively wide range of the near infrared wavelength band, oil is recognized in whitish colors as compared with water in an image after conversion into the visible wavelength band. On the other hand, because the spectral intensity of water increases rapidly in the range up to around 1100 nm, and decreases rapidly in the range after around 1250 nm, it differs significantly from the spectral intensity of oil in the band up to around 1100 nm and the band after around 1200 nm. Because the spectral intensity corresponding to the wavelength bands of the NIR2 filter and the NIR3 filter is relatively low, water is recognized in bluish colors overall in an image after conversion into the visible wavelength band.
(b), (c), (d) and (e) of
In the example of (b) of
In the example of (c) of
In the example of (d) of
In the example of (e) of
As described above, colors of objects after conversion into the visible wavelength band can be adjusted by the position of the reference point RefP. By setting the position of the reference point RefP corresponding to an object spectrum, an object can be color-discriminated easily. If color difference discrimination thresholds such as those of a McAdam ellipse are stored in the system memory 206 in advance, the position of the reference point RefP may be set by referring to the discrimination thresholds so as to make color-discrimination easy. Although in the explanation above, an example of color-discrimination of water and oil was given, the reference point RefP may be set in the region 501 if foreign substances mixed into water are to be discriminated.
(f), (g), (h) and (i) of
(f) of
(g) of
(h) of
(i) of
As illustrated, in the infrared wavelength information, various types of information are written while being classified into respective categories. In the following, main types of information are explained.
In the category of file information, a type, size, image information and the like are written. Specifically, “JPG image” is written as the type, and “4.09 MB” is written as the size. As the image information, the numbers of dots in the x direction and the y direction of the present image data, and the number of bits which is the number of colors that each dot has are written. Specifically, “2020×1624 dots” is written as the number of dots, and “24 bits” is written as the number of bits.
In the category of photographing information, a date and time of photographing, exposure time, F number, ISO speed, focal distance and the like are written. Here, “2014/11/28 14:22:22” is written as the date and time of photographing, “1/500” is written as the exposure time, “4” is written as the F number, “200” is written as the ISO speed, and “50 mm” is written as the focal distance.
In the category of image-capturing system color space information, a light source characteristic, peak wavelength λa, peak wavelength λb, peak wavelength λc, primary stimulus value 1 (vertex AB), primary stimulus value 2 (vertex CD), primary stimulus value 3 (vertex EF), reference point, brightness correction, color processing, display color space, lower limit value and upper limit value of a target band, and the like are written.
The light source characteristic indicates the type of light source in a photographing condition. Here, “halogen light” is written. The peak wavelength λa is the peak wavelength of the NIR1 filter. Here, “1150 nm” is written. The peak wavelength λb is the peak wavelength of the NIR2 filter. Here, “1400 nm” is written. The peak wavelength λc is the peak wavelength of the NIR3 filter. Here, “1650 nm” is written. The region 302 explained with reference to
The primary stimulus value 1 is determined by the peak wavelength λa and the half width of the NIR1 filter, and the photosensitivity of the image sensor 100. Likewise, the primary stimulus value 2 is determined by the peak wavelength λb and the half width of the NIR2 filter, and the photosensitivity of the image sensor 100. The primary stimulus value 3 is determined by the peak wavelength λc and the half width of the NIR3 filter, and the photosensitivity of the image sensor 100. Although as explained already with reference to
In the reference point, “(0.33, 0.33)” is written as the coordinate of the reference point. This coordinate is a coordinate in the case where the reference point is set to the default, and is a coordinate corresponding to the reference white. If the reference point is set by a user, the coordinate of the set reference point is written. “γ=1” is written in the brightness correction.
The color processing indicates the resolution of display. In the color processing, “separate water and oil” is written. From this information, the gain corresponding to objects to be targets of color-discrimination can be determined, and the resolution of display can be determined.
The display color space indicates the color gamut that is to be set in a display system. Here, “sRGB” is written in the display color space. This information include the coordinate of the reference white RefW, and the coordinates of the three primary stimulus values in sRGB. In the example of the visible color space explained with reference to
The lower limit value and upper limit value of the target band indicate the range of the output characteristic of the image sensor 100. Here, “850 nm” is written as the lower limit value, and “1950 nm” is written as the upper limit value. As mentioned above, the region 301 explained with reference to
If the infrared wavelength information like the one described above is generated, the infrared wavelength information can be converted into a generally used image format according to a rule of conversion into the generally used image format as long as such a conversion rule is determined. That is, as explained already, if the virtual color space is associated with the visible color space, the same conversion can be performed for the infrared wavelength information related to a particular object. Thereby, a particular visible color is associated with a particular object. Because the same object is to be displayed in the same color, the reproducibility of reproduction can be ensured. For example, water as an object is displayed in bluish colors, and oil can be displayed in whitish colors. Also, regardless of the type of display device, the compatibility of reproduction can be ensured. From what is described above, infrared wavelength information can also be generated as information necessary for conversion into a generally used image format.
Although in the explanation above, the image sensor 100 includes three types of bandpass filter with which the lower limit value and upper limit value, as its output characteristic, become 850 nm and 1950 nm, respectively, it may include other bandpass filters. Hereinafter, the details are explained.
In the visible color space already explained, the purities of colors increase as the distance from the reference white RefW increases. In other words, a color of higher purity can be expressed as a visible color space region becomes larger. The same also applies to a case of generating an infrared wavelength band-based image. That is, a visible color space region becomes larger as a corresponding virtual color space region becomes larger; as a result, a color of higher purity can be expressed.
It can be known from comparison between the sensitivity distribution in (a) of
Although in the present example, the lower limit value and upper limit value of the output characteristic shown in (a) of
Although in the explanation above, the image sensor 100 is configured to include three types of bandpass filter, it may be configured to include further different three types of bandpass filter. That is, two types of filter set may be provided in a mixed manner. In this case, a filter set that allows easy color-discrimination may be selected according to an object. By generating a difference in hue to a certain extent at the stage of spectral diffraction, a more effective process can be expected at the stage of adjusting the position of a reference point so as to allow easy color-discrimination.
Depending on how the filters are set in relation to the spectra of the object P and the object Q, the regions in which the object P and the object Q are mapped to the virtual color space change. Comparison between (a) of
As shown in (c) of
On the other hand, as shown in (a) of
The control unit 201 judges whether or not the SW1 is pressed (Step S101). If it is judged that the SW1 is pressed (YES at Step S101), the control unit 201 judges whether or not the SW2 is pressed (Step S102). If it is judged that the SW2 is pressed (YES at Step S102), the control unit 201 performs a process of photographing (Step S103).
The control unit 201 reads out, as image-capturing parameters from the system memory 206, information indicative of the transmittances of three types of bandpass filter and information indicative of the photosensitivity of the image sensor 100. Then, it calculates the lower limit value and upper limit value of the output characteristic (Step S104). For example, as explained in association with
The control unit 201 judges whether there is an instruction about a reference point from a user (Step S105). If it is judged that there is an instruction about a reference point (YES at Step S105), a reference point is set following the instruction from the user (Step S106). For example, as explained in association with
The image processing unit 205 generates main image-capturing data by performing various types of processing on the image-capturing data obtained by digital conversion (Step S108). On the other hand, the control unit 201 generates infrared wavelength information (Step S109). For example, it generates the infrared wavelength information following the format shown in
The image processing unit 205 relates the infrared wavelength information to the main image-capturing data (Step S110) to make them into a file; thereby, it generates an image-capturing file (Step S111). The image processing unit 205 stores the image-capturing file in the memory card 220 (Step S112). The control unit 201 converts the infrared wavelength information into visible color space information (Step S113). For example, as explained with reference to
Although in the explanation above, the control unit 201 relates infrared wavelength information as tag information to image-capturing data, it may relate the information as a link file to the image-capturing data. Although in the explanation above, the CIE color coordinate system was explained as an example of a visible color coordinate system, it may be another color coordinate system.
Although in the explanation above, the digital camera 10 as one example of an image-capturing device is configured to include the display unit 209, the image-capturing device may be configured not to include the display unit 209. In this case, the image-capturing device may transmit an image-capturing file to another device including a display unit. The other device can convert infrared wavelength-based image-capturing data into image data and display it by performing the processes from Step S113 to Step S115 in
Although in the explanation above, bandpass filters are provided to the image sensor 100, the location at which bandpass filters are arranged is not limited to the image sensor 100. Bandpass filters may be provided as a filter unit crossing the optical axis 21 at a later stage of the taking lens 20. In this case, following user setting, the control unit 201 arranges three types of bandpass filter in order in the range of an object light flux and performs a photographing operation in synchronization with the arrangement of the respective ones. Then, the image processing unit 205 acquires sequentially three pieces of image-capturing data as image-capturing plain data configured with image-capturing data of all the pixels from the image sensor 100. Because with a configuration using a filter unit, image-capturing data corresponding to each one among the three types of bandpass filter can be obtained for all the pixels, the above-mentioned interpolation process does not have to be performed.
If the digital camera 10 is a digital camera whose lens is replaceable, and filter units are configured integral with replaceable lenses, image-capturing parameters may be stored in a lens memory in the replaceable lenses. The camera body may acquire image-capturing parameters from the lens memories.
Although in the explanation above, a reference point is associated with a reference white, because the reference point is a point just set for the purpose of separating colors, it does not necessarily have to be associated with the reference white. For example, if two objects are to be color-discriminated, it may be set right in the middle of regions of the two objects mapped to the virtual color space. Also, although the reference point is set in the region 303, it may be set outside the region 303 as long as it is set in the region 302. Although in the explanation above, the coordinate of a reference point is variable by user setting, it may be fixed. In this case, the items of reference values do not have to be present in the format explained with reference to
Although in the explanation above, the virtual color space is defined with x′ as the horizontal axis and y′ as the vertical axis, the virtual color space is first of all obtained by defining the infrared wavelength region space virtually as a color space, it may be defined with other axes as the horizontal axis and the vertical axis, respectively. Although in the explanation above, pitch widths of wavelengths from the lower limit value to the upper limit value of the region 301 are set in a manner similar to that for the region 401, they may be set to equal widths. Although in the explanation above, the lower limit value and upper limit value of the region 301 are variable, the lower limit value and upper limit value may be fixed. In this case, the shape of the region 301 becomes variable.
Although in the explanation above, in order to associate the virtual color space and the visible color space with each other, a reference point and a reference white are associated with each other after the lower limit values of target bands are associated with each other and upper limit values of the target bands are associated with each other, respectively, virtual color space three primary stimulus values and visible color space three primary stimulus values may be associated with each other, and additionally a reference point and a reference white may be associated with each other. In this case, the lower limit values of the target bands do not necessarily have to be associated with each other, and the upper limit values of the target bands do not necessarily have to be associated with each other. This is because the horseshoe-shape region 401 is just a range determined by monochromatic light, and the visible color space is represented by the region 403. Also, if virtual color space three primary stimulus values and visible color space three primary stimulus values are associated with each other, information indicative of the visible color space primary stimulus value to which each one among the virtual color space three primary stimulus values is associated may be written in the image-capturing system color space information.
The corresponding primary stimulus value 1 (vertex GH) indicates the visible color space primary stimulus value with which the virtual color space primary stimulus value 1 is associated. Likewise, the corresponding primary stimulus value 2 indicates the visible color space primary stimulus value with which the virtual color space primary stimulus value 2 is associated, and the corresponding primary stimulus value 3 indicates the visible color space primary stimulus value with which the virtual color space primary stimulus value 3 is associated. Here, the coordinate “(0.14, 0.06)” of the vertex GH is written as the corresponding primary stimulus value 1, the coordinate “(0.20, 0.68)” of the vertex IJ is written as the corresponding primary stimulus value 2 and the coordinate “(0.63, 0.32)” of the vertex KL is written as the corresponding primary stimulus value 3. That is, the virtual color space primary stimulus value 1 (vertex AB) is associated with the visible color space vertex GH, the virtual color space primary stimulus value 2 (vertex CD) is associated with the visible color space vertex IJ and the virtual color space primary stimulus value 3 (vertex EF) is associated with the visible color space vertex KL. Also, the coordinate of the reference point RefP is associated with (0.33, 0.33). As explained already, (0.33, 0.33) is the coordinate of the reference point RefP if it is set to a default, and a user can freely set the coordinate of the reference point. In this case also, the coordinate of the reference point RefP is associated with (0.33, 0.33) which is the coordinate of the reference white RefW.
Also, as explained with reference to
Although in the explanation above, the respective virtual color space three primary stimulus values are associated with the corresponding visible color space primary stimulus values with the corresponding stimulus value 1, the corresponding stimulus value 2 and the corresponding stimulus value 3, the virtual color space three primary stimulus values and the visible color space three primary stimulus values may be related to each other by another method. For example, the virtual color space three primary stimulus values and the visible color space three primary stimulus values can be associated with each other by utilizing the order in which the three primary stimulus values are written. Specifically, a primary stimulus value written first is associated with the coordinate (0.14, 0.06) of the vertex GH in the visible color space, a primary stimulus value written second is associated with the coordinate (0.20, 0.68) of the vertex IJ in the visible color space, and a primary stimulus value written third is associated with the coordinate (0.63, 0.32) of the vertex KL in the visible color space.
Here, as explained already, if “sRGB” is written as the display color space, (0.14, 0.06) is written as the coordinate (g, h) of the vertex GH, (0.20, 0.68) is written as the coordinate (i, j) of the vertex IJ, and (0.63, 0.32) is written as the coordinate (k, l) of the vertex KL; on the other hand, if information indicative of another color gamut is written, different values from those described above are written as three primary stimulus values. That is, information written as the display color space and visible color space three primary stimulus values correspond to each other in a one-to-one relationship. Accordingly, if the order in which three primary stimulus values are written is utilized, only a display color space has to be designated even if visible color space three primary stimulus values are not written for each piece of image data. The display system can associate virtual color space three primary stimulus values with visible color space three primary stimulus values according to a designated display color space following a rule related to the above-mentioned order in which three primary stimulus values are written.
Using (a) of
In the explanation above, the virtual color space three primary stimulus values and the visible color space three primary stimulus values are respectively associated according to wavelengths. That is, primary stimulus values that are on the short wavelength side are related to each other, primary stimulus values on the long wavelength side are related to each other, and primary stimulus values of wavelengths between the short wavelength side and the long wavelength side are related to each other, but a manner of relating them is not limited to this. Hereinafter, the details are explained. In the following example, a case where the virtual color space three primary stimulus values and the visible color space three primary stimulus values are associated with each other by utilizing the order in which the three primary stimulus values are written is explained. Accordingly, as illustrated in the figure, the image-capturing system color space information in (b) of
The order in which the three primary stimulus values are written in the image-capturing system color space information in (b) of
Although in the explanation above, the near infrared band is mentioned as an example of the invisible band, it may be another band as along as it is a band outside the visible band. For example, if the image sensor 100 is photosensitive to light in the ultraviolet band, the control unit 201 can generate ultraviolet wavelength information for converting ultraviolet band-based image-capturing data into visible color space image data.
(a) of
The vertex AB which is the virtual color space primary stimulus value 1 is associated with the visible color space vertex GH. Likewise, the vertex CD which is the virtual color space primary stimulus value 2 is associated with the visible color space vertex IJ, and the vertex EF which is the virtual color space primary stimulus value 3 is associated with the visible color space vertex KL. Also, the reference point RefP is associated with the reference white RefW. Similar to the case of the infrared wavelength band, if coordinates representing an object are mapped to the region 303, the coordinates can be converted into coordinates in the region 403. That is, they can be converted into visible colors.
Although in the explanation above, the NIR1 filter and the NIR3 filter are transmissive to the peak wavelength of the NIR2 filter, and the NIR2 filter is transmissive to the respective peak wavelengths of the NIR1 filter and the NIR3 filter, in the output characteristic of the image sensor 100 explained already with reference to (c) of
If the image processing unit 205 converts infrared wavelength band-based image-capturing data into visible wavelength band-based image-capturing data to generate the image data, not only the NIR1 filter, the NIR2 filter and the NIR3 filter are associated with mutually different visible wavelength bands, respectively, but also respective wavelength band pixel signals are used and converted into visible wavelength band pixel signals. For example, a pixel signal of the wavelength band of the NIR1 filter is converted into a pixel signal of a visible wavelength band using, in addition to the pixel signal, a pixel signal of the wavelength band of the NIR2 filter and a pixel signal of the wavelength band of the NIR3 filter. In this manner, by using pixel signals of all the wavelength bands of the NIR1 filter, the NIR2 filter and the NIR3 filter, color representation at a high resolution is made possible.
In order to enhance the resolution of color representation, all of the NIR1 filter, the NIR2 filter and the NIR3 filter preferably have transmittances equal to or higher than 1% in the output characteristic of the image sensor 100 as mentioned above. However, this is not the only configuration that enhances the resolution of color representation. At least, in one possible configuration, at the peak wavelength of the wavelength band of the NIR1 filter which is on the shortest wavelength side, the wavelength band of the NIR3 filter which is on the longest wavelength side overlaps the former wavelength band. Specifically, at 1150 nm which is the peak wavelength of the wavelength band of the NIR1 filter, the NIR3 filter may have a transmittance equal to or higher than 1%. On the other hand, in one possible configuration, at least, at the peak wavelength of the wavelength band of the NIR3 filter which is on the longest wavelength side, the wavelength band of the NIR1 filter which is on the shortest wavelength side overlaps the former wavelength band. Specifically, at 1650 nm which is the peak wavelength of the wavelength band of the NIR3 filter, the NIR1 filter may have a transmittance equal to or higher than 1%. With such a combination also, because the NIR1 filter, the NIR2 filter and the NIR3 filter overlap one another over wavelength bands at least in a range including the respective peak wavelengths, color representation in multiple colors becomes possible.
In the example of
As explained already using (a) of
In the example of
In this example, the water Objw is mapped to a position close to the point IJ in the color coordinate system chromaticity diagram. Thus, it is recognized in greenish colors overall in an image after conversion into the visible wavelength band. On the other hand, the oil Objo is mapped to a position close to the reference white RefW in the color coordinate system chromaticity diagram. Thus, it is recognized in somewhat whitish color as compared with water in an image after conversion into the visible wavelength band.
In the example of
In this example, the water Objw is mapped to a position close to the point KL in the color coordinate system chromaticity diagram. Thus, it is recognized in reddish colors overall in an image after conversion into the visible wavelength band. On the other hand, the oil Objo is mapped to a position close to the reference white RefW in the color coordinate system chromaticity diagram. Thus, it is recognized in somewhat whitish color as compared with water in an image after conversion into the visible wavelength band.
As explained above, depending on how the virtual color space three primary stimulus values and the visible color space three primary stimulus values are related to each other, an observation target can be represented in different colors. Humans are known to be generally more sensitive to differences in bluish colors and reddish colors than to greenish colors. In view of this, in the present embodiment, the observation information for color-discriminating water and oil indicates that the coordinate (a, b) of the point AB is associated with the coordinate (k, l) of the point KL, the coordinate (c, d) of the point CD is associated with the coordinate (g, h) of the point GH and the coordinate (e, f) of the point EF is associated with the coordinate (i, j) of the point IJ. By performing color allocation according to this observation information, the image processing unit 205 can allocate a primary stimulus value corresponding to the highest spectral intensity among spectral intensities corresponding to the respective wavelength bands of the NIR1 filter, the NIR2 filter and the NIR3 filter to the point KL, that is, a reddish color. The observation information may be generated to indicate that the coordinate (a, b) of the point AB is associated with the coordinate (g, h) of the point GH, that is, a bluish color, the coordinate (c, d) of the point CD is associated with the coordinate (i, j) of the point IJ, and the coordinate (e, f) of the point EF is associated with the coordinate (k, l) of the point KL. As described above, by switching over manners of relating colors in accordance with the human visual characteristics corresponding to an object spectrum as appropriate, it becomes possible to make color-discrimination of an object easy in a visible image.
The control unit 201 judges whether or not the SW1 is pressed (Step S201). If it is judged that the SW1 is pressed (YES at Step S201), the control unit 201 judges whether or not the SW2 is pressed (Step S202). If it is judged that the SW2 is pressed (YES at Step S202), the control unit 201 performs a process of photographing (Step S103).
The image processing unit 205 generates main image-capturing data by performing various types of processing on the image-capturing data obtained by digital conversion (Step S204). The image processing unit 205 acquires, from the system memory 206, observation information corresponding to a set observation target (Step S205). Here, observation information for color-discriminating water and oil is acquired. The image processing unit 205 acquires virtual color space three primary stimulus values and visible color space three primary stimulus values (Step S206). The image processing unit 205 uses the acquired observation information, and the virtual color space three primary stimulus values and the visible color space three primary stimulus values to determine color allocation (Step S207). As explained already with reference to
After converting the color image data into display signals by the LCD drive circuit 210, it is displayed on the display unit 209 (Step S209). The control unit 201 judges whether the power source is turned off (Step S210). If it is judged that the power source remains turned on (NO at Step S210), it proceeds to Step S201, and if it is judged that the power source is turned off (YES at Step S210), the series of processing ends.
Although in the explanation above, the image processing unit 205 associates the respective virtual color space three primary stimulus values with points in the visible color space, at least one of the virtual color space three primary stimulus values may be associated with a point outside the visible color space.
In the example of
Although in the explanation above, at least one of virtual color space three primary stimulus values is associated with a point outside the visible color space, it does not have to be outside the visible color space. For example, if by applying, to the coordinate (i, j) of the point IJ, an amplification gain in a direction toward the outside of the region 401, the color-saturation can be improved, it may be associated with a point in the visible color space. An amplification gain may be applied to another primary stimulus value other than the coordinate (i, j) of the point IJ, or an amplification gain may be applied to a plurality of primary stimulus values.
Although in the explanation above, the image processing unit 205 associates the primary stimulus value corresponding to the highest spectral intensity with the point KL or the point GH, if the system memory 206 stores a color difference discrimination threshold, it may perform association using this color difference discrimination threshold. As one example of color difference discrimination thresholds, a discrimination threshold corresponding to a McAdam ellipse can be used.
In the example of (a) and (b) of
In the example of (c) and (d) of
Based on what is described above, the image processing unit 205 may alter color allocation if the observation target Obj1 and the observation target Obj2 are mapped to the same McAdam ellipse. For example, it may alter color allocation into the one explained with reference to (c) and (d) of
Responding to a user manipulation accepted by the manipulating unit 208, the image processing unit 205 may change any one or more of the hue, color-saturation, brightness and gradation characteristic of at least one piece of pixel data related to a region. For example, data can be converted into a color that allows easier color-discrimination by changing the hue.
Although in the explanation above, observation information indicates how virtual color space three primary stimulus values and visible color space three primary stimulus values are related to each other, it may be information indicative of the spectrum itself of an observation target. In this case, the image processing unit 205 may determine how to relate virtual color space three primary stimulus values and visible color space three primary stimulus values based on the output characteristic of the image sensor 100 and the observation information. For example, as mentioned above, a primary stimulus value corresponding to the highest spectral intensity may be associated with the point KL or the point GH.
The control unit 201 may generate infrared wavelength information as one example of invisible wavelength information for converting image-capturing data generated from the output signal of the image sensor 100 into visible color space image data. As described in detail below, the infrared wavelength information is defined based on the sensitivity characteristic which is the output characteristic of the image sensor 100 to an object light flux wavelength.
The image processing unit 205 may record, in the memory card 220, an image-capturing file in which infrared wavelength information as tag information is related to image-capturing data having undergone various types of processing. Then, using the infrared wavelength information, it may convert the infrared wavelength band-based image-capturing data into visible wavelength band-based image data to generate color image data.
While the embodiments of the present invention have been described, the technical scope of the invention is not limited to the above described embodiments. It is apparent to persons skilled in the art that various alterations and improvements can be added to the above-described embodiments. It is also apparent from the scope of the claims that the embodiments added with such alterations or improvements can be included in the technical scope of the invention.
The operations, procedures, steps, and stages of each process performed by an device, system, program, and method shown in the claims, embodiments, or diagrams can be performed in any order as long as the order is not indicated by “prior to,” “before,” or the like and as long as the output from a previous process is not used in a later process. Even if the photographing process flow is described using phrases such as “first” or “next” in the claims, embodiments, or diagrams, it does not necessarily mean that the process must be performed in this order.
Various embodiments of the present invention may be described with reference to flowcharts and block diagrams whose blocks may represent (1) steps of processes in which operations are performed or (2) units of apparatuses responsible for performing operations. Certain steps and units may be implemented by dedicated circuitry, programmable circuitry supplied with computer-readable instructions stored on computer-readable media, and/or processors supplied with computer-readable instructions stored on computer-readable media. Dedicated circuitry may include digital and/or analog hardware circuits and may include integrated circuits (IC) and/or discrete circuits. Programmable circuitry may include reconfigurable hardware circuits comprising logical AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, memory elements, etc., such as field-programmable gate arrays (FPGA), programmable logic arrays (PLA), etc.
Computer-readable media may include any tangible device that can store instructions for execution by a suitable device, such that the computer-readable medium having instructions stored therein comprises an article of manufacture including instructions which can be executed to create means for performing operations specified in the flowcharts or block diagrams. Examples of computer-readable media may include an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, etc. More specific examples of computer-readable media may include a floppy (registered trademark) disk, a diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a BLU-RAY (registered trademark) disc, a memory stick, an integrated circuit card, etc.
Computer-readable instructions may include assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, JAVA (registered trademark), C++, etc., and conventional procedural programming languages, such as the “C” programming language or similar programming languages.
Computer-readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to programmable circuitry, locally or via a local area network (LAN), wide area network (WAN) such as the Internet, etc., to execute the computer-readable instructions to create means for performing operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2015-072660 | Mar 2015 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 3884552 | Gerharz | May 1975 | A |
| 4343020 | Stapleton | Aug 1982 | A |
| 4418359 | Stapleton | Nov 1983 | A |
| 8724894 | Jensen | May 2014 | B1 |
| 8941678 | Kurtz | Jan 2015 | B2 |
| 9425227 | Wang | Aug 2016 | B1 |
| 20050182328 | Matsumoto | Aug 2005 | A1 |
| 20060071156 | Masaki | Apr 2006 | A1 |
| 20080203305 | Suzuki | Aug 2008 | A1 |
| 20090009621 | Yamaguchi | Jan 2009 | A1 |
| 20090012405 | Hasegawa et al. | Jan 2009 | A1 |
| 20110102563 | Johnson, Jr. | May 2011 | A1 |
| 20110228097 | Motta | Sep 2011 | A1 |
| 20110309755 | Wirth | Dec 2011 | A1 |
| 20130141574 | Dalal | Jun 2013 | A1 |
| 20150062347 | Jin | Mar 2015 | A1 |
| Number | Date | Country |
|---|---|---|
| S57-48782 | Mar 1982 | JP |
| 2002-171519 | Jun 2002 | JP |
| 2004-077143 | Mar 2004 | JP |
| 2006-109120 | Apr 2006 | JP |
| 2007-174126 | Jul 2007 | JP |
| 2007174126 | Jul 2007 | JP |
| 2015053578 | Mar 2015 | JP |
| 2007-083437 | Jul 2007 | WO |
| WO-2009098968 | Aug 2009 | WO |
| Entry |
|---|
| M. Vilaseca et al., Color Visualization System for Near-Infrared Multispectral Images, 2004, CGIV, The Second European Conference on Colour Graphics, Imaging and Vision, pp. 431-436. (Year: 2004). |
| Jun. 21, 2016 International Search Report issued in International Patent Application No. PCT/JP2016/060571. |
| Number | Date | Country | |
|---|---|---|---|
| 20180020200 A1 | Jan 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2016/060571 | Mar 2016 | US |
| Child | 15715999 | US |
