LIGHT DETECTING DEVICE, LIGHT DETECTING SYSTEM, AND FILTER ARRAY

Abstract

A light detecting device includes: a filter array including filters two-dimensionally arrayed and an image sensor including light detection elements. Each of first and second filters included in the filters includes a first reflective layer, a second reflective layer, and an intermediate layer therebetween and has a resonance structure having resonant modes whose orders are different from each other. A refractive index and/or a thickness of the intermediate layer in the first and second filters is different depending on the filter. A transmission spectrum of each of the first and second filters has local maximum value of transmittance at each of wavelengths included in a wavelength region, and the wavelengths correspond to the resonant modes, respectively. The image sensor is disposed at a position where the image senor receives passing light that passes through the filter array, to detect components in the wavelengths included in the passing light.

Description

BACKGROUND

1. Technical Field

The present disclosure relates to a light detecting device, a light detecting system, and a filter array.

2. Description of the Related Art

Utilizing spectral information of a large number of bands, for example, several tens of bands, each of which is a narrowband, makes it possible to determine detailed properties of a target object, the determination having been impossible with conventional RGB images. Cameras that acquire such multi-wavelength information are called “hyperspectral cameras”. For example, as disclosed in U.S. Patent Application Publication No. 2016/138975, U.S. Pat. Nos. 7,907,340, 9,929,206, Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2013-512445, and Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2015-501432, the hyperspectral cameras are utilized in various fields for food inspection, living-body examination, drug development, mineral component analysis, and so on.

SUMMARY

In one general aspect, the techniques disclosed here feature a light detecting device comprising: a filter array including a plurality of filters that is two-dimensionally arrayed and an image sensor including a plurality of light detection elements. The plurality of filters includes a first filter and a second filter. Each of the first filter and the second filter includes a first reflective layer, a second reflective layer, and an intermediate layer between the first reflective layer and the second reflective layer and has a resonance structure having resonant modes. orders of the resonant modes being different from each other. At least one selected from the group consisting of a refractive index and a thickness of the intermediate layer in the first filter is different from the at least one selected from a refractive index and a thickness of the intermediate layer in the second filter. A transmission spectrum of each of the first filter and the second filter has a local maximum value of transmittance at each of a plurality of wavelengths included in a wavelength region, and the wavelengths correspond to the resonant modes, respectively. The image sensor is disposed at a position where the image senor receives passing light that passes through the filter array, to detect components in the plurality of wavelengths included in the passing light.

It should be noted that general or specific embodiments may be implemented as a system, a method, an integrated circuit, a computer program, a storage medium, or any selective combination thereof.

Additional benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. The benefits and/or advantages may be individually obtained by the various embodiments and features of the specification and drawings, which need not all be provided in order to obtain one or more of such benefits and/or advantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram schematically showing a light detecting system in an exemplary embodiment;

FIG. 2A is a view schematically showing an example of a filter array;

FIG. 2B is a view showing one example of spatial distributions of transmittances of light in respective wavelength regions in a target wavelength region;

FIG. 2C is a graph showing an example of a transmission spectrum of one of two areas included in a plurality of areas in the filter array shown in FIG. 2A;

FIG. 2D is a graph showing an example of a transmission spectrum of the other of the two areas included in the plurality of areas in the filter array shown in FIG. 2A;

FIG. 3A is a diagram for describing a relationship between the target wavelength region and a plurality of wavelength regions included therein;

FIG. 3B is a diagram for describing a relationship between the target wavelength region and the plurality of wavelength regions included therein;

FIG. 4A is a graph for describing characteristics of a transmission spectrum in a certain area in the filter array;

FIG. 4B is graph showing a result of averaging the transmission spectra for each of the wavelength regions shown in FIG. 4A;

FIG. 5 is a sectional view schematically showing a light detecting device in an exemplary embodiment;

FIG. 6 is a graph schematically showing an example of transmission spectra of pixels;

FIG. 7 is a graph showing one example of a calculation result of transmission spectra of a Fabry-Perot filter;

FIG. 8A includes graphs showing respective transmission spectra of nine types of multi-mode filter;

FIG. 8B includes graphs showing respective transmission spectra of nine types of single-mode filter;

FIG. 8C is a view showing original images and two examples of reconstructed separate images;

FIG. 8D is a graph showing a calculation result of mean squared errors between the original image and each reconstructed separate image;

FIG. 8E is a graph showing two transmission spectra that are overlayed, the two transmission spectra being included in the transmission spectra of the nine types of multi-mode filter shown in FIG. 8A;

FIG. 8F is a graph obtained by extracting the wavelength region of 550 nm to 650 nm in the transmission spectrum in FIG. 8E;

FIG. 9 is a diagram schematically showing an example of two Fabry-Perot filters whose thicknesses of intermediate layers are the closest in the filter array;

FIG. 10 is a graph for describing the wavelengths of light detected by two pixels when light is orthogonally or obliquely incident on a Fabry-Perot filter;

FIG. 11A is a graph showing incident angle dependency of a transmission spectrum of a Fabry-Perot filter including an intermediate layer with a refractive index=1.5;

FIG. 11B is a graph showing incident angle dependency of a transmission spectrum of a Fabry-Perot filter including an intermediate layer with a refractive index=2.35;

FIG. 12A is a view schematically showing a first modification of the light detecting device shown in FIG. 5;

FIG. 12B is a view schematically showing a second modification of the light detecting device shown in FIG. 5;

FIG. 12C is a view schematically showing a third modification of the light detecting device shown in FIG. 5;

FIG. 12D is a view schematically showing a fourth modification of the light detecting device shown in FIG. 5;

FIG. 12E is a view schematically showing a fifth modification of the light detecting device shown in FIG. 5; and

FIG. 12F is a view schematically showing a sixth modification of the light detecting device shown in FIG. 5.

DETAILED DESCRIPTION

Before an embodiment of the present disclosure is described, a description will be given of knowledge underlying the present disclosure.

U.S. Patent Application Publication No. 2016/138975 discloses an imaging device that can acquire a high-resolution multi-wavelength image. In the imaging device, an optical element called an “encoding element” encodes an optical image from a target object to perform imaging. The encoding element has a plurality of areas that is two-dimensionally arrayed. The transmission spectrum of each of at least two areas of the plurality of areas has local maximum values of transmittance in respective wavelength regions. The areas are disposed, for example, so as to respectively correspond to pixels of an image sensor. In imaging using the encoding element, data of each pixel includes information of a plurality of wavelength regions. That is, image data that is generated is data resulting from compression of wavelength information. Accordingly, it is sufficient to hold two-dimensional data, thus making it possible to reduce the amount of data. For example, even when the capacity of a recording medium has a constraint, it is possible to obtain data of long-term video.

The encoding element can be manufactured using various methods. For example, a method using organic material, such as dye or colorant, is conceivable. In this case, the areas in the encoding element are formed of light-absorbing materials having different light transmission characteristics. In such a structure, the number of manufacturing steps increases according to the number of types of light-absorbing material that are disposed. Thus, it is not easy to fabricate the encoding element using organic materials.

Meanwhile, U.S. Pat. Nos. 7,907,340, 9,929,206, Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2013-512445, and Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2015-501432 each disclose a device including a plurality of Fabry-Perot filters having mutually different transmission spectra. The Fabry-Perot filters can be more easily fabricated than filters formed of organic materials. However, in any of the examples disclosed in U.S. Pat. Nos. 7,907,340, 9,929,206, Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2013-512445, and Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2015-501432, data of each pixel includes only information of a single wavelength region. Thus, the spatial resolution is sacrificed.

Based on the consideration above, the present inventors have conceived a light detecting device and a filter array as recited in the following items.

(First Item)

A light detecting device according to a first item includes: a filter array including a plurality of filters that is two-dimensionally arrayed; and an image sensor including a plurality of light detection elements. The plurality of filters includes a first filter and a second filter. Each of the first filter and the second filter includes a first reflective layer, a second reflective layer, and an intermediate layer between the first reflective layer and the second reflective layer and has a resonance structure having resonant modes whose orders are different from each other. At least one selected from the group consisting of a refractive index and a thickness of the intermediate layer in the first filter is different from the at least one selected from a refractive index and a thickness of the intermediate layer in the second filter. In other words, a refractive index of the intermediate layer in the first filter is different from a refractive index of the intermediate layer in the second filter and/or a thickness of the intermediate layer in the first filter is different from a thickness of the intermediate layer in the second filter. A transmission spectrum of each of the first filter and the second filter has a local maximum value of transmittance at each of a plurality of wavelengths included in a certain wavelength region. The wavelengths correspond to the resonant modes, respectively. In other words, each of the plurality of wavelengths corresponds to a corresponding one of the resonant modes. The image sensor is disposed at a position where the image senor receives passing light that passes through the filter array, to detect components in the plurality of wavelengths included in the passing light.

With this light detecting device, it is possible to acquire a high-resolution multi-wavelength image.

(Second Item)

In the light detecting device according to the first item, the wavelength region may be greater than or equal to 400 nm and be less than or equal to 700 nm.

With this light detecting device, it is possible to acquire a high-resolution multi-wavelength image in a visible light region.

(Third Item)

In the light detecting device according to the second item, the transmission spectrum may have the local maximum value of transmittance at each of four or more wavelengths included in the wavelength region.

With this light detecting device, it is possible to obtain pieces of spectroscopic information, the number thereof surpassing three wavelengths of RGB in the visible light region.

(Fourth Item)

In the light detecting device according to the second item, the transmission spectrum may have the local maximum value of transmittance at each of six or more wavelengths included in the wavelength region.

With this light detecting device, spectroscopic information of a larger number of wavelengths can be obtained in the visible light region.

(Fifth Item)

In the light detecting device according to the first item, at least one m that satisfies both

$n_{1}L\le 350m\mathrm{and}{n}_{2}L\ge 200\left(m+1\right)$

may exist, where L represents the thickness of the intermediate layer in the first filter or the second filter, n₁ represents a refractive index of the intermediate layer with respect to light with a wavelength of 700 nm, n₂ represents a refractive index of the intermediate layer with respect to light with a wavelength of 400 nm, and m is an integer greater than or equal to 1.

In this light detecting device, two or more wavelength peaks exist in the visible light region.

(Sixth Item)

In the light detecting device according to the first item, at least one m that satisfies both

$n_{1}L\le 350m\mathrm{and}{n}_{2}L\ge 200\left(m+3\right)$

may exist, where L represents the thickness of the intermediate layer in the first filter or the second filter, n₁ represents a refractive index of the intermediate layer with respect to light with a wavelength of 700 nm, n₂ represents a refractive index of the intermediate layer with respect to light with a wavelength of 400 nm, and m is an integer greater than or equal to 1.

In this light detecting device, four or more wavelength peaks exist in the visible light region.

(Seventh Item)

In the light detecting device according to the first item, at least one m that satisfies both

$n_{1}L\le 350m\mathrm{and}{n}_{2}L\ge 200\left(m+5\right)$

may exist, where L represents the thickness of the intermediate layer in the first filter or the second filter, n₁ represents a refractive index of the intermediate layer with respect to light with a wavelength of 700 nm, n₂ represents a refractive index of the intermediate layer with respect to light with a wavelength of 400 nm, and m is an integer greater than or equal to 1.

In this light detecting device, six or more wavelength peaks exist in the visible light region.

(Eighth Item)

In the light detecting device according to one of the first to seventh items,

$\Delta L>L\left(1-\sqrt{1-\frac{{\sin }^2{\theta }_i}{n^2}}\right)$

may be satisfied, where L represents the thickness of the intermediate layer in the first filter, L+ΔL represents the thickness of the intermediate layer in the second filter, n represents the refractive index of the intermediate layer in the first filter and the refractive index of the intermediate layer in the second filter, and θ_i represents a maximum incident angle of light that is incident on the filter array.

With this light detecting device, it is possible to suppress false detection, that is, light to be detected by one light detection element being detected by another light detection element.

(Ninth Item)

In the light detecting device according to the eighth item,

$\Delta L>2L\left(1-\sqrt{1-\frac{{\sin }^2{\theta }_i}{n^2}}\right)$

may be further satisfied.

With this light detecting device, it is possible to further suppress the false detection, that is, light to be detected by one light detection element being detected by another light detection element.

(Tenth Item)

The light detecting device according to one of the first to ninth items, at least one selected from the group consisting of the first reflective layer and the second reflective layer may include at least one selected from the group consisting of a dielectric multilayer film and a metal film.

(11th Item)

The light detecting device according to one of the first to tenth items, the intermediate layer may contain at least one selected from the group consisting of silicon, silicon nitride, titanium oxide, niobium oxide, and tantalum oxide.

With this light detecting device, it is possible to obtain an intermediate layer having a high refractive index.

(12th Item)

In the light detecting device according to one of the first to 11th items, the intermediate layer may be continuously provided across the first filter and the second filter.

This light detecting device makes it possible to simplify the manufacturing process.

(13th Item)

In the light detecting device according to one of the first to 12th items, at least one selected from the group consisting of the first reflective layer and the second reflective layer may be continuously provided across the first filter and the second filter.

This light detecting device makes it possible to simplify the manufacturing process.

(14th Item)

In the light detecting device according to one of the first to 13th items, at least one of the plurality of filters may be transparent.

This light detecting device can also simultaneously acquire a monochrome image without passing through multi-mode filters.

(15th Item)

In the light detecting device according to one of the first to 14th items, the filter array may be in contact with the image sensor.

In this light detecting device, the light detecting device can be monolithically configured.

(16th Item)

In the light detecting device in any of the first to 14th items, the filter array may be separated from the image sensor.

This light detecting device makes it possible to enhance the degree of design freedom.

(17th Item)

In the light detecting device according to one of the first to 16th items, the filter array may further include a transparent layer for planarizing a level difference between a surface of the first filter and a surface of the second filter.

In this light detecting device, the transparent layer planarizes the level difference between the surface of the first filter and the surface of the second filter to thereby facilitate that another member is disposed on the transparent layer.

(18th Item)

In the light detecting device according to one of the first to 17th items, each of the plurality of filters may have the resonance structure.

(19th Item)

The light detecting device according to one of the first to 18th items may further include a first microlens disposed above the first filter and a second microlens disposed above the second filter.

In this light detecting device, the first microlens and the second microlens concentrate incident light to thereby make it possible to efficiently detect light.

(20th Item)

In the light detecting device according to one of the first to 19th items, the transmission spectrum of each of the first filter and the second filter may have a peak including the local maximum value of transmittance and portions that respectively extend in a wavelength-decreasing direction and a wavelength-increasing direction from the local maximum value; and the peak in the transmission spectrum of the first filter and the peak in the transmission spectrum of the second filter may at least partly overlap each other.

In this light detecting device, in addition to wavelength information of the light detection element that receives light that passes through the first filter and the light detection element that receives light that passes through the second filter, information of correlation between these two light detection elements that share the wavelength information can also be utilized for a multi-wavelength image, and thus, it is possible to acquire a high-accuracy multi-wavelength image. In the light detecting device according to one of the first to 19th items, each of the plurality of light detection elements may detect the components in the plurality of wavelengths included in the passing light that passes through at least one filter included in the filter array. In the light detecting device according to one of the first to 19th items, a level difference between a surface of the first filter facing the image sensor and a surface of the second filter facing the image sensor may exist.

(21st Item)

A light detecting system according to a 21st item includes: the light detecting device according to one of the first to 20th items; and a signal processing circuit. The signal processing circuit generates image data including information of a wavelength included in the wavelength region, based on signals from the light detection elements.

This light detecting system can generate image data including information of a wavelength included in the wavelength region.

(22nd Item)

In the light detecting system according to the 21st item, the image data may include data representing images corresponding to respective two or more wavelengths included in the wavelength region.

This light detecting system obtains image data including data representing images corresponding to respective two or more wavelengths included in the wavelength region.

(23rd Item)

In the light detecting system according to the 22nd item, the number of picture elements included in each of the images may be larger than M/N, where N represents the number of images, and M represents the number of light detection elements.

This light detecting system makes it possible to suppress a reduction in the resolution of each of the plurality of images.

(24th Item)

In the light detecting system according to the 23rd item, the number of picture elements may be equal to the number of light detection elements.

This light detecting system can acquire a high-resolution multi-wavelength image. In the light detecting system according to the 21st item, each of the plurality of light detection elements may detect the components in the plurality of wavelengths included in the passing light that passes through at least one filter included in the filter array, and the signal processing circuit may generate the image data based on the components in the plurality of wavelengths detected by each of the plurality of light detection elements.

(25th Item)

A filter array according to a 25th item includes a plurality of filters two-dimensionally arrayed. The plurality of filters includes a first filter and a second filter. Each of the first filter and the second filter includes a first reflective layer, a second reflective layer, and an intermediate layer between the first reflective layer and the second reflective layer and has a resonance structure having resonant modes whose orders are different from each other. At least one selected from the group consisting of a refractive index and a thickness of the intermediate layer in the first filter is different from the at least one selected from a refractive index and a thickness of the intermediate layer in the second filter. In other words, a refractive index of the intermediate layer in the first filter is different from a refractive index of the intermediate layer in the second filter and/or a thickness of the intermediate layer in the first filter is different from a thickness of the intermediate layer in the second filter. A transmission spectrum of each of the first filter and the second filter has a local maximum value of transmittance at each of a plurality of wavelengths included in a certain wavelength region. The wavelengths correspond to the resonant modes, respectively. In other words, each of the plurality of wavelengths corresponds to a corresponding one of the resonant modes.

This filter array makes it possible to acquire a high-resolution multi-wavelength image.

In the present disclosure, all or a part of circuits, units, devices, members, or portions or all or a part of functional blocks in the block diagrams can be implemented by, for example, one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), or a large-scale integration (LSI). The LSI or IC may be integrated into one chip or may be constituted by combining a plurality of chips. For example, functional blocks other than a storage element may be integrated into one chip. Although the name used here is an LSI or IC, it may also be called a system LSI, a very large scale integration (VLSI), or an ultra large scale integration (ULSI) depending on the degree of integration. A field programmable gate array (FPGA) that can be programmed after manufacturing an LSI or a reconfigurable logic device that allows reconfiguration of the connection relationship or setup of circuit cells inside the LSI can also be used for the same purpose.

In addition, functions or operations of all or a part of circuits, units, devices, members, or portions can be executed by software processing. In this case, the software is recorded on one or more non-transitory recording media, such as a ROM, an optical disk, or a hard disk drive, and when the software is executed by a processing device (a processor), the processing device (the processor) and peripheral devices execute the functions specified by the software. A system or a device may include one or more non-transitory recording media on which the software is recorded, a processing device (a processor), and necessary hardware devices, for example, an interface.

A more specific embodiment of the present disclosure will be described below with reference to the accompanying drawings. However, an overly detailed description may be omitted. For example, a detailed description of already well-known things and a redundant description of substantially the same configurations may be omitted herein. This is to avoid the following description becoming overly redundant and to facilitate understanding of those skilled in the art. The inventors provide the accompanying drawings and the following description in order for those skilled in the art to fully understand the present disclosure, and these drawings and description are not intended to limit the subject matter as recited in the claims. In the following description, the same or similar constituent elements are denoted by the same reference numerals.

Embodiment

<Light Detecting System>

First, a light detecting system in the present embodiment will be described.

FIG. 1 is a diagram schematically showing a light detecting system 400 in an exemplary embodiment. The light detecting system 400 includes an optical system 40, a filter array 100C, an image sensor 60, and a signal processing circuit 200. The filter array 100C has functions that are similar to those of the “encoding element” disclosed in U.S. Patent Application Publication No. 2016/138975. Thus, the filter array 100C can also be referred to as an “encoding element”. The optical system 40 and the filter array 100C are disposed on an optical path of light that is incident from a target object 70.

The filter array 100C includes a plurality of light-transmissive areas arrayed in a row or column. The filter array 100C is an optical element in which the light transmission spectrum, that is, the wavelength dependency of the light transmittance, differs depending on the area. The filter array 100C modulates the intensity of incident light and passes the modulated incident light. The filter array 100C may be disposed in the vicinity of or directly above the image sensor 60. The “vicinity” as used herein means being in close proximity to a degree that an optical image from the optical system 40 is formed on a plane of the filter array 100C in a clear state to some degree. The “directly above” means that both are in close proximity to each other to a degree that almost no gap occurs therebetween. The filter array 100C and the image sensor 60 may be integrated together. A device including the filter array 100C and the image sensor 60 is referred to as a “light detecting device 300”.

The optical system 40 includes at least one lens. Although the optical system 40 is shown as one lens in FIG. 1, the optical system 40 may be constituted by a combination of a plurality of lens. The optical system 40 forms an image on an imaging plane of the image sensor 60 via the filter array 100C.

Based on an image 120 acquired by the image sensor 60, the signal processing circuit 200 reconstructs a plurality of separate images 220 including multi-wavelength information. Details of the plurality of separate images 220 and a processing method for image signals in the signal processing circuit 200 are described later. The signal processing circuit 200 may be incorporated into the light detecting device 300 or may be a constituent element of a signal processing device electrically connected to the light detecting device 300 by wire or wirelessly.

<Filter Array>

The filter array 100C in the present embodiment will be described below. The filter array 100C is used in a spectroscopy system that generates images for respective wavelength regions included in a wavelength region to be imaged. Herein, the wavelength region to be imaged may be referred to as a “target wavelength region”. The filter array 100C is disposed on an optical path of light that is incident from a target object, modulates the intensity of the incident light for respective wavelengths, and outputs the modulated light. This process involving the filter array 100C, that is, the encoding element, is herein referred to as “encoding”.

FIG. 2A is a view schematically showing an example of the filter array 100C. The filter array 100C has a plurality of areas that is two-dimensionally arrayed. The areas may herein be referred to as “cells”. A filter having a transmission spectrum that is individually set is disposed in each area. The transmission spectrum is represented by a function T (λ), where λ is the wavelength of incident light. The transmission spectrum T (λ) can take a value that is greater than or equal to 0 and is less than or equal to 1. Details of the configuration of the filters are described below.

In the example shown in FIG. 2A, the filter array 100C has 48 rectangular areas arrayed in six rows by eight columns. This is merely exemplary, and a larger number of areas than that number can be provided in actual applications. The number of areas can be, for example, approximately the same as the number of pixels in a typical photodetector, such as an image sensor. The number of pixels is, for example, a few hundred thousand to tens of millions. In one example, the filter array 100C may be disposed directly above a photodetector, and each area may be disposed so as to correspond to one pixel in the photodetector. Each area faces, for example, one pixel in the photodetector.

FIG. 2B is a view showing one example of spatial distributions of transmittances of light in respective wavelength regions W1, W2, . . . , and Wi included in a target wavelength region. In the example shown in FIG. 2B, differences in darkness/lightness in the areas represent differences in the transmittances. The lighter the area is, the higher the transmittance is, and the darker the area is, the lower the transmittance is. As shown in FIG. 2B, the spatial distribution of the light transmittances differs depending on the wavelength region.

FIG. 2C and FIG. 2D are graphs showing examples of transmission spectra in area A1 and area A2, respectively, included in the areas in the filter array 100C shown in FIG. 2A. The transmission spectrum in area A1 and the transmission spectrum in area A2 differ from each other. Thus, the transmission spectrum of the filter array 100C differs depending on the area. However, the transmission spectra in all the areas do not necessarily have to differ from each other. In the filter array 100C, the transmission spectra of at least some of the areas are different from each other. The at least some of the areas are two or more areas. That is, the filter array 100C includes two or more filters whose transmission spectra are different from each other. In one example, the number of patterns of the transmission spectra of the areas included in the filter array 100C can be greater than or equal to the number i of wavelength regions included in the target wavelength region. The filter array 100C may be designed such that the transmission spectra of more than half of the areas are different.

FIGS. 3A and 3B are diagrams for describing a relationship between a target wavelength region W and the wavelength regions W1, W2, . . . , and Wi included therein. The target wavelength region W can be set to various ranges, depending on the application. The target wavelength region W can be, for example, a visible-light wavelength region of about 400 nm to about 700 nm, a near-infrared wavelength region of about 700 nm to about 2500 nm, and a near-ultraviolet wavelength region of about 10 nm to about 400 nm, or radio wave ranges of mid infrared, far infrared, terahertz waves, millimeter waves, or the like. Thus, the wavelength region that is used is not limited to a visible light region. Herein, not only visible light but also non-visible light, such as near-ultraviolet, near-infrared, and radio waves, is referred to as “light”, for the sake of convenience”.

In the example shown in FIG. 3A, i wavelength regions obtained by equally dividing the target wavelength region W are referred to as “wavelength regions W1, W2, . . . , and Wi”, where i is an arbitrary integer greater than or equal to 4. However, the present disclosure is not limited to such an example. The plurality of wavelength regions included in the target wavelength region W may be arbitrarily set. For example, the bandwidths may be made unequal depending on the wavelength region. A gap may be present between the adjacent wavelength regions. In the example shown in FIG. 3B, the bandwidth differs depending on the wavelength region, and a gap exists between two adjacent wavelength regions. In such a manner, it is sufficient as long as the wavelength regions be different from each other, and how they are determined is arbitrary. The number i of divided wavelengths may be less than or equal to 3.

FIG. 4A is a graph for describing characteristics of a transmission spectrum in a certain area in the filter array 100C. In the example shown in FIG. 4A, the transmission spectrum has a plurality of local maximum values P1 to P5 and a plurality of local minimum values with respect to the wavelengths in the target wavelength region W. In the example shown in FIG. 4A, normalization is performed so that the maximum value of the light transmittance in the target wavelength region W is 1, and the minimum value thereof is 0. In the example shown in FIG. 4A, the transmission spectrum has the local maximum values in wavelength regions, such as wavelength regions W2 and Wi−1. Thus, in the present embodiment, the transmission spectrum in each area has the local maximum values at at least two wavelengths of the wavelength regions W1 to Wi. As can be seen from FIG. 4A, the local maximum values P1, P3, P4, and P5 are greater than or equal to 0.5.

As described above, the light transmittance in each area differs depending on the wavelength. Thus, the filter array 100C passes a large amount of components in one wavelength region of incident light and does not pass much components in other wavelength regions of the incident light. For example, the transmittances for light in k wavelength regions of the i wavelength regions can be larger than 0.5, and the transmittances for light in the remaining i-k wavelength regions can be smaller than 0.5, where k is an integer that satisfies 2≤k<i. If incident light is white light equally including wavelength components of all visible light, the filter array 100C modulates, for each area, the incident light into light having a plurality of discrete intensity peaks with respect to the wavelength, superimposes the light having multi wavelengths, and outputs the light.

FIG. 4B is a graph showing one example of a result of averaging the transmittances shown in FIG. 4A for each of wavelength regions W1, W2, . . . , and Wi. The averaged transmittance is obtained by integrating the transmission spectrum T (λ) for each wavelength region and dividing the result thereof by the bandwidth of the wavelength region. The value of the transmittance averaged for each wavelength region in such a manner will herein be referred to as a “transmittance” in the wavelength region. In this example, in three wavelength regions in which the local maximum values P1, P3, and P5 are reached, the transmittances are prominently high. In particular, in two wavelength regions where the local maximum values P3 and P5 are reached, the transmittances exceed 0.8.

The resolution in a wavelength direction of the transmission spectrum in each area can be set to approximately the bandwidth of a desired wavelength region. In other words, in a wavelength range including one local maximum value on a transmission spectrum curve, the width of a range that takes values that are greater than or equal to the average value of the local maximum value and a local minimum value that is the most adjacent to the local maximum value can be set to approximately the bandwidth of a desired wavelength region. In this case, when the transmission spectrum is resolved into frequency components, for example, by a Fourier transform, the values of frequency components corresponding to the wavelength region increase relatively.

The filter array 100C is typically divided into a plurality of cells sectioned into a lattice form, as shown in FIG. 2A. These cells have transmission spectra that are different from each other. The wavelength distribution and the spatial distribution of the light transmittances of the areas in the filter array 100C can be, for example, random distributions or pseudo-random distributions.

The concepts of the random distribution and the pseudo-random distribution are as follows. First, each area in the filter array 100C can be regarded as, for example, a vector element having a value of 0 to 1 according to the light transmittance. In this case, when the transmittance is 0, the value of the vector element is 0, and when the transmittance is 1, the value of the vector element is 1. In other words, a collection of areas that are arranged in one line in a row direction or a column direction can be regarded as a multidimensional vector having values of 0 to 1. Accordingly, the filter array 100C can be said to include a plurality of multidimensional vectors in the column direction or the row direction. In this case, the random distribution means that two arbitrary multidimensional vectors are independent from each other, that is, are not parallel to each other. Also, the pseudo-random distribution means that a non-independent configuration is included between some multidimensional vectors. Accordingly, in the random distribution and the pseudo-random distribution, a vector whose elements are the values of the transmittances of light in a first wavelength region in the areas belonging to a collection of the areas arranged in one row or column included in the plurality of areas and a vector whose elements are the values of the transmittances of light in the first wavelength region in the areas belonging to a collection of the areas arranged in another row or column are independent from each other. In a second wavelength region that is different from the first wavelength region, similarly, a vector whose elements are the values of the transmittances of light in the second wavelength region in the areas belonging to a collection of the areas arranged in one row or column included in the plurality of areas and a vector whose elements are the values of the transmittances of light in the second wavelength region in the areas belonging to a collection of the areas arranged in another row or column are intendent from each other.

When the filter array 100C is disposed in the vicinity of or directly above the image sensor 60, a cell pitch, which is the interval between the areas in the filter array 100C, may be made to generally match the pixel pitch of the image sensor 60. With this arrangement, the resolution of an encoded optical image emitted from the filter array 100C generally matches the resolution of the pixels. When light that passes through each cell is adapted to be incident on only one corresponding pixel, it is possible to easily perform an arithmetic operation described below. When the filter array 100C is disposed away from the image sensor 60, the cell pitch may be reduced according to the distance therebetween.

In the example shown in FIGS. 2A to 2D, a grayscale transmittance distribution in which the transmittance of each area can take a value that is greater than or equal to 0 and is less than or equal to 1 is envisaged. However, the transmittance distribution does not necessarily have to be made to be a grayscale transmittance distribution. For example, a binary-scale transmittance distribution in which the transmittance of each area can take a value of either generally 0 or generally 1 may be employed. In the binary-scale transmittance distribution, each area passes a majority of light in at least two wavelength regions of wavelength regions included in the target wavelength region and does not pass a majority of light in the remaining wavelength regions. The “majority” herein refers to about 80% or more.

Some of all the cells, for example, half of the cells, may be replaced with transparent areas. Such transparent areas pass light in all the wavelength regions W1 to Wi included in the target wavelength region at approximately the same high transmittance. The high transmittance is, for example, greater than or equal to 0.8. In such a configuration, the transparent areas can be disposed, for example, in a checkered pattern. That is, in two array directions of the areas in the filter array 100C, the area in which the light transmittance differs depending on the wavelength and the transparent area can be alternately arrayed. In the example shown in FIG. 2A, the two array directions are a horizontal direction and a vertical direction.

<Signal Processing Circuit>

Next, a description will be given of a method in which the signal processing circuit 200 shown in FIG. 1 reconstructs multi-wavelength separate images 220 based on the image 120 and spatial distribution characteristics of transmittances for respective wavelengths in the filter array 100C. The “multi wavelengths” as used herein means a larger number of wavelength regions than the wavelength regions in three colors of RGB obtained by, for example, an ordinary color camera. The number of wavelength regions can be, for example, about 4 to 100. The number of wavelength regions may be referred to as the “number of spectral bands”. The number of spectral bands may exceed 100, depending on the application.

Data to be acquired are separate images 220, and the data are represented as f. When the number of spectral bands is represented as w, f is data obtained by integrating pieces of image data f₁, f₂, . . . , and f_w in the individual bands. When the number of picture elements in an x-direction of the image data to be acquired is represented as n, and the number of picture elements in a y-direction thereof is represented as m, each of the pieces of image data f₁, f₂, . . . , and f_w is a collection of pieces of two-dimensional data for n×m picture elements. Accordingly, the data f is three-dimensional data in which the number of elements is n×m×w. Meanwhile, the number of elements in data g of the image 120 acquired through encoding and multiplexing performed by the filter array 100C is nxm. The data g in the present embodiment can be given by expression (1):

$\begin{array}{cc}g=\mathrm{Hf}=H\left[\begin{array}{c}{f}_1\\ f_2\\ ⋮\\ f_w\end{array}\right]& \left(1\right)\end{array}$

In this case, f₁, f₂, . . . , f_w are each a piece of data having n×m elements. Thus, the vector on the right-hand side is, strictly speaking, a one-dimensional vector for n×m×w rows by one column. The vector g is converted into and expressed by a one-dimensional vector for n×m rows by one column, and the one-dimensional vector is calculated. A matrix H represents conversion for encoding and intensity-modulating the components f₁, f₂, . . . , and f_w of the vector f with pieces of encoding information that are different for each wavelength region and adding resulting components. Thus, H is a matrix with n×m rows by n×m×w columns.

When the vector g and the matrix H are given, it seems that f can be determined by solving an inverse problem of expression (1). However, since the number n×m×w of elements in the data f to be determined is larger than the number n×m of elements in the acquired data g, this problem is an ill-posed problem and cannot be directly solved. Hence, the signal processing circuit 200 in the present embodiment utilizes redundancy of images included in the data f to determine the solution by using a scheme for compressive sensing. Specifically, the data f to be determined is estimated by solving expression (2):

$\begin{array}{cc}{f}^{\prime }=\underset{f}{\arg \min }\left\{{g-\mathrm{Hf}}_{l_2}+\mathrm{\tau \Phi }\left(f\right)\right\}& \left(2\right)\end{array}$

In this case, f′ represents data of estimated f. The first term in the braces in the above equation represents the amount of deviation between an estimation result Hf and the acquired data g, that is, the so-called residual term. Although the sum of squares is used as the residual term, an absolute value, the square root of the sum of squares, or the like may be used as the residual term. The second term in the braces is a regularization term or a stabilization term described below. Expression (2) means determining f that minimizes the sum of the first term and the second term. The signal processing circuit 200 can calculate the final solution f′ by converging solutions through a recursive iteration arithmetic operation.

The first term in the braces in expression (2) means an arithmetic operation for determining the sum of squares of differences between the acquired data g and Hf obtained by system-converting f in an estimation process by using the matrix H. Φ(f) in the second term is a constraint condition in regularization of f and is a function that reflects sparse information of the estimated data. Φ(f) serves to provide an effect of smoothing or stabilizing the estimated data. The regularization term can be represented by, for example, a discrete cosine transform (DCT), a wavelet transform, a Fourier transform, total variation (TV), or the like of f. For example, when total variation is used, it is possible to acquire stable estimated data with reduced influences of noise in the observed data g. The sparseness of the target object 70 in the space of each regularization term differs depending on the texture of the target object 70. A regularization term with which the texture of the target object 70 becomes sparser in the space of the regularization term may be selected. Alternatively, the arithmetic operation may include a plurality of regularization terms. In expression (2), τ is a weighting factor. As the weighting factor t increases, the amount of reduction in redundant data increases, and the rate of compression increases. As the weighting factor τ decreases, the convergence to the solution becomes weak. The weighting factor τ is set to an appropriate value at which f converges to some degree and no over-compression occurs.

Although an arithmetic operation example using the compressive sensing indicated in expression (2) has been described above, another method may be used to determine the solution. For example, another statistical method, such as a maximum likelihood estimation method or a Bayesian estimation method, can be used. Also, the number of separate images 220 is arbitrary, and each wavelength region may also be arbitrarily set. Details of the method for the reconstruction are disclosed in U.S. Patent Application Publication No. 2016/138975. The entire contents disclosed in U.S. Patent Application Publication No. 2016/138975 are hereby incorporated by reference.

<Filter Array Including Fabry-Perot Filters>

Next, a description will be given of an example of a more specific structure of the filter array 100C.

FIG. 5 is a sectional view schematically showing the light detecting device 300 in the exemplary embodiment. The light detecting device 300 includes the filter array 100C and the image sensor 60.

The filter array 100C includes a plurality of filters 100 that is two-dimensionally arrayed. The filters 100 are arrayed in a row and a column, for example, as shown in FIG. 2A. FIG. 5 schematically shows a cross-sectional structure of one of the rows shown in FIG. 2A. Each of the filters 100 has a resonance structure. The resonance structure means a structure in which light with a certain wavelength forms a standing wave and exists stably. The state of the light may be referred to as a “resonant mode”. The resonance structure shown in FIG. 5 includes a first reflective layer 28a, a second reflective layer 28b, and an intermediate layer 26 between the first reflective layer 28a and the second reflective layer 28b. The first reflective layer 28a and/or the second reflective layer 28b can be formed of a dielectric multilayer film or a metal thin film. The intermediate layer 26 can be formed of a dielectric or semiconductor that is transparent in a specific wavelength region. The intermediate layer 26 can be formed from, for example, at least one selected from the group consisting of silicon (Si), silicon nitride (Si₃N₄), titanium dioxide (TiO₂), niobium pentoxide (Nb₂O₅), and tantalum pentoxide (Ta₂O₅). The refractive index and/or the thickness of the intermediate layer 26 in the filters 100 differ depending on the filter. The transmission spectrum of each of the filters 100 has local maximum values of transmittance at a plurality of wavelengths. The wavelengths respectively correspond to resonant modes whose orders are different from each other in the above-described resonance structure. In the present embodiment, all the filters 100 in the filter array 100C include the above-described resonance structures. The filter array 100C may include filters that do not have the above-described resonance structures. For example, the filter array 100C may include filters, such as transparent filters or neutral density (ND) filters, that do not have wavelength dependency of the light transmittance. In the present disclosure, two or more of the filters 100 each has the above-described resonance structure.

The image sensor 60 includes a plurality of light detection elements 60a. Each of the light detection elements 60a is disposed to face one of the plurality of filters 100. Each of the light detection elements 60a has sensitivity to light in a specific wavelength region. This specific wavelength region corresponds to the above-described target wavelength region W. In the present disclosure, “having sensitivity to light in a certain wavelength region” refers to having substantial sensitivity needed to detect light in the wavelength region. For example, it means that an external quantum efficiency in the wavelength region is 1% or more. An external quantum efficiency of the light detection elements 60a may be 10% or more. The external quantum efficiency of the light detection elements 60a may be 20% or more. The wavelengths at which the light transmittance of each filter 100 takes the local maximum values are all included in the target wavelength region W. The light detection elements 60a may be referred to as “pixels” in the description below.

Besides the example shown in FIG. 5, the filter array 100C and the image sensor 60 may be separated from each other. Even in such a case, each of the light detection elements 60a is disposed at a position where it receives light that has passed through one of the filters. Constituent elements may be disposed so that light that have passed through the filters are incident on the light detection elements 60a, respectively, through a mirror. In this case, each of the light detection elements 60a is not disposed directly below one of the filters.

Herein, the filters 100 having the above-described resonance structures may be referred to as “Fabry-Perot filters”. Herein, the portion of a transmission spectrum having a local maximum value may be referred to as a “peak”, and a wavelength at which the transmission spectrum has a local maximum value may be referred to as a “peak wavelength”.

Next, a description will be given of the transmission spectra of the filters 100, which are Fabry-Perot filters.

In this case, a peak wavelength λ_m of the transmission spectrum of the filter 100 is given by:

$\begin{array}{cc}{\lambda }_m=\frac{2\mathrm{nL}}{m}\sqrt{1-{\left(\frac{\sin {\theta }_i}{n}\right)}^2}& \left(3\right)\end{array}$

where, in each filter 100, L represents the thickness of the intermediate layer 26, n represents the refractive index, θ_i represents the incident angle of light that is incident on the filter 100, and m represents the mode order of the resonant mode and is an integer greater than or equal to 1.

A shortest wavelength in the target wavelength region W is represented by hui, and a longest wavelength therein is represented by λ_e. Herein, the filter 100 with which the number of m's that satisfy λ_i≤λ_mλ_e is one is referred to as a “single-mode filter”. The filter 100 with which the number of m's that satisfy λ_i≤λ_mλ_e is two or more is referred to as a “multi-mode filter”. An example when the shortest wavelength in the target wavelength region W is given by λ_i=400 nm, and the longest wavelength therein is given by λ_e=700 nm will be described below.

For example, in the filter 100 with the thickness L=300 nm, the refractive index n=1.0, and the orthogonal incidence θ_i=0°, the peak wavelength for m=1 is λ₁=600 nm, and the peak wavelength for m≥2 is λ_m≥2≤300 nm. Accordingly, this filter 100 is a single-mode filter with which one peak wavelength is included in the target wavelength region W.

On the other hand, when the thickness L is increased to a thickness larger than 300 nm, a plurality of peak wavelengths is included in the target wavelength region W. For example, in the filter 100 with the thickness L=3000 nm, n=1.0, and the orthogonal incidence θ_i=0°, the peak wavelength for 1≤m≤8 is λ_1≤m≤8≤750 nm, the peak wavelength for 9≤m≤15 is 400 nm≤λ_9≤m≤15<700 nm, and the peak wavelength for m≥16 is λ_m≥16≤375 nm. Accordingly, this filter 100 is a multi-mode filter with which seven peak wavelengths are included in the target wavelength region W.

In the above-described filter 100, although seven peak wavelengths are included in the target wavelength region W, spectroscopic information of a plurality of wavelengths can be obtained when the number of peak wavelengths is two or more in the target wavelength region W. In particular, when the number of peak wavelengths in the target wavelength region W is four or more when the target wavelength region W is greater than or equal to 400 nm and is less than or equal to 700 nm, a number of pieces of spectroscopic information, the number surpassing three wavelengths of RGB, can be obtained in the visible light region.

The number of peak wavelengths included in the target wavelength region W can be controlled with the thickness of the intermediate layer 26. For the orthogonal incidence θ_i=0°, the peak wavelength λ_m of the transmission spectrum of the filter 100 is given by:

$\begin{array}{cc}{\lambda }_m=\frac{2\mathrm{nL}}{m}& \left(4\right)\end{array}$

In this case, considering the wavelength dispersion of the refractive index in order to more strictly calculate the thickness of the intermediate layer 26, the refractive index of the intermediate layer 26 with respect to light with a wavelength of 700 nm is represented by n₁, and the refractive index of the intermediate layer 26 with respect to light with a wavelength of 400 nm is represented by n₂. A condition for two wavelength peaks to exist in the target wavelength region W that is greater than or equal to 400 nm and is less than or equal to 700 nm is that a lowest-order peak wavelength λ_m at the long wavelength side satisfies expression (5), and a highest-order peak wavelength λ_m+1 at the short wavelength side satisfies expression (6). In this case, m is an integer greater than or equal to 1.

$\begin{array}{cc}{\lambda }_m=\frac{2n_{1}L}{m}\le 700& \left(5\right)\end{array}$ $\begin{array}{cc}{\lambda }_{m+1}=\frac{2n_{2}L}{m+1}\ge 400& \left(6\right)\end{array}$

When expressions (5) and (6) are modified, expressions (7) and (8) below are obtained respectively.

$\begin{array}{cc}{n}_{1}L\le 350m& \left(7\right)\end{array}$ $\begin{array}{cc}{n}_{2}L\ge 200\left(m+1\right)& \left(8\right)\end{array}$

Existence of m that satisfies both expressions (7) and (8) is a condition for two wavelength peaks to be included in the target wavelength region W that is greater than or equal to 400 nm and is less than or equal to 700 nm.

Similarly, a condition for four wavelength peaks to exist in the target wavelength region W is that the lowest-order peak wavelength λ_m at the long wavelength side satisfies expression (9), and a highest-order peak wavelength λ_m+3 at the short wavelength side satisfies expression (10). In this case, m is an integer greater than or equal to 1.

$\begin{array}{cc}{\lambda }_m=\frac{2n_{1}L}{m}\le 700& \left(9\right)\end{array}$ $\begin{array}{cc}{\lambda }_{m+3}=\frac{2n_{2}L}{m+3}\ge 400& \left(10\right)\end{array}$

When expressions (9) and (10) are modified, expressions (11) and (12) below are obtained respectively.

$\begin{array}{cc}{n}_{1}L\le 350m& \left(11\right)\end{array}$ $\begin{array}{cc}{n}_{2}L\ge 200\left(m+3\right)& \left(12\right)\end{array}$

Existence of m that satisfies both expressions (11) and (12) is a condition for four wavelength peaks to be included in the target wavelength region W that is greater than or equal to 400 nm and is less than or equal to 700 nm.

Similarly, a condition for six wavelength peaks to exist in the target wavelength region W is that the lowest-order peak wavelength λ_m at the long wavelength side satisfies expression (13), and the highest-order peak wavelength λ_m+5 at the short wavelength side satisfies expression (14). In this case, m is an integer greater than or equal to 1.

$\begin{array}{cc}{\lambda }_m=\frac{2n_{1}L}{m}\le 700& \left(13\right)\end{array}$ $\begin{array}{cc}{\lambda }_{m+5}=\frac{2n_{2}L}{m+5}\ge 400& \left(14\right)\end{array}$

When expressions (13) and (14) are modified, expressions (15) and (16) below are obtained respectively.

$\begin{array}{cc}{n}_{1}L\le 350m& \left(15\right)\end{array}$ $\begin{array}{cc}{n}_{2}L\ge 200\left(m+5\right)& \left(16\right)\end{array}$

Existence of m that satisfies both expressions (15) and (16) is a condition for six wavelength peaks to be included in the target wavelength region W that is greater than or equal to 400 nm and is less than or equal to 700 nm.

Appropriately designing the thickness of the intermediate layer 26 in the filter 100, as described above, makes it possible to realize a multi-mode filter. The refractive index of the intermediate layer 26, instead of the thickness of the intermediate layer 26, in the filter 100 may be appropriately designed. Alternatively, both the thickness and the refractive index of the intermediate layer 26 in the filter 100 may be appropriately designed.

FIG. 6 is a graph schematically showing an example of transmission spectra at pixels when multi-mode filters whose transmission spectra are different from each other are respectively disposed above the pixels, which are the light detection elements 60a. FIG. 6 illustrates transmission spectra at pixels A, B, and C. The multi-mode filters are designed so that the peak wavelengths are slightly different from one pixel to another. Such a design can be realized by slightly varying the thickness L and/or the refractive index n in expression (3). In this case, in each pixel, a plurality of peaks appears in the target wavelength region W. The mode orders of the respective peaks are the same among the pixels. The mode orders of the respective peaks shown in FIG. 6 are m, m+1, and m+2. The light detecting device 300 in the present embodiment can simultaneously detect light with a plurality of peak wavelengths that differ from one pixel to another.

FIG. 7 is a graph showing one example of a calculation result of transmission spectra of the filters 100. In this example, each of the first reflective layer 28a and the second reflective layer 28b in each filter 100 is formed of a dielectric multilayer film in which a TiO₂ layer and a silicon dioxide (SiO₂) layer are alternately stacked. The intermediate layer 26 in the filter 100 is formed of a TiO₂ layer. In the example shown in FIG. 7, the thickness of the intermediate layer 26 corresponding to the transmission spectrum denoted by a solid line is different from the thickness of the intermediate layer 26 corresponding to the transmission spectrum denoted by a dotted line. DiffractMOD based on Rigorous Coupled-Wave Analysis (RCWA) of Synopsys Inc. (formerly, RSoft Inc.) was used for calculating the transmission spectra. As shown in FIG. 7, the plurality of peak wavelengths in the target wavelength region W differs depending on the pixel. As described above, changing the thickness of the intermediate layer 26 in the multi-mode filter 100 for each pixel allows the light detecting device 300 in the present embodiment to simultaneously detect light with a plurality of peak wavelengths that differ for each pixel.

In the example of the transmission spectra shown in FIG. 7, each of the transmission spectra of a first filter and a second filter has skirt portions that extend in a wavelength-reducing direction and a wavelength-increasing direction from each of the local maximum values of transmittance, and the wavelength region of the skirt portion for the first filter and the wavelength region of the skirt portion for the second filter at least partly overlap each other.

Next, a plurality of separate images 220 reconstructed by a plurality of multi-mode filters will be described in comparison with a plurality of separate images 220 reconstructed by a plurality of single-mode filters.

FIG. 8A has graphs showing respective transmission spectra of nine types of multi-mode filter. FIG. 8B has graphs showing respective transmission spectra of nine types of single-mode filter. In the example shown in FIG. 8A, the transmission spectrum of each multi-mode filter exhibits eight or nine peaks in the target wavelength region W. In the example shown in FIG. 8B, the transmission spectrum of each single-mode filter exhibits one peak in the target wavelength region W. The filter array 100C includes, for example, one million filters 100 that are two-dimensionally arrayed. The million filters 100 randomly include nine types of multi-mode filter shown in FIG. 8A. Alternatively, the million filters 100 randomly includes nine types of single-mode filter shown in FIG. 8B. Because of the random arrangement, adjacent filters may be of the same type of filter. Such cases, however, are thought to be rare. Accordingly, a major problem does not occur.

FIG. 8C is a view showing original images and two examples of a plurality of reconstructed separate images 220. The upper stage in FIG. 8C shows original images. The middle stage in FIG. 8C shows an example of a plurality of separate images 220 reconstructed via the nine types of multi-mode filter shown in FIG. 8A. The lower stage in FIG. 8C shows an example of a plurality of separate images 220 reconstructed via the nine types of single-mode filter shown in FIG. 8B. Thirty images in each of the upper stage, the middle stage, and the lower stage were respectively acquired by detecting light in 30 wavelength regions. The 30 wavelength regions were obtained by equally dividing the target wavelength region W from 400 nm to 700 nm into 30 regions in increments of 10 nm. For example, the first, second, and third images from the top left in each of the upper stage, the middle stage, and the lower stage were obtained by respectively detecting light in a wavelength region from 400 nm to 410 nm, light in a wavelength region from 410 nm to 420 nm, and light in a wavelength region from 420 nm to 430 nm, respectively. In the example shown in FIG. 8C, the images in the lower stage are darker than the images in the middle stage. This is thought to be because the amount of light that passed through the single-mode filters is smaller than the amount of light that passed through the multi-mode filters.

FIG. 8D is a graph showing a calculation result of mean squared errors (MSEs) between the original image and each reconstructed separate image 220. The mean squared errors are calculated using expression (17):

$\begin{array}{cc}\mathrm{MSE}=\frac{1}{N·M}\sum _{i=1}^N\sum _{j=1}^M{\left(I_{i,j}^{\prime }-I_{i,j}\right)}^2& \left(17\right)\end{array}$

In this case, N and M are the number of picture elements in a vertical direction and the number of picture elements in a lateral direction, respectively. l_i,j is a picture element value of the original image at the picture element at position (i,j). l′_i,j is a picture element value in each reconstructed separate image 220 at the picture element at position (i,j).

As shown in FIG. 8D, the MSE between the original image and each separate image 220 reconstructed via the multi-mode filters is sufficiently smaller than the MSE between the original image and each separate image 220 reconstructed via the single-mode filters. Accordingly, in the light detecting device 300 in the present embodiment, the original images can be accurately reproduced with the plurality of separate images 220 reconstructed via the multi-mode filters. The original images cannot be accurately reproduced with the plurality of separate images 220 reconstructed via the single-mode filters.

As described above, in the light detecting system 400 in the present embodiment, the signal processing circuit 200 shown in FIG. 1 generates image data including information of a plurality of wavelengths, based on signals output from the plurality of pixels. The image data includes data representing the separate images 220 spectrally separated for the respective wavelengths.

The number of picture elements in each of the separate images 220 is larger than M/N, where N presents the number of separate images 220, and M represents the number of pixels. In the example shown in the middle stage in FIG. 8C, the number of picture elements is equal to M. Thus, even when the images are spectrally separated for the respective wavelengths, the light detecting system 400 in the present embodiment can suppress a reduction in the resolution of each of the plurality of separate images 220.

As described above, each of the transmission spectra of the first filter and the second filter of the plurality of multi-mode filters included in the filter array 100C in the present embodiment has skirt portions that respectively extend in a wavelength-reducing direction and a wavelength-increasing direction from each of the plurality of local maximum values of transmittance, and the wavelength region of the skirt portion for the first filter and the wavelength region of the skirt portion for the second filter may overlap each other. FIG. 8E is a graph showing the transmission spectra of the upper-left two filters in FIG. 8A, the transmission spectra being overlayed for ease of understanding, and FIG. 8F is a graph obtained by extracting the wavelength region of 550 nm to 650 nm in FIG. 8E. Each peak is constituted by a local maximum value and skirt portions that extend at both ends of the local maximum value, and wavelength regions overlap each other at the skirt portion of one filter and the skirt portion of another filter. The transmittances of the two filters are the same at the wavelength corresponding to the intersection of two skirt portions indicated by an arrow in FIG. 8F.

With such a configuration, at least some of the pixels in the image sensor 60 disposed directly below the filter array 100C have an overlap in the wavelength regions in which light is received, and these pixels share wavelength information from a target object. Meanwhile, when peaks in the plurality of multi-mode filters included in the filter array 100C do not match each other, and all the pixels receive light having wavelengths that are different from each other, all the pixels obtain pieces of wavelength information that are different from each other. That is, in the former case, since information of correlation between the pixels that share the wavelength information, in addition to the pieces of wavelength information of the pixels, can also be utilized to reconstruct the separate images, there is an advantage that the original images can be accurately reproduced.

For example, suppose three pixels are arranged in one line, and two pixels at both ends thereof share the wavelength information from a target object. In general, since three pixels are physically close in distance, it can be expected that the wavelength of light received by the middle pixel located between the two pixels is the same as the wavelength of light received by the two pixels at both ends. That is, based on the information of correlation between the pixels that share the wavelength information, wavelength information of another pixel can be presumed, and the original images can be accurately reproduced.

Next, a description will be given of an influence that the refractive indices of the intermediate layers 26 in the filters 100 have on the wavelength resolution of the light detecting device 300.

As described above, the filters 100 in the filter array 100C include the intermediate layers 26 having thicknesses that are different from each other, as shown in FIG. 5. FIG. 9 is a diagram schematically showing an example of two filters 100 whose thicknesses of the intermediate layers 26 are the closest in the filter array 100C. In the example shown in FIG. 9, the filter 100 in which the thickness of the intermediate layer 26 is L and the filter 100 in which the thickness of the intermediate layer 26 is L+ΔL are disposed adjacent to each other. The pixel for which the filter 100 in which the thickness of the intermediate layer 26 is L is disposed and the pixel for which the filter 100 in which the thickness of the intermediate layer 26 is L+ΔL is disposed are referred to as “pixel A” and “pixel B”, respectively. In the filter array 100C, even when two filters 100 shown in FIG. 9 are disposed away from each other, the following discussion holds true.

In the example shown in FIG. 9, the light detecting device 300 includes the optical system 40. When an optical image from a target object passes through the center of the lens included in the optical system 40, the optical image is orthogonally incident on the filter 100. On the other hand, when an optical image from a target object passes through a portion other than the center of the lens included in the optical system 40, the optical image is obliquely incident on the filter 100 at a finite incident angle. The finite incident angle is determined by a numerical aperture NA of the optical system 40. That is, the minimum value of the incident angle of light that is incident on the filter array 100C from the optical system 40 is 0°, and the maximum value of the incident angle is sin⁻¹ (NA)°. Accordingly, in the transmission spectrum of the filter 100, in accordance with expression (3), the peak wavelength shifts toward the short wavelength side as the incident angle θ_i increases.

FIG. 10 has graphs for describing the wavelengths of light detected by pixels A and B when light is orthogonally or obliquely incident on the filter 100. The upper stage in FIG. 10 shows peaks of light detected by pixels A and B in the case of orthogonal incidence. The middle stage in FIG. 10 shows peaks of light detected by pixels A and B in the case of oblique incidence. The lower stage in FIG. 10 shows peaks of light detected by pixel B in the case of oblique incidence. In the description below, the peak wavelength of the mode order m for light detected by pixel A is referred to as a “peak wavelength of pixel A”, and the peak wavelength of the mode order m for light detected by pixel B is referred to as a “peak wavelength of pixel B”. In the orthogonal incidence, the peak wavelength of pixel A is λ_A=2 nL/m, and the peak wavelength of pixel B is λ_B=2n (L+ΔL)/m. The difference ΔL between the thicknesses of the intermediate layers 26 in the two filters 100 shown in FIG. 9 is the smallest of combinations of two arbitrary filters 100 in the filter array 100C. Accordingly, an interval Δλ_ΔL=2nΔL/m of the peak wavelengths of pixels A and B is the smallest of the combinations of two arbitrary pixels in the image sensor 60. The interval of the peak wavelengths of pixels A and B corresponds to the wavelength resolution of the light detecting device 300.

On the other hand, in the oblique incidence, the peak wavelength shifts toward the short wavelength side. An amount Δλ_θi of shift of the peak wavelength in the oblique incidence is given by expression (18):

$\begin{array}{cc}{\mathrm{\Delta \lambda }}_{{\theta }_i}=\frac{2\mathrm{nL}}{m}\left[1-\sqrt{1-{\left(\frac{\sin {\theta }_i}{n}\right)}^2}\right]& \left(18\right)\end{array}$

Thus, for Δλ_θi≥Δλ_ΔL, there is a possibility that the peak wavelength of pixel B in the oblique incidence matches the peak wavelength of pixel A in the orthogonal incidence. In the example shown in FIG. 9, both light of the orthogonal incidence and light of the oblique incidence are simultaneously detected by pixel B. Accordingly, for Δλ_θi≥Δλ_ΔL, light that should be detected by pixel A is falsely detected by pixel B.

From the above discussion, it follows that a condition that the false detection does not occur is Δλ_θi<Δλ_ΔL. When Δλ_θi<Δλ_ΔL is modified, expression (19) below is obtained.

$\begin{array}{cc}\Delta L>L\left[1-\sqrt{1-{\left(\frac{\sin {\theta }_i}{n}\right)}^2}\right]& \left(19\right)\end{array}$

In addition, the condition that the false detection does not occur may be set to Δλ_θi<Δλ_ΔL/2. In the following description, the wavelength region in which light with a peak wavelength at the mode order m is detected by pixel A is referred to as “wavelength region A”, and the wavelength region in which light with a peak wavelength at the mode order m is detected by pixel B is referred to as “wavelength region B”. When the upper limit of wavelength region A and the lower limit of wavelength region B are both set to (λ_A+λ_B)/2=λ_A+Δλ_ΔL/2=λ_B−Δλ_ΔL/2, the peak wavelength of image B does not enter wavelength region A even for oblique incidence, because of Δλ_θi<Δλ_ΔL/2. This allows the signal processing circuit 200 to process a peak wavelength in wavelength region A as the peak wavelength of pixel A and to process a peak wavelength in wavelength region B as the peak wavelength of pixel B, regardless of the incident angle θ_i. As a result, occurrence of the false detection can be more reduced than in the case of expression (19). When Δλ_θi<Δλ_ΔL/2 is modified, expression (20) below is obtained.

$\begin{array}{cc}\Delta L>2L\left[1-\sqrt{1-{\left(\frac{\sin {\theta }_i}{n}\right)}^2}\right]& \left(20\right)\end{array}$

According to expressions (19) and (20), increasing the refractive index n of the intermediate layer 26 makes it possible to reduce the influences of the incident angle θ_i. As a result, it is possible to improve the wavelength resolution of the light detecting device 300.

FIG. 11A and FIG. 11B are graphs showing incident angle dependency of the transmission spectra of the filter 100 including the intermediate layer 26 with the refractive index n=1.5 and the filter 100 including the intermediate layer 26 with the refractive index n=2.35, respectively. In the example shown in FIG. 11A, when the incident angle θ_i changes from 0° to 30°, the peak wavelength shifts by 26.1 nm. In the example shown in FIG. 11B, when the incident angle changes from 0° to 30°, the peak wavelength shifts by 17.1 nm. That is, increasing the refractive index of the intermediate layer 26 makes it possible to reduce the amount of shift in the peak wavelength, the shift being caused by a change in the incident angle θ_i. Accordingly, appropriately designing the refractive index of the intermediate layer 26 in accordance with expressions (19) and (20) makes it possible to improve the wavelength resolution of the light detecting device 300.

A difference between the light detecting device 300 in the present embodiment and the device disclosed in U.S. Pat. No. 9,929,206 will be described next.

U.S. Pat. No. 9,929,206 discloses a device in which a plurality of single-mode filters is two-dimensionally arrayed. The peak wavelengths of the plurality of single-mode filters differ from one filter to another. When the number of peaks in the transmission spectra of the filter array in which the plurality of single-mode filters is two-dimensionally arrayed is represented by N, the spatial resolution of each separate image decreases to 1/N even when a plurality of separate images is reconstructed using the filter array. Accordingly, the original images cannot be accurately reproduced with the separate images, unlike the example shown in the middle stage in FIG. 8C. Thus, the device disclosed in U.S. Pat. No. 9,929,206 cannot provide an advantage that is analogous to that of the light detecting device 300 in the present embodiment.

Also, in the device disclosed in Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2015-501432, each of a plurality of sensors in a sensor array does not receive light with a plurality of wavelengths corresponding to peak wavelengths of a multi-mode filter and also does not reconstruct separate images 220 by using information of the plurality of wavelengths. Accordingly, the device disclosed in Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2015-501432 cannot provide an advantage that is analogous to that of the light detecting device 300 in the present embodiment.

Modifications of the light detecting device 300 shown in FIG. 5 will be described next.

FIGS. 12A to 12F are views schematically showing modifications of the light detecting device 300 shown in FIG. 5.

As shown in FIG. 12A, in the filter array 100C, the filters 100 may be divided. All the filters 100 do not have to be divided. Some of the filters 100 may be divided.

As shown in FIG. 12B, the filters 100 do not have to be disposed above some of the pixels. In other words, in the filter array 100C, at least one of the filters 100 may be transparent.

As shown in FIG. 12C, space may be provided between the filter array 100C and the image sensor 60. In other words, the filter array 100C and the image sensor 60 may be separated from each other with space being interposed therebetween.

As shown in FIG. 12D, one filter 100 may be disposed across a plurality of pixels. In other words, the intermediate layer 26 may be continuously provided across two or more pixels. The first reflective layer 28a and/or the second reflective layer 28b may be continuously provided across two or more pixels.

As shown in FIGS. 12E and 12F, a transparent layer 27 may be disposed to planarize level differences on the filter array 100C. In other words, the filter array 100C may further include a transparent layer 27 that planarizes level differences on two or more filters 100 having the above-described resonance structures. In the example shown in FIG. 12E, level differences exist at upper surfaces of the second reflective layers 28b in the filter array 100C. In the example shown in FIG. 12F, level differences exist at lower surfaces of the first reflective layers 28a in the filter array 100C. Planarizing level differences on two or more filters 100 by using the transparent layer 27 facilitates that another member is disposed on the transparent layer 27.

As shown in FIGS. 12E and 12F, a plurality of microlenses 40a may be disposed on the filter array 100C. Each of the microlenses 40a is disposed on one of the filters 100. In other words, the filter array 100C includes two or more microlenses 40a. Each of the two or more microlenses 40a is disposed on one of the two or more filters 100 having the above-described resonance structures. Concentrating incident light by using the two or more microlenses 40a makes it possible to efficiently detect light.

The light detecting device and the filter array in the present disclosure are useful for, for example, cameras and measurement equipment that acquire multi-wavelength two-dimensional images. The light detecting device and the filter array in the present disclosure can also be applied to sensing for living bodies, medical care, and beauty care, inspection systems for foreign matter and residual agricultural chemicals in food, remote sensing systems, vehicle-mounted sensing systems, and so on.

Claims

1. A light detecting device comprising: a filter array including filters that are arranged two-dimensionally; andan image sensor including light detection elements, whereinthe filters include a first filter and a second filter,each of the first filter and the second filter includes a first reflective layer, a second reflective layer, and an intermediate layer between the first reflective layer and the second reflective layer and has a resonance structure,each of the first filter and the second filter has a transmission spectrum such that first light passing through the first filter has first peaks as first local maximum values and second light passing through the second filter has second peaks as second local maximum values,the first peaks result from the resonance structure of the first filter,the second peaks result from the resonance structure of the second filter,each of the light detection elements is disposed at a position where each of the detection elements receives passing light that passes through at least one of the filters, the passing light having peaks as local maximum values, andeach of the light detection elements detects the peaks included in the passing light.

2. The light detecting device according to claim 1, wherein the image sensor detects light with wavelengths of λi to λe,the first local maximum values is observed within a wavelength range from λi to λe, andthe second local maximum values is observed within the wavelength range.

3. The light detecting device according to claim 2, wherein at least one m that satisfies both

4. The light detecting device according to claim 1, wherein one of the first peaks and one of the second peak at least partly overlap each other.

5. The light detecting device according to claim 1, wherein the second reflective layer is disposed between the intermediate layer and the image sensor, anda level difference between a surface of the second reflective layer of the first filter and a surface of the second reflective layer of the second filter exists.

6. The light detecting device according to claim 1, wherein each of the filters has a resonance structure.

7. The light detecting device according to claim 1, wherein the λi is 400 nm, and the λe is 700 nm.

8. The light detecting device according to claim 1, wherein the light detection elements includes a first light detection element and a second light detection element,the first light detection element is disposed at a first position where the first detection element receives the first light passing through the first filter,the second light detection element is disposed at a second position where the second detection element receives the second light passing through the first filter,the first light detection element detects the first peaks, andthe second light detection element detects the second peaks.

9. A system comprising: a light detecting device according to claim 1; anda signal processing circuit, whereinthe signal processing circuit generates, based on first information output from the image sensor, images, the images corresponding one-to-one to wavelength bands.

10. The system according to claim 9, wherein the signal processing circuit generates the images based on the first information and second information based on transmission spectra of the filters.