The invention relates to the field of multispectral imaging, in particular the invention relates to a method for configuring a multispectral filter for a multispectral sensor and the definition of its structure.
An image sensor is composed of a plurality of photosites arranged into a grid over the entire surface of the sensors. The photosites capture the intensity of the light to which they are exposed without distinction of the wavelengths of the light and therefore of the colors. To distinguish colors by the image sensor, a multispectral filter is placed on the sensor. The multispectral filter filters at each photosite one single wavelength range, conventionally characterized by its so-called average or central wavelength.
Thus, the filter allows detecting one single color for each photosite, which color corresponds to the spectral integration of the received flux passing through the filter with the spectral transmission of the filter specifically placed on said photosite, and corresponds to the average or central wavelength of said filter. A subsequent image processing of the data acquired by the photosites allows reconstructing a multicolor image of an observed scene.
Current multispectral filters are characterized by an elementary pattern consisting of a number of photosites close to the number of wavelength ranges to be detected. For example, the Bayer filter, called the RGB filter, consists of a 2*2 elementary pattern having two photosites detecting the green color, one photosite detecting the red color and one photosite detecting the blue color. Each photosite detects only one color, the optronic chain downstream of the sensor should interpolate for each photosite the two remaining colors which have not been detected. This step is called demosaicing or debayering. For multispectral applications, there are filters having a 3 times 3 elementary pattern allowing detecting a larger number of spectral bands, namely nine. However, this filter type could cause problems during the debayering step, leading to a non-constant image quality for the nine colors. Indeed, a sampling problem related to the point spread function (PSF) appears, said response itself being related by diffraction phenomena to the wavelength and the aperture number. Thus, this type of multispectral filters does not take into account the sampling capabilities of the sensor with regards to the diffraction of the optical system at the photosites, which reduces the accuracy and fidelity of the complete optronic chain optics-sensor-demosaicing. This type of multispectral filters does not take into account the sampling capabilities of the sensor with regards to the diffraction of the optical system at the photosites, which reduces the accuracy and fidelity of the complete optronic chain optics-sensor-demosaicing.
The present disclosure aims to overcome these drawbacks.
To this end, the present disclosure relates to a multispectral filter with a filtering array for a sensor with an array of elementary sensors or photosites, said filtering array comprising an elementary pattern formed of an arrangement of N elementary cells capable of filtering central wavelengths λ1, . . . , λk, . . . , λN, the position of the elementary cells sensitive to the central wavelengths in the elementary pattern being determined so that: each wavelength being associated with an integer e1, . . . , ek, . . . , eN, selected so that each of the products e1x λ1, . . . , ekx λk, . . . , eNx λN is substantially constant, each elementary cell sensitive to the central wavelength λk, is positioned at ek positions in the elementary pattern so that the maximum ratio between the distance between two proximal positions of the elementary cell sensitive to said central wavelength and the associated central wavelength is minimal.
The associated central wavelength of a cell may correspond to the weighted barycenter of the wavelengths by the spectral transmission of the considered filter, so-called “at the central wavelength.”. In other words, the wavelengths are weighted by the spectral transmissions at the considered wavelengths.
In particular, the associated central wavelength of a cell may be determined according to the following formula:
with transmissionλ
With the wavelengths {λ1-λ9} being the following series {1,100; 917; 786; 688; 611; 550; 500; 458; 423}, and Δλk being a predetermined wavelength deviation for each wavelength.
The wavelength deviation Δλk may be equal to the difference between two consecutive wavelengths λk, λk+1.
The wavelength deviation Δλk may be equal to half the difference between two consecutive wavelengths Δλk, Δλk+1.
The wavelength deviation Δλk may be equal to a quarter of the difference between two consecutive wavelengths λk, λk+1.
The present invention also relates to a method for determining a multispectral filtering array for a sensor, the method comprising:
The selection of the integers ensuring a substantially constant product between each integer and the associated central wavelength allows statistically uniformizing the transfer function of the image sensor. The proper placement of the ek central wavelengths λk allows taking into account the optical cutoff frequency and the spatial sampling frequency of the sensor for each wavelength. Thus, the filtering array determined by the method is more reliable and accurate because it takes into account the characteristics of the image sensor.
The filtering array may have dimensions corresponding to a grid of photosites forming the image sensor. The elementary pattern may be repeated in the filtering array.
According to one embodiment, the determination of the ek positions of each of the elementary cells sensitive to the N central wavelengths λk may comprise determining a cost function ø, said cost function being determined by the following formula:
With k an integer comprised between {1, . . . , N}, ik, jk being integers comprised between {1, . . . , ek}, jk being different from ik, dist(ik, jk) being the Euclidean distance in the plane between the positions of the ikth and jkth photosites sensitive to the central wavelength λk and mini
Said distance may be the actual distance between elementary cells.
Hence, the plurality of positions of each elementary cell sensitive to the wavelength may be determined by the positions for which the cost function is the lowest.
Thus, the products ekx λk are considered constant when they meet the following formula:
A being selected less than or equal to 15% and preferably less than 7%.
Thus, the integers ek are selected so as to meet this formula.
The elementary pattern may be a square array. This allows reducing the computing resources necessary to implement the process and simplify manufacturing.
The dimension of the elementary pattern is determined by the following formula:
with p being the dimension of the elementary pattern.
The method further comprises: assigning to each position of the elementary pattern a central wavelength associated with the integer arranged at said position of the elementary pattern.
The method may comprise, for one or more position(s) of the elementary pattern, a step of reassigning a first wavelength of the wavelengths by a second different wavelength having a cost function at said position lower than the cost function of the first wavelength at said position.
The predetermined number N of wavelengths is greater than three and is in particular equal to nine.
The present disclosure also relates to a filtering array for a multispectral sensor obtained by the method as mentioned before.
The present disclosure relates to a multispectral sensor comprising a filter corresponding to a filtering array obtained by the aforementioned filtering array determination method.
The present disclosure relates to a device for determining a filtering array for a multispectral sensor including a processing circuit for the implementation of the aforementioned method.
The present disclosure relates to a computer program including instructions for the implementation of the aforementioned filtering array determination method, when said instructions are executed by a processor of a processing circuit.
The image capturing device 300 of
The electronics stage 310 is also configured to carry out the following steps:
Step 312 of demosaicing, also called debayering, of interpolating of the data detected by the image sensor 308.
Step 314 of processing images from the data interpolated during step 312.
Step 316 of returning a final image representing the scene 302.
For the multispectral imaging system, the sensor 300 is located at the optical focus of these systems, the image formed by the optical stage 304 located upstream of the sensor 300 from the observed scene. The filter 302 is arranged at the image sensor 308.
The method 100 determines a filtering array to be applied to the image sensor. In particular, the method 100 determines an elementary pattern repeated in the filtering array.
The method 100 comprises a step 102 of receiving or determining central wavelengths to be filtered λ1, . . . , λk, . . . , λN, with k being an integer and N being the number of wavelengths λk.
For example, for a sensor made of Si, the wavelengths are comprised between 380 nm and 1,100 nm. In a particular embodiment, the central wavelengths may be {1, 100; 917; 786; 688; 611; 550; 500; 458; 423} nm. In this example, a sensor made of Si or InGaAs is preferred because these materials are photosensitive to the wavelengths of the range of mentioned wavelength values between 423 nm and 1,100 nm.
The central wavelengths may be different by +/−100 nm from the values in the list {1,100; 917; 786; 688; 611; 550; 500; 458; 423} nm. The spectral widths of each of the filters will be at mid-height proportional to a fraction comprised between ¼ and 1 of the deviation between 2 consecutive central wavelengths. Each wavelength varies in an interval defined by a wavelength deviation around said wavelength, the wavelength deviation could be a fraction comprised between 0.25 and 1 of the difference between two consecutive central wavelengths.
Thus, the minimum and maximum limits of the filters could have values from among those of the following table, with an uncertainty of +/−20 nm:
The method 100 comprises a step 104 of determining a series of integers e1, . . . , ek, . . . , eN.
Each integer ek is associated with a central wavelength λk. The integers ek are determined so that the product e1x λ1, . . . , ekx λk, . . . , eNx λN of each integer ek and of the associated wavelength λk is substantially constant.
For example, the integers ek are determined so as to meet the following formula:
The value A is selected so as to be less than or equal to 15%. Preferably, the value A is less than 7%. The selection of the integers ek allows uniformizing the amount of energy folded back into each central wavelength λk. Indeed, for a signal having a bandwidth [0, fc] where fc represents the optical cutoff frequency at 1/(λ*F), F being the aperture number and with a sampling frequency fech, then for the frequencies belonging to [0; fech], the signal is well sampled and will be faithfully reproduced but for the frequencies belonging to [fech; fc], the signal will pass but will not be faithfully reproduced. In the latter case, it will literally be folded back over the correctly reproduced part.
This allows uniformizing, i.e. making as equal/constant as possible, the ratio between the optical cutoff frequency and the spatial sampling frequency fechk of the sensor for each wavelength λk.
Indeed, the spatial sampling frequency fechk of the sensor is on average equal to the inverse of the average distance between the photosites sensitive to the central wavelength λk and therefore proportional to the number ek of photosites sensitive to λk in the elementary pattern. Thus, the uniformization of the optronic chains at the different wavelengths is achieved by a product ekx λk that is as homogeneous as possible, and even constant.
The method 100 comprises a step 106 of determining the dimension of the elementary pattern. For example, the elementary pattern may be square, which simplifies the implementation of the method 100 in terms of necessary computing resources. In this case, step 106 may be carried out by solving the following formula:
With p being the dimension of the elementary pattern.
For the following central wavelengths: {1, 100; 917; 786; 688; 611; 550; 500; 458; 423} nm, the series of integers may be {5; 6; 7; 8; 9; 10; 11; 12; 13} and the dimension of the elementary pattern may be 9×9.
Afterwards, the method 100 comprises a step 108 of determining one or more position(s) for each central wavelength λk. The central wavelengths are positioned (l, m) in the elementary pattern so that the ratio between the distance at the nearest photosite sensitive to the same central wavelength and the central wavelength is minimum.
Step 108 comprises the minimization of a cost function ø. For this purpose, a cost function ø is determined for each integer ek according to the following formula:
With k an integer between {1, . . . , N}, ik, jk being integers comprised between {1, . . . , ek}, ik being different from jk, dist(ik, jk) being the Euclidean distance in the plane between the positions of the ikth and jkth photosites sensitive to the central wavelength λk and mini
The distance dist(ik, jk) may be determined between two positions of photosites sensitive to the central wavelength λk within the same elementary pattern or between a first photosite position λk in a first elementary pattern and a second photosite position λk in a second elementary pattern adjacent to the first elementary pattern. Thus, the cost function ø integrates the repetition of the pattern.
The method 100 may comprise a step of generating the filtering array by repetition of the elementary pattern in the filtering array. The filtering array has dimensions corresponding to the number and distribution of the photosites in the image sensor. Thus, a central wavelength λk is assigned to each photosite of the image sensor.
Step 108 may lead to a plurality of distinct elementary patterns. In this case, the method 100 may comprise a step of selecting an elementary pattern among the elementary patterns resulting from step 108. This selection may be random.
The method 100 may comprise a step of optimizing the elementary pattern. This step comprises reassigning a position sensitive to a central wavelength λk to a second central wavelength λk in the elementary pattern. The elementary pattern after reassignment has a lower cost function than the initial elementary pattern. The reassignments are selected by observation or by combination trial. The integers associated with the central wavelengths then evolve into e′k=ek−1 and e′h=eh+1.
The optimization step may be carried out several times, for several central wavelengths and several positions.
The method 100 further comprises a step 110 of repeating the pattern determined in the filtering array.
For example, the method may lead to the filtering array 200 shown in
The filtering array 200 obtained by the method 100 is more accurate and more reliable, in particular for a large number of wavelengths, i.e. a number greater than 3. In addition, such a filtering array 200 allows reducing subsequent interpolation errors during image processing because the filtering array takes into account the spatial sampling frequency of the sensor and its optical cutoff frequency.
The wavelengths of the filtering array 400 correspond to the integers of the filtering array 200.
The image sensor may be equipped with the filtering array 200 or the filtering array 400 of
Number | Date | Country | Kind |
---|---|---|---|
2104978 | May 2021 | FR | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/FR2022/050866 | 5/5/2022 | WO |