Patent Grant
11178349
Cameras may include a lens and an imaging sensor. The lens may direct light from a scene onto the image sensor. The image sensor may convert the light captured by the lens into electrical signals. The scene may be reconstructed based on the electrical signals generated by the image sensor and the optical properties of the lens.
In one aspect, a lens-free imaging system for generating an image of a scene in accordance with one or more embodiments of the invention includes an electromagnetic (EM) radiation sensor; a mask disposed between the EM radiation sensor and the scene; and an image processor that obtains signals from the EM radiation sensor while the EM radiation sensor is exposed to the scene and estimates the image of the scene based on, at least in part, the signals and a transfer function between the scene and the EM radiation sensor.
In one aspect, a method of generating an image using a lens-free imaging system in accordance with one or more embodiments of the invention includes obtaining a transfer matrix that relates electrical signals generated by an image sensor when exposed to electromagnetic (EM) radiation from a scene to an image of a scene; and generating an image of a scene based on the transfer matrix and the electrical signals generated by the image sensor when exposed to the scene.
Certain embodiments of the invention will be described with reference to the accompanying drawings. However, the accompanying drawings illustrate only certain aspects or implementations of the invention by way of example and are not meant to limit the scope of the claims.
Specific embodiments will now be described with reference to the accompanying figures. In the following description, numerous details are set forth as examples of the invention. It will be understood by those skilled in the art that one or more embodiments of the present invention may be practiced without these specific details and that numerous variations or modifications may be possible without departing from the scope of the invention. Certain details known to those of ordinary skill in the art are omitted to avoid obscuring the description.
Embodiments of the invention relate to lens-less imaging systems, lens-less imaging sensors, and methods of operating lens-less imaging systems. A lens-less imaging system may generate electrical signals in response to receiving electromagnetic (EM) radiation. The electrical signals generated by the lens-less imaging system may be processed by an image processor to generate an image of a scene from which the electromagnetic energy was scattered.
In one or more embodiments of the invention, a lens-less imaging system may include an electromagnetic radiation sensor, one or more masks, and an image processor. The electromagnetic radiation sensor may generate electronic signal in response to receiving electromagnetic radiation. The electromagnetic radiation sensor may be, for example, a charge capture device (CCD) that responds to EM radiation. In one or more embodiments of the invention, the electromagnetic radiation may be visible light, short-wave infrared light, mid-wave infrared light, long-wave infrared light, terahertz, millimeter wave, wireless or other parts of the electromagnetic spectrum.
Each of the masks may be a planar structure disposed between the EM sensor and a scene. In one or more embodiments of the invention, the masks may include regions that are opaque and other regions that are transparent to EM radiation. The opaque and transparent regions may cause EM radiation that is incident on the mask to be partially reflected away from the EM radiation sensor and partially transmitted to the EM radiation sensor.
In one or more embodiments of the invention, the masks may include regions that have a different refractive indices or thickness of transparent materials. The different regions having different refractive indices or thickness may generate amplitude and phase module of incident EM radiation.
EM radiation that is transmitted to the EM radiation sensor may be diffracted and/or amplitude/phase modulated when compared to EM radiation that is incident on the masks.
In one or more embodiments of the invention, the mask may be disposed at a predetermined distance from the EM radiation sensor. The predetermined distance may be, for example, 0.5 mm. In one or more embodiments of the invention, the predetermined distance may be 0 mm, e.g., disposed directly on the EM radiation sensor. In one or more embodiments of the invention, the ratio of the distance between the EM radiation sensor and the mask divided by the width of the EM radiation sensor may be less than 0.1. In one or more embodiments of the invention, the ratio of the distance between the EM radiation sensor and the mask divided by the width of the EM radiation sensor may be less than 0.25. In one or more embodiments of the invention, the ratio of the distance between the EM radiation sensor and the mask divided by the width of the EM radiation sensor may be less than 0.5. In one or more embodiments of the invention, the ratio of the distance between the EM radiation sensor and the mask divided by the width of the EM radiation sensor may be less than 0.3.
In one or more embodiments of the invention, the image processor may generate an image of a scene based on the electronic signals received from the EM radiation sensor. The image processor may be, for example, a digital signal processor (DSP).
Additional embodiments of the invention may relate to a method of operating a lens-less imaging system. The method may include displaying a number of test patterns to the lens-less imaging system and determining a transfer function between EM radiation that is radiated by a scene and electrical signals produced by an EM radiation sensor of the lens-less imaging system.
The system may include an EM radiation sensor (130). The EM radiation sensor (130) may be a physical device that generates electrical signals in response to incident EM radiation.
The sensing regions (200) of the EM radiation sensor (130) may be disposed over a two dimensional area. While the sensing regions (200) are shown as square regions in a uniform two dimensional grid, the sensing regions (200) may have other shapes and may be distributed in a uniform or irregular pattern over the surface without departing from the invention. In one or more embodiments of the invention, the two dimensional area may be a flat surface. In one or more embodiments of the invention, the two dimensional area may be a curved surface.
In one or more embodiments of the invention, the EM radiation sensor (130) may generate electrical signals in response to visible light. In one or more embodiments of the invention, the EM radiation sensor (130) may generate electrical signals in response to infrared radiation. In one or more embodiments of the invention, the EM radiation sensor (130) may generate electrical signals in response to ultraviolet light, short-wave infrared light, mid-wave infrared light, long-wave infra-red light, wireless, millimeter wave or terahertz radiation. The EM radiation sensor (130) may generate electrical signals in response to EM radiation having other spectral content without departing from the invention.
Returning to
In one or more embodiments of the invention, the mask may include a first number of regions (300) that are opaque and a second number of regions (310) that are transparent. The opaque first number of regions (300) may prevent the transmission of EM radiation through the mask and the transparent second number of regions (310) may allow the transmission of EM radiation through the mask.
In one or more embodiments of the invention, opaque first number of regions (300) may be gold, aluminum, chrome, or other metals. In one or more embodiments of the invention, the transparent second number of regions (310) may be glass, quartz, or may be a void space.
In one or more embodiments of the invention, the mask may include a first number of regions (300) that have refractive indices or thicknesses that are different than the refractive indices and/or thicknesses of a second number of regions (310) that are transparent. The first number of regions (300) and the second number of regions (310) may amplitude and/or phase module EM radiation that is transmitted through the mask.
In one or more embodiments of the invention, the first number of regions (300) and the second number of regions (310) may be distributed over a two dimensional surface. Each region may be disposed adjacent to other regions. While the first number of regions (300) and second number of regions (310) are shown in
In one or more embodiments of the invention, the first number of regions (300) and second number of regions (310) are distributed across the mask in the form of an outer product of two one-dimensional binary sequences. Each of the binary sequences may have any values. Values of 1 may indicate an opaque region (300) while values of 0 may indicate a transparent region (310), or the reverse. In one or more embodiments of the invention, the two one dimensional binary sequences are pseudorandom binary sequences. In one or more embodiments of the invention, the binary sequences are maximum length sequences (M-length sequences). In one or more embodiments of the invention the binary sequences are subsets of the Hadamard sequence or the S-matrix sequence.
In one or more embodiments of the invention, there may be an equal number of first number of regions (300) and second number of regions (310). In other words, if there are 8000 first number of regions (300) there may be 8000 second number of regions (310).
In one or more embodiments of the invention, the feature size of each opaque region (300) and each transparent region (310) may be 10% of the distance between the masks (120) and the EM radiation sensor (130,
In one or more embodiments of the invention, each opaque region (300) and each transparent region (310) may have a square cross section. The square cross section of each region may have an edge length of the feature size of each region. In other words, a feature size of 30 μm indicates that each region is a 30 μm×30 μm square.
In one or more embodiments of the invention, the masks (120) may be disposed on a transparent support structure. The transparent support structure may be, for example, a quartz wafer.
Returning to
The system may include an image processor (140). The image processor (140) may receive the electrical signals generated by the EM radiation sensor (130) and generate an image of the scene (100) based on the electrical signals.
In one or more embodiments of the invention, the processor (400) may be a general purpose processor, embedded processor, programmable gate array, digital signal processor, or any other type of data processing device.
The storage (420) may be a non-transitory computer readable storage including instruction that, when executed by the processor (400), cause the image processor (140) to perform the functionality shown in
When executing the instructions stored on the storage (420), the image processor (140) may generate an image of the scene (100) based on a relationship between EM radiation scattered by the scene (100) and the electrical signals generated by the EM radiation sensor (130). When EM radiation interacts with the scene (100), it may be scattered. A portion of the scattered EM radiation may be scattered toward the system as scene EM radiation (150), as seen in
When scene EM radiation (150) is incident on the system, the masks (120) may scatter some of the scene EM radiation (150) as reflected EM radiation (160) and may scatter some of the scene EM radiation (150) as transmitted radiation (170). The reflected EM radiation (160) does not interact with the EM radiation sensor (130) while the transmitted EM radiation (170) interacts with the EM radiation sensor (130), e.g., causes the EM radiation sensor (130) to generate electrical signals. The transmitted EM radiation (170) may be modulated and diffracted by the masks (120) when compared to the scene EM radiation (150). Modulating and diffracting the transmitted EM radiation (170) may multiplex the transmitted radiation onto a number of sensing regions of the EM radiation sensor.
For example, a first transparent region (320) is shown in
To generate an image of the scene (100), it is assumed that the scene includes a number of EM radiation sources, e.g., pixels, equal to the number of features of the masks (120). In other words, if the mask is a 100×100 feature array, it is assumed that the scene consists of 100×100 pixels.
Based on the above, the transfer function between an image of the scene and the electrical signals generated by the EM radiation sensor (130) is:
y=ϕx+e Equation (1)
where x is a matrix of the pixelated image of the scene, y is a matrix of the electrical signals generated by the EM radiation sensor (130), Φ is the transfer matrix, and e is the error due to system noise and other factors. In one or more embodiments of the invention, the incident field is coherent and is represented as a complex-valued field or vector. In one or more embodiments of the invention, the incident field is incoherent and is represented as a real-valued field or vector.
The image of the scene may be determined by inverting equation (1), assuming the transfer matrix is known. For example, the image of the scene may be determined by recording the electrical signals of the EM radiation sensor (130) when the system is exposed to scene EM radiation (150). Equation (1) may be inverted using any method such as a least-squares problem or other regularized least-squares problem.
If a mask is used that is formed using two one-dimensional sequences, as described above, the transfer function between the image of the scene and the electrical signals generated by the EM radiation sensor (130) is:
Y=ϕLXϕRT+E Equation (2)
where ΦL, ΦR are sequences that correspond to one-dimensional convolution along the rows and columns of the scene, X is a matrix of the pixelated image of the scene, y is a matrix of the electrical signals generated by the EM radiation sensor (130), and E is the error due to system noise and other factors.
In one or more embodiments of the invention, the transfer matrix may be determined by modeling of the modulation and diffraction of EM radiation due to the masks (120). For example, the masks (120) may be modeled computationally to determine how light from a scene (100) is distributed onto the EM radiation sensor and thereby determine the transfer function.
In one or more embodiments of the invention, the transfer matrix may be determined by the method shown in
In Step 500, a display screen is aligned to the lens-free imaging system. The display screen may be, for example, a liquid crystal display. Other types of display may be used without departing from the invention. In one or more embodiments of the invention a physical light source, such as a point or a line, may be spatially translated to calibrate the transfer matrix.
In one or more embodiments of the invention, the display screen may be aligned to the lens-free imaging system by displaying a test pattern on the display while recording electrical signals from an EM radiation sensor of the lens-free imaging system. The display pattern may be, for example, a cross-hairs pattern or point pattern. The recorded electrical signals may indicate the degree of alignment between the display and the EM radiation sensor. The position of the display and lens-free imaging system may be manually adjusted to improve alignment.
In Step 510, a first plurality of patterns may be displayed on the display. During the display of each test pattern, electrical signals generated by the EM radiation sensor may be recorded and stored. In one or more embodiments of the invention, the plurality of images may be obtained by physically translating a fixed illumination pattern such as a line or a point light source.
In one or more embodiments of the invention, each pattern of the first plurality of patterns displayed may be a unique Hadamard pattern, unique line patterns, or unique dot patterns.
In one or more embodiments of the invention, the number of patterns of the first plurality of patterns may be equal to the number of regions across a width of a mask of the lens-free imaging system.
In Step 520, a second plurality of patterns may be displayed on the display. During the display of each test pattern, electrical signals generated by the EM radiation sensor may be recorded and stored.
In one or more embodiments of the invention, each pattern of the second plurality of patterns displayed may be a unique Hadamard pattern. In other words, all of the patterns of the first plurality of patterns and second plurality of patterns may be unique Hadamard patterns. By utilizing unique patterns, an over determined system of linear relationship may be generated.
In one or more embodiments of the invention, the number of patterns of the first plurality of patterns may be equal to the number of regions across a height of a mask of the lens-free imaging system.
Thus, during steps 510 and 520, a total of 2N patterns may be displayed where N is the number of regions across a width or height of a mask.
In Step 530, a first one-dimensional convolution sequences and a second one-dimensional convolution sequence are determined. The one dimensional convolutions sequences may correspond to the ΦL, ΦR sequences. The one dimensional convolution sequences may be found by, for example, computing the truncated singular value decomposition of:
Y=(ϕLa)(ϕRb)T Equation (3)
In Step 540, the first one-dimensional convolution sequence and the second one-dimensional convolution sequence are stored.
Each of the aforementioned stored sequences may be used to calculate a scene image.
In Step 600, electrical signals from an EM radiation sensor are obtained by an image processor while the EM radiation sensor is exposed to EM radiation from a scene. For example, the EM radiation sensor may be exposed to EM radiation for 10 milliseconds.
In Step 610, the image processor generates an image of the scene based the obtained electrical signals and a transfer function. The transfer function may be stored in the image processor.
In one or more embodiments of the invention, the transfer function may be a matrix having a number of elements corresponding to the number of regions of a mask of the lens-free imaging system.
In one or more embodiments of the invention, the transfer function may be two one dimensional sequences. Each sequence may have a number of elements corresponding to the number of regions along a width or height of a mask of the lens-free imaging system.
In one or more embodiments of the invention, the image of the scene may be generated by solving the least-squares minimization problem:
{circumflex over (X)}LS=arg minX∥ϕLXϕRT−Y∥22 Equation (4)
The least-squares minimization problem may be solved in any way without departing from the invention.
In one or more embodiments of the invention, the image of the scene may be generated by solving the regularized least-squares minimization problem:
{circumflex over (X)}LS=arg minX(∥ϕLXϕRT−Y∥22+θ(X)) Equation (5)
where ⊖(X) represents a regularizer. In one or more embodiments of the invention, The regularizer is the total variation, Wavelet sparsity, L1 norm of wavelet, or other patch-based or dictionary-based regularizer. The regularizer may be other regularizers without departing from the invention.
The least-squares minimization problem may be solved in any way without departing from the invention.
Once solved, the image of the scene {circumflex over (X)}LS may be stored in the image processor and/or displayed and/or stored. For example, the image of the scene may be stored in the storage of the image processor.
While the above description of generating an image of a scene has been for single images, embodiments of the invention may include generating a number of images using the above method at different points in time to generate a video. In one or more embodiments of the invention, an image of a scene at a first point in time may be used to improve the speed of reconstruction of a second image scene at a second point in time. For example, there may be little difference between images of a scene at different points in time that are temporally separated by small amounts, e.g., fractions of a second. An image of a scene at a first time that is temporally separated from an image as a scene at a second time may substantially reduce the computation power and time required to solve either equation 4 or equation 5 when compared to generating an image of a scene without a previously generated image of the scene.
The following are explanatory examples of lens-free camera systems in accordance with one or more embodiments of the invention.
A first example of a lens-free camera system includes a 1036×1048 pixel color EM radiation sensor. Each pixel of the EM radiation sensor may be 6.45 μm wide. The EM radiation sensor may have a size of 6.7 mm×8.9 mm.
The first example also includes a mask formed from a fused quartz plate with a chrome metallization layer. The chrome metallization layer may include a two dimensional patter of opaque regions, e.g., metal areas, and transparent regions, e.g., void regions.
The pattern of the mask consisted of 510×510 regions. Each region had a feature size of 30 μm, e.g., square boxes each having an edge length of 30 μm.
To generate the pattern, a maximum length sequence of length 255 was generated. The sequence was repeated twice to generate all 510 elements. The M-sequence is shown in
An outer product of the M-sequence was generated to form a matrix. The matrix is shown in
The chrome metallization layer on the fused quartz plate was patterned according to the matrix shown in
To form the lens-free imaging system, the mask was placed 0.5 mm away from the color EM radiation sensor. Once formed, images of scenes were taken with an average exposure time of 10 milliseconds. Electrical signals from the EM radiation sensor were recorded while exposed to the scene. Images generated from the scene utilizing the methods shown in
A second example of a lens-free camera system includes a short wave infrared EM radiation sensor having a resolution of 256×300 pixels. Each pixel of the EM radiation sensor may be 25 μm wide.
The second example also includes a mask formed from a fused quartz plate with a chrome metallization layer. The chrome metallization layer may include a two dimensional patter of opaque regions, e.g., metal areas, and transparent regions, e.g., void regions.
The pattern of the mask consisted of 64×64 regions. Each region had a feature size of 100 μm, e.g., square boxes each having an edge length of 100 μm.
To generate the pattern, a maximum length sequence of length 127 was generated. The M-sequence is shown in
An outer product of the M-sequence was generated to form a matrix. The matrix is shown in
The chrome metallization layer on the fused quartz plate was patterned according to the matrix shown in
To form the lens-free imaging system, the mask was placed 5 mm away from the color EM radiation sensor. Once formed, images of scenes were taken with an average exposure time of 10 milliseconds. Electrical signals from the EM radiation sensor were recorded while exposed to the scene. Images generated from the scene utilizing the methods shown in
One or more embodiments of the invention may provide one or more of the following advantages: (i) A system in accordance with embodiments of the invention may have a thickness-to-width ration of less than 0.1, (ii) the system may have a very high light collection ability, (iii) the system may be much less expensive to produce when compared to an imaging system that integrates a lens, and (iv) the system may be much simpler to produce when compared to lens based imaging systems.
While the invention has been described above with respect to a limited number of embodiments, those skilled in the art, having the benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.
The invention was made with government support under Grant Number CCF-1117939 awarded by the National Science Foundation. The invention was made with government support under Grant Number HS-1116718 awarded by the National Science Foundation. The invention was made with government support under Grant Number N00014-10-1-10989 awarded by the Department of Defense: Office of Naval Research. The government has certain rights in the invention.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2016/015734 | 1/29/2016 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2016/123529 | 8/4/2016 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4991020 | Zwirn | Feb 1991 | A |
| 10753869 | Veeraraghavan | Aug 2020 | B2 |
| 20060157640 | Perlman | Jul 2006 | A1 |
| 20090095912 | Slinger | Apr 2009 | A1 |
| 20100321501 | Arndt | Dec 2010 | A1 |
| 20110026141 | Barrows | Feb 2011 | A1 |
| 20110085051 | Chi | Apr 2011 | A1 |
| 20110204209 | Barrows | Aug 2011 | A1 |
| 20120062760 | Klapp et al. | Mar 2012 | A1 |
| 20120268574 | Gidon | Oct 2012 | A1 |
| 20130043375 | Baleine | Feb 2013 | A1 |
| 20130113921 | Richards | May 2013 | A1 |
| 20140184754 | Suh | Jul 2014 | A1 |
| 20140301535 | Williams | Oct 2014 | A1 |
| 20150139560 | DeWeert | May 2015 | A1 |
| 20150215529 | Wang | Jul 2015 | A1 |
| 20150293018 | Stork | Oct 2015 | A1 |
| 20150381958 | Zalevsky | Dec 2015 | A1 |
| 20160003994 | Gill | Jan 2016 | A1 |
| 20160065824 | Geng | Mar 2016 | A1 |
| 20160169667 | Stork | Jun 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 2 434 934 | Aug 2007 | GB |
| 2008-527944 | Jul 2008 | JP |
| 2008-191661 | Aug 2008 | JP |
| 2008-542863 | Nov 2008 | JP |
| 2009-522825 | Jun 2009 | JP |
| 2009-529160 | Aug 2009 | JP |
| 2010-177918 | Aug 2010 | JP |
| 2012-23498 | Feb 2012 | JP |
| 2006125975 | Nov 2006 | WO |
| 2007091051 | Aug 2007 | WO |
| Entry |
|---|
| Office Action issued in corresponding Japanese Application No. 2017-540643 dated Jun. 19, 2018, and English translation thereof (31 pages). |
| International Search Report issued in PCT/US2016/015734 dated Jul. 5, 2016 (3 pages). |
| Written Opinion of the International Searching Authority issued in PCT/US2016/015734 dated Jul. 5, 2016 (8 pages). |
| Cannon, T.M. et al.; “Coded aperture imaging: many holes make light work”; Optical Engineering, vol. 19, No. 3, May 1, 1980, pp. 283-289 (4 pages). |
| Busboom, A. et al.; “Uniformly Redundant Arrays”; Experimental Astronomy, vol. 8, Jun. 1, 1998, pp. 97-123 (27 pages). |
| First Examination Report issued in corresponding IN Application No. 201727030079 dated Sep. 30, 2019 (5 pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20180027201 A1 | Jan 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62109358 | Jan 2015 | US |
