A relatively new type of image-sensing device employs an odd-symmetry grating to project an interference pattern for capture by a photodetector array. The grating offers considerable insensitivity to the wavelength of incident light in a wavelength band of interest, and also to the manufactured distance between the grating and the array. The grating produces an interference pattern quite different from the captured scene, but that contains sufficient information to mathematically reconstruct the scene or aspects of the scene. Images can thus be captured without a lens, and cameras can be made smaller than those that are reliant on lenses and ray-optical focusing. Embodiments of such image-sensing devices are detailed in U.S. Publication 2014/0253781, which is incorporated herein by reference.
Some imaging applications do not require reconstruction of the imaged scene. For example, tracking movement of a point source using an odd-symmetry grating does not require the overall scene be reconstructed. Where image reconstruction is desired, however, the mathematical operations used to invert the raw image data can be computationally cumbersome.
The detailed description is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
Light in a wavelength band of interest strikes grating 105 from a direction that is normal to the plane 120 of grating 105. (Unless otherwise stated, the wavelength band of interest is the visible spectrum. Cameras developed for use in different applications can have different bands of interest, as is well understood by those of skill in the art.) Grating 105 produces an interference pattern for capture by array 110. Digital photographs and other image information can then be extracted from the pattern. Device 100 is constructed to produce raw image data of high fidelity to support efficient algorithms for image extraction. Efficient extraction algorithms based on Fourier deconvolution introduce barrel distortion, which can be removed by resampling using correction functions.
Grating 105 is formed by an interface between light-transmissive media of different refractive indices, an optical Lanthanum dense flint glass layer 115 and polycarbonate plastic layer 116 above grating 105 in this example. Each of three boundaries of odd symmetry 125 is indicated using a vertical, dashed line. The lower features 130 of grating 105 induce phase retardations of half of one wavelength (π radians) relative to higher features 135. Features on either side of each boundary exhibit odd symmetry. With this arrangement, paired features induce respective phase delays that differ by approximately half a wavelength over the wavelength band of interest (e.g., visible light). Due to dispersion, the difference in the refractive index of the Lanthanum dense flint glass layer 115 and the polycarbonate above grating 105 is an increasing function of wavelength, facilitating a wider wavelength band of interest over which the phase delay is approximately π radians. These elements produce an interference pattern for capture by array 110.
Device 100 includes an optional opaque layer 117 patterned to include an aperture that encompasses or defines the effective limits of grating 105. The aperture windows captured interference patterns, which tends to reduce edge effects that result from subsequent image-recovery algorithms. The aperture can also improve angle sensitivity and spurious light rejection, which can be advantageous for e.g. motion detection and measurement. Opaque layer 117 can be applied directly to a layer forming grating 105, and may be coplanar or nearly coplanar with grating 105. Other embodiments omit the aperture, or may include an aperture spaced away from device 100 instead of or in addition to the aperture in layer 117.
The example of
Although device 100 can include or be used with a focusing element (e.g., a lens), device 100 does not require a focusing element to produce images. Rather than focusing, as would be done by a traditional camera, device 100 captures a diffraction pattern that bears little resemblance to an imaged scene, but that is nevertheless interpretable by a computer or processor. Grating 105 creates a certain point-spread function (PSF), a multi-armed thin spiral in this example, on the sensor array for every point of light in the imaged scene. The location of the center of the PSF is uniquely determined by the incident angle of light from the point source. Since faraway scenes can be thought of as collections of point sources of varying intensity, the sensed signals resemble a convolution of the PSF with the faraway scene. A scene can be computationally reconstructed from its corresponding interference pattern if there is a 1:1 map of scenes to sensor readings. In the case where the sensed signals are well approximated by a convolution with a fixed PSF, the Fourier components of the scene that are recoverable are the same as the Fourier components of the PSF with sufficient power to be observable above the noise sources in the system.
Device 100 is a linear system, so image extraction can be accomplished by applying general linear inversion techniques to pattern 205. Such techniques multiply sensed data with the regularized pseudoinverse of the transformation exhibited by grating 105. However, these general techniques are computationally cumbersome, taking O(n4) operations and using O(n4) data for an n-by-n pixel array.
General linear algebra techniques fail to make use of the fact that the optical transform is approximately a convolution of a scene with the PSF of grating 105. If the optical transfer function instead were a simple convolution with the PSF, then one could reconstruct scenes using e.g. Fourier-domain regularized deconvolution algorithms. The computational complexity of these Fourier methods is O(n2 log(n)) and they require storage of a calibration occupying only O(n2) memory for an n-by-n pixel array. This advantage is offset for device 100 because Fourier deconvolution introduces significant barrel distortion.
Due to Snell's law,
light incident the grating at a certain angle is refracted to propagate at an angle closer to the normal direction. For small angles, such as for ray 300, sin θ≈θ≈tan θ; this has the effect of demagnifying the image by a factor
For larger incident angles, however, Snell's law imposes greater refraction and concomitant demagnification. The effect of this distortion is to pull light from greater incident angles (such as those at the corners of the field of view) towards the center of array 110 by a greater demagnification factor than for relatively smaller incident angles (such as those at the sides of the field of view).
Barrel distortion can be undone computationally by resampling as follows. The distance rs of the image of object 500 from the optical axis on array 110 is given by trigonometry and Snell's law as follows:
A mathematically-equivalent form of equation (1) that does not involve calls to trigonometric functions is:
Here h is the grating-sensor separation, ni is the index of refraction between the sensor and the light source, and nr is the index of refraction in the medium between the grating and the sensor. Equation (1) can be used to calculate the location on array 110 whose center corresponds to light incident from any angle, making it possible to construct a distortion-free map of the imaged scene. Distance dw need not be known; distortion-free images are typically defined to be those where the image is geometrically similar to the object, and this similarity will hold for any assumed distance dw.
A lookup table generated using equation (1), or on-the-fly computation using equation (1), can be used to sample the distorted raw Fourier reconstruction (e.g., image data 400 of
In one embodiment, layer 116 is a twenty-micron phase-shift layer of a UV-cured plastic with a refractive index of about 1.4; layer 117 is a 2,000 Angstrom tungsten film patterned with an aperture 55 microns in diameter; and layer 115 is a 145 micron layer of glass adhered to array 110 via a five-micron layer of an optical adhesive (not shown). Array 110, in such an embodiment, can be a 200-by-200 pixel array with a pixel pitch of 1.67 microns.
IC 810 includes a processor 815, random-access memory (RAM) 820, and read-only memory (ROM) 825. ROM 825 can store a digital representation of the deconvolution kernel for the PSF of grating 105, along with other parameters or lookup tables in support of image processing. Processor 815 captures digital image data from array 110 and uses that data with the stored PSF to compute e.g. image data 400 and image 700 of
The PSF of grating 105, possibly in combination with the underlying array, is known from a prior calibration or high-fidelity simulation. This information is represented by a response 930, a digital representation of which can be stored within device 100 or elsewhere. Alternatively, the spatial- or Fourier-domain deconvolution kernel needed to undo the effects of convolving with the PSF may be stored. Sampled pattern 205 is deconvolved using response 930, using e.g. spatial or Fourier deconvolution, to construct barrel-distorted image data 400 (940). Finally, distorted image data 400 is resampled (950) to obtain reduced-distortion image 700.
The noise level and operational requirements for the system may not be constant in time. For instance, with changing light levels which influence the signal to noise level in captured data or operational requirements for at times high resolution, and at other times low noise each lead to differences in the appropriate deconvolution kernel. These changing requirements can be met by using deconvolution kernels with changing regularization parameters. For example, a regularized deconvolution kernel can be computed as follows:
where γ depends on the degree of noise robustness desired. It may thus be desirable for a sensor to have access to a spectrum of deconvolution kernels, with a variety of noise rejection characteristics. In this case, a variety of deconvolution kernels can be stored directly in memory, or can be computed as needed either from interpolation from two or more stored deconvolution kernels or from the PSF itself, as will be evident to those skilled in the art.
Grating 1005 is formed by an interface between light-transmissive media of different refractive indices, a twenty-micron phase-shift layer 1012 of UV-cured plastic and a twenty-micron layer 1013 of thermoplastic with respective indices n1=1.41 and n2=1.61 in this example. Different combinations of indices can be used, and index n2 can be less than index n1. A space 1014 with refractive index ni separates layer 1013 from array 1010.
In this example, space 1014 provides a separation h3 of about 145 microns and has a refractive index ni near one to match the refractive index ns of the medium through which light travels from objects in an imaged scene to device 1000. For example, index ns might be air, in which case index ni would be close to one and the medium separating array 1010 from the grating might be e.g. a gas or a vacuum. For aquatic or in vivo applications index ni might be closer to that of the fluid environment.
Error e1 is proportional to thickness h1, a value that is inconsequential in comparison with the dimensions of imaged scenes. In the embodiment of
Error e2 is proportional to thickness h2, which is the same twenty microns as for thickness h1. Thickness h2 is far from inconsequential, however, as the relevant dimensions are those of array 1010. In this example, error e2 is 9.8 microns for rays incident at an angle of 45 degrees. Pixel pitch is 1.67 microns, so the offset imposed by error e2 translates into an error of about six pixels for light at incident angles of over 45 degrees. Such an error ease easily observed in the image data. Error e2 increases with incident angle, leading to barrel distortion.
IC 1410 includes a processor 1415, random-access memory (RAM) 1420, and read-only memory (ROM) 1425. ROM 1425 can store a digital representation of the PSF of grating 1005 from which a noise-dependent deconvolution kernel may be computed. ROM 1425 can store a deconvolution kernel for the PSF along with other parameters or lookup tables in support of image processing. Processor 1415 captures digital image data from array 1010 and uses that data with the stored PSF to compute e.g. images and other image data. Processor 1415 uses RAM 1420 to read and write data in support of image processing. Processor 1415 may include SIMD instructions, butterflies accelerating the Cooley-Tukey FFT algorithm in hardware, and other specialized processing elements which aid fast, power-efficient Fourier- or spatial-domain deconvolution.
The noise level and operational requirements for the system may not be constant in time. For instance, with changing light levels which influence the signal to noise level in captured data or operational requirements for at times high resolution, and at other times low noise each lead to differences in the appropriate deconvolution kernel. These changing requirements can be met by using deconvolution kernels with changing regularization parameters. For example, a regularized deconvolution kernel can be computed as follows:
where γ depends on the degree of noise robustness desired. It may thus be desirable for a sensor to have access to a spectrum of deconvolution kernels, with a variety of noise rejection characteristics. In this case, a variety of deconvolution kernels can be stored directly in memory, or can be computed as needed either from interpolation from two or more stored deconvolution kernels or from the PSF itself, as will be evident to those skilled in the art.
While the subject matter has been described in connection with specific embodiments, other embodiments are also envisioned. For example; while each grating detailed previously may be used in connection with photoreceptors to collect incident light, gratings in accordance with these and other embodiments can be used more generally in imaging devices that project images using photoelements that admit light; the wavelength band of interest can be broader or narrower than the visible spectrum, may be wholly or partially outside the visible spectrum, and may be discontinuous; cameras and gratings detailed herein can be adapted for use in multi-aperture or programmable-aperture applications; and imaging devices that employ other types of gratings can benefit by application of methods disclosed herein. Suitable gratings are detailed in U.S. application Ser. No. 14/458,179 to Patrick Gill, David Stork, and Jay Endsley, filed 12 Aug. 2014 and entitled “Patchwork Fresnel Zone Plates for Lensless Imaging,” which is incorporated herein by reference. Other variations will be evident to those of skill in the art. Therefore, the spirit and scope of the appended claims should not be limited to the foregoing description. Only those claims specifically reciting “means for” or “step for” should be construed in the manner required under the sixth paragraph of 35 U.S.C. §112.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/US16/16837 | 2/5/2016 | WO | 00 |
Number | Date | Country | |
---|---|---|---|
62115586 | Feb 2015 | US | |
62115590 | Feb 2015 | US |