Patent Application
20170214861
This disclosure relates generally to imaging and, more particularly, to optically multiplexed imaging of a field of view.
Many different fields of endeavor have a need to image extended fields of view with high resolution to detect and observe objects within the field of view or track movement relative to reference points. For example, observational astronomy, celestial navigation systems, and security/surveillance applications all need to monitor extended fields of view with high resolution. Conventional imaging systems are limited by a tradeoff between field of view and resolution: with a finite number of pixels on an image sensor the sampling resolution in object space (i.e., the number of pixels devoted to a given area in the scene being imaged) is decreased as the field of view is increased. When requirements demand a combination of extended field of view and resolution that exceeds a conventional single-camera fixed field of view architecture, these needs are often met using arrays of multiple cameras or image sensors arranged to view different regions of a scene, or using a single sensor or pixel array with a scanning mechanism (e.g., a pan-tilt-zoom mechanism) to sweep out a high-resolution image of an extended field of view over time. The former is bulky and costly because it requires discrete optical and sensor assemblies for each region of the field of view. The latter suffers from the need for a scanning mechanism and intermittent temporal sampling (i.e., the device cannot view the entire field of view at any one time). Other designs incorporate both a bank of cameras and scanning mechanisms to improve upon some aspects of dedicated array or scanning devices, but these hybrid devices also suffer the disadvantages of both.
Other fields endeavor to create a stereo image or a 3-dimensional (3D) depth image of a scene. This can be done using two or more cameras that observe an object from different perspectives, or with a single camera that produces images from two or more perspectives on a single focal plane. The former method suffers from the added cost, power, volume, and complexity of using multiple cameras, as well as geometric and intensity differences in the images resulting from the different optical systems. Methods using a single camera approach typically either (a) use prisms or mirrors to produce two or more shifted images on a camera's focal plane where each image fills only a fraction of the focal plane's area to prevent overlap, thereby resulting in a reconstructed stereo image that has a smaller field of view and fewer pixels than are available in the image sensor, or (b) use a moving element that allows a sequence of frames to be captured from different perspectives. This latter approach is more complex and restricts the sampling rate of the system.
Optically multiplexed imaging is a developing field in the area of computational imaging. Images from different regions of a scene, or from different perspectives of the same region, are overlaid on a single sensor to form a multiplexed image in which each pixel on the focal plane simultaneously views multiple object points, or the same object point from multiple perspectives. Information bandwidth can be increased with optically multiplexed imaging because each pixel simultaneously views multiple object points. A combination of hardware and software processes are used to disambiguate the measured pixel intensities and produce a de-multiplexed image. The result can be a higher resolution and wider field of view image than is possible with conventional imaging systems that view only one object point with each image sensor pixel. For a system with N multiplexed channels, for example, the resulting image can have N-times greater pixels than the format of the image sensor used to capture the multiplexed image. This technique allows a multiplexed imaging device to increase its effective resolution (i.e. the number of pixels in the reconstructed image), which can then be applied to extending the field of view or capturing images from multiple perspectives without resolution loss.
Optically multiplexed imaging, then, can overcome fundamental tradeoffs and disadvantages associated with conventional imaging solutions, especially such solutions that implement image sensor arrays or scanning image sensors to observe large fields of view. Optically multiplexed imaging can, for example, deliver the high spatial and temporal resolution of a staring array of image sensors while requiring only a single optical telescope and focal plane image sensor array. This can save size, weight, power, and cost.
There are, however, challenges associated with optically multiplexed imaging. For example, optically multiplexed imaging can require significant computational resources to disambiguate the captured image and reconstruct de-multiplexed images therefrom. Inability to perform the required computations in a timely manner can prevent use of the system for, e.g., high frame rate video capture. By way of further example, prior techniques for optically multiplexed imaging can lack sufficient precision and/or speed to encode one or more multiplexed images in a manner that allows for efficient disambiguation. In some systems, for example, full-aperture beam splitters can be used to combine fields of view and a continuously scanning mirror can shift layers of the multiplexed image between frames to encode a single field of view. This technique can result in motion blur due to the continually scanning mirror and imprecise control of the encoding. In other multiplexed imaging systems, liquid crystal shutters are used to encode at least one image being multiplexed. Such a system also lacks dynamically variable sub-pixel precision that can allow for enhanced multiplexing performance. In still other systems, an array of interleaved micro-prisms and micro-eyelid shutters can be used to multiplex and encode multiple fields of view. While this can be a compact and efficient architecture, it can suffer from a limited spectral bandwidth due to chromatic aberrations caused by the prisms and it does not provide a capability for super-resolution or encoding with a spatial Point Spread Function, as described in more detail below.
Accordingly, there is a need for improved optically multiplexed imaging devices and methods. More particularly, there is a need for such devices and methods that can provide faster and more precise control of encoding elements to permit encoding in a manner that maximizes performance and efficiency of an optically multiplexed imaging system.
The present disclosure generally provides improved optically multiplexed imaging devices and methods through dynamically variable encoding of one or more image channels. In some embodiments, the dynamically variable image encoding can be any of rapid and precise, that is, occurring at frequencies at or above a capturing frame rate of an image sensor or array and with precision that is less than an angular sampling of an image sensor pixel. The devices and methods for dynamically variable image encoding described herein can be applied to a number of different optical design architectures and can provide a number of advantages over prior imaging systems or methods. For example, dynamically variable encoding of one or more channels in an optically multiplexed system can be utilized to provide flexibility for the optically multiplexed imaging system to operate in different modes optimized for specific scene conditions and sensing objectives. In addition, the devices and methods described herein can provide improved signal efficiency and robustness of disambiguation over prior imaging systems. This can enable, for example, efficient snapshot (i.e., single-frame) extended field of view disambiguation of sparse scenes. In addition, extended field of view imaging of a scene can be performed at rates adequate for motion video capture with the improved efficiency and performance of the devices and methods described herein.
The ability to dynamically encode one or more channels of an optically multiplexed imaging system at speeds at or above a capturing frame rate of an image sensor and with precision that is less than an angular sampling of an image sensor pixel can also provide a capability of recovering spatial resolutions finer than a pixel sampling. This can add a multiplicative factor to a resolution of an optically multiplexed imaging system, which already utilizes a single image sensor pixel to view multiple points in object space (i.e., in the scene being observed). The devices and methods described herein can also achieve temporal super-resolution due to the ability to rapidly and precisely vary encoding of one or more channels in the imaging device or system. More particularly, temporal information can be recovered at frequencies that exceed the image sensor or array frame rate.
Still another advantage provided by the devices and methods described herein is lower computational complexity during image reconstruction or disambiguation. More particularly, the computational architectures described herein can significantly lower complexity of image reconstruction when compared to conventional techniques for directly solving for an image through, for example, a matrix inversion process.
In one aspect, a method of imaging a scene is provided that includes capturing light from a plurality of regions of the scene in a plurality of channels, directing each of the plurality of channels onto a focal plane of an image sensor, and encoding an image formed by one or more of the plurality of channels prior to detection by the image sensor. Furthermore, the encoding of the image can be varied by a precise amount over time.
The devices and methods described herein can have a number of additional features and/or variations, all of which are within the scope of the present disclosure. In some embodiments, for example, in some embodiments encoding the image can include shifting the image. In such an embodiment, shifting the image can be performed with a precision that is less than an angular sampling of an image sensor pixel. In other embodiments, shifting the image can be performed at rates equal to, or faster than, a capturing frame rate of the image sensor.
Variations in encoding can be implemented in several manners. For example, in some embodiments a magnitude of image shift used to encode the image can be varied over time. Variations in the magnitude of image shift can occur at rates equal to, or greater than, a capturing frame rate of the image sensor. In other embodiments, a direction of image shift used to encode the image can be varied over time. Here again, variations in the direction of image shift can occur at rates equal to, or greater than, a capturing frame rate of the image sensor. In certain embodiments, a time delay between shifting the image can be varied over time.
A number of different imaging device architectures can be utilized to accomplish the teachings described herein. For example, in some embodiments shifting the image can be accomplished by tilting a mirror using an actuator. The actuator can, in certain embodiments, be a piezoelectric actuator that can be precisely controlled and capable of adjusting the tilt of a mirror rapidly.
In other embodiments, encoding the image can include applying an engineered point spread function instead of shifting an image, and a spatial structure of the engineered point spread function can be varied over time.
Still another method for encoding the image can include at least partially attenuating the image, and any of a duration and an extent of the at least partial attenuation can be varied over time. Encoding via attenuation can be implemented in a variety of manners. In some embodiments, attenuating the image can include placing a partially transparent attenuator in a light path of the channel being encoded. In other embodiments, attenuating the image can include placing a fully absorbing attenuator in a light path of the channel being encoded. In certain embodiments, attenuating the image can include rotating an attenuating element about an axis to place different regions of its area into a light path of the channel being encoded. The different regions of the attenuator can be fully absorbing or partially transparent.
Another method for encoding the image can include imparting illumination to the channel being encoded to amplify a signal thereof relative to other channels.
In certain embodiments, encoding the image can include modifying a phase of the image by imparting any of an aberration and a diffraction effect into a wavefront moving through the channel being encoded. Modifying the phase of the image can include placing a wedged optical element into a light path of the channel being encoded in some embodiments. In other embodiments, modifying the phase of the image can include placing a non-plano surface into a light path of the channel being encoded.
In order to maintain a compact size, imaging devices according to the teachings provided herein can make use of miniaturized components that provide rapid and precise positioning capabilities. For example, in some embodiments encoding the image can be performed using a micro-electromechanical system (MEMS) light modulating array. The MEMS array can be, in some embodiments, an array of mirrors that can be rapidly and precisely tilted or translated to a variety of positions. In other embodiments, encoding the image can include deforming a mirror to alter characteristics of an image being reflected thereby.
While a variety of different techniques for encoding an image are disclosed above, in some embodiments the image formed by one or more of the plurality of channels can be encoded using at least two different techniques in combination with one another. For example, in some embodiments the at least two different techniques can include modifying a phase of the image and attenuating the image. Furthermore, modifying the phase of the image can include any of shifting the image and applying an engineered point spread function to the image.
In another aspect, a method of imaging a scene is provided that includes capturing light from a plurality of regions of the scene in a plurality of channels, directing each of the plurality of channels onto a focal plane of an image sensor, and capturing a frame from the image sensor containing all of the images formed by the plurality of channels in a first state. The method can further include modifying an image formed by at least one of the plurality of channels to a second state, as well as capturing a frame from the image sensor containing all of the images formed by the plurality of channels in the second state. The method can also include repeating the steps of modifying an image formed by at least one of the plurality of channels and capturing a frame from the image sensor for each of a plurality of predetermined states.
Any number of predetermined states can be employed, and in some embodiments the method can further include repeatedly cycling through the plurality of predetermined states. Further, the plurality of predetermined states can follow a predetermined pattern. In other embodiments, the plurality of predetermined states can be any of random and non-repeating. In some embodiments, the plurality of predetermined states can include two states and an image formed by at least one of the plurality of channels can oscillate between the two states in time with a capturing frame rate of the image sensor.
In certain embodiments, modifying the image can include shifting the image by a magnitude equal to, or greater than, one pixel at the focal plane. Moreover, in some embodiments shifting the image can occur at a rate equal to, or greater than, a capturing frame rate of the image sensor. In other embodiments, modifying the image can include at least partially attenuating the image. In still other embodiments, modifying the image can include applying an engineered point spread function to the image.
In another aspect, a method of imaging a scene is provided that includes capturing light from a plurality of regions of the scene in a plurality of channels and directing each of the plurality of channels onto a focal plane of an image sensor. The method can further include encoding an image formed by one or more of the plurality of channels prior to capture by the image sensor, and decoding the image formed by one or more of the plurality of channels using an algorithm paired to the encoding method. Such a method can select encoding and decoding methods in connection with one another to provide advantages, such as less computationally intensive disambiguation, disambiguation with fewer frame captures, disambiguation with lower noise levels, etc.
As with the methods described above, a number of variations and additional features are possible. For example, in some embodiments encoding the image can include spatially shifting the image. The magnitude of the shift can vary and, in some embodiments, the image can be spatially shifted by an integer amount of pixels. Moreover, the timing of the shifts can be adjusted. In some embodiments, for example, the image can be spatially shifted per frame captured by the image sensor. In such an embodiment, decoding the image can include taking differences between sequential frames to yield a spatial derivative of the image along a direction of motion.
In other embodiments, encoding the image can include attenuating one of the plurality of channels per frame captured by the image sensor. In still other embodiments, encoding the image can include spatially shifting an image formed by each of the plurality of channels using a predetermined unique frequency, and decoding the image can include conducting a frequency analysis of a time series for each pixel of the image sensor.
A similar encoding strategy can be employed utilizing any of the other encoding methods described herein. For example, in some embodiments encoding the image can include any of defocusing and point spread function encoding an image formed by each of the plurality of channels using a predetermined unique frequency, and decoding the image can include conducting a frequency analysis of a time series for each pixel in the image sensor.
In other embodiments, encoding the image can include attenuating an image formed by each of the plurality of channels using a predetermined function of time such that the image can be measured using a matrix with positive, bounded entries, and decoding the image can include measuring a time series for each pixel of the image sensor and constructing the image with a matrix inverse. The predetermined function of time can any of activate and deactivate each of the plurality of channels at a unique frequency, and decoding the image can include computationally projecting the time series of each pixel of the image sensor onto a corresponding channel frequency. In some embodiments, the matrix inverse can be performed within logic of each pixel of the image sensor. Still further, in some embodiments performing the matrix inverse can include projecting measured light onto rows of an inverse matrix using logic that implements a dot product.
In other embodiments, attenuating an image formed by each of the plurality of channels can include reflecting light off a light modulating array and measuring a distinct time series per pixel at two different focal planes, wherein each time series corresponds to two directions light could be reflected from the array. In such an embodiment, decoding the image can include taking a difference between the time series in order to instantiate a matrix with bounded entries that are any of negative and positive. And the method can further include computationally inverting the matrix with bounded entries to recover the image formed by one of the plurality of channels. As noted above, the light modulating array can, in some embodiments, be a micro-electromechanical (MEMS) mirror array.
In certain embodiments, encoding the image can include spatially shifting all but one of the plurality of channels during a single integration period to blur images created by all but one of the plurality of channels. In such an embodiment, decoding the image can include removing the one channel not spatially shifted from the blurred background of the other channels.
Alternatively, encoding the image can include continuously shifting each of the plurality of channels along different trajectories. In such an embodiment, decoding the image can include shifting any of a charge and a digital measurement of the image sensor to follow a trajectory of the channel being decoded, thereby allowing the image to be removed from a blurred background of other channels. In some embodiments, the method can further include simultaneously decoding images formed by a plurality of channels by simultaneously shifting any of a charge and a digital measurement of the image sensor along a plurality of trajectories used to shift images formed by the plurality of channels.
In other embodiments, encoding the image can include differentially rotating each of the plurality of channels so that an image formed by each channel moves in a different direction on the focal plane of the image sensor. In such an embodiment, decoding the image can include shifting any of a charge and a digital measurement of the image sensor to follow a direction of the channel being decoded, thereby allowing the image to be removed from a blurred background of other channels. In some embodiments, the method can further include simultaneously decoding images from a plurality of channels by simultaneously shifting any of a charge and a digital measurement of the image sensor along a plurality of directions used to rotate images formed by the plurality of channels.
In another aspect, a method of imaging a scene is provided that includes capturing light from a plurality of regions of the scene in a plurality of channels and directing each of the plurality of channels onto a focal plane of an image sensor simultaneously. The method can also include encoding one or more of the plurality of channels in a first mode that permits disambiguation of an image formed by each of the plurality of channels from a single frame capture of the image sensor.
In some embodiments, the scene can be sparse in at least one dimension. This can reduce the number of observed items that change in the image over time. As with the above-described methods, encoding can be accomplished in a variety of manners. In some embodiments, for example, encoding one or more of the plurality of channels in a first mode can include applying an engineered point spread function to the channel being encoded.
In certain embodiments, the method can further include encoding one or more of the plurality of channels in a second mode that permits disambiguation of an image formed by each of the plurality of channels using a plurality of single frame captures of the image sensor. For example, encoding one or more of the plurality of channels in a second mode can include shifting and settling images formed by one or more of the plurality of channels with a precision that is less than an angular sampling of an image sensor pixel and at a rate equal to, or greater than, a capturing frame rate of the image sensor. Or, in other embodiments encoding one or more of the plurality of channels in a second mode can include at least partially attenuating images formed by one or more of the plurality of channels.
The method can include switching between encoding in the first mode and encoding in the second mode in some embodiments. For example, an imaging system adapted for surveillance might operate in a first mode during a “standby” period during which a relatively sparse, or unchanging, scene is observed. Upon detection of activity, however, the system can switch to operating in the second mode to process a more information-rich, or dense, scene. In some embodiments, switching between encoding in the first mode and encoding in the second mode can occur at a predetermined rate slower than a capturing frame rate of the image sensor. As noted above, in some embodiments switching between encoding in the first mode and encoding in the second mode can occur in response to information detected in the scene being imaged. In other embodiments, however, switching between encoding in the first mode and encoding in the second mode can occur in response to receiving a command, such as a command from a user or other system managing an imaging system.
In another aspect, a method of imaging a scene can include capturing light from a plurality of regions of the scene in a plurality of channels and directing each of the plurality of channels onto a focal plane of an image sensor. The method can further include constructing an image of the scene at a resolution higher than a native resolution of the image sensor by shifting and settling images formed by the plurality of channels with precision that is less than an angular sampling of an image sensor pixel. In some embodiments, shifting and settling of images formed by the plurality of channels can occur at rates equal to, or faster than, a capturing frame rate of the image sensor.
In another aspect, an imaging device can include an image sensor and a multiplexing assembly configured to collect light from a plurality of regions of a scene into a plurality of channels and direct each channel to the image sensor. Moreover, the multiplexing assembly can be configured to encode an image formed by one or more of the plurality of channels in a manner that varies over time by a precise amount.
As with the methods described above, a number of variations and additional features can be included in the imaging device. For example, in some embodiments encoding an image formed by one or more of the plurality of channels can include shifting the image with a precision that is less than an angular sampling of an image sensor pixel at a rate that is equal to, or faster than, a capturing frame rate of the image sensor. In other embodiments, encoding an image formed by one or more of the plurality of channels can include applying an engineered point spread function, and a spatial structure of the engineered point spread function can be varied over time.
Other encoding methods are also possible and, in certain embodiments, encoding an image formed by one or more of the plurality of channels can include at least partially attenuating the image, and any of a duration and an extent of the at least partial attenuation can be varied over time. In still other embodiments, encoding an image formed by one or more of the plurality of channels can include modifying a phase of the image by imparting any of an aberration and a diffraction effect into a wavefront moving through the channel being encoded. As noted above, any of the various encoding techniques described herein can be used in isolation, or can be combined with one another such that, in some embodiments, encoding an image formed by one or more of the plurality of channels can include encoding with at least two different techniques. The at least two different techniques can include, for example, modifying a phase of the image and attenuating the image. Further, modifying the phase of the image can include any of shifting the image and applying an engineered point spread function to the image.
The imaging device can have a variety of additional components. For example, in some embodiments the multiplexing assembly can include a mirror coupled to an actuator configured to tilt the mirror. Any of a variety of actuators can be utilized and, in some embodiments, the actuator can be piezoelectric. In other embodiments, the multiplexing assembly can include a deformable mirror. In still other embodiments, the multiplexing assembly can include a deformable mirror.
In certain embodiments, the multiplexing assembly can include other light modulating components. For example, in some embodiments the multiplexing assembly can include a micro-electromechanical system (MEMS) light modulating array. This can, in some embodiments, include a MEMS mirror array. In other embodiments, the multiplexing assembly can include an attenuator configured to at least partially block light from one or more of the plurality of channels before it reaches the image sensor. The attenuator can be partially transparent in some embodiments, and can be fully absorbing in other embodiments. The attenuator can be configured to rotate about an axis to place different regions of its area into a light beam path of one or more of the plurality of channels.
In still other embodiments, the multiplexing assembly can include a source of illumination configured to amplify light from one or more of the plurality of channels before it reaches the image sensor. In some embodiments, the multiplexing assembly can include still other components, such as a phase encoding element. The phase encoding element can be any of transparent and reflective. For example, in some embodiments, the phase encoding element can be a wedge-shaped optical element that moves to shift an image formed by one of the plurality of channels. In other embodiments, the phase encoding element can be a non-plano surface that encodes a point spread function of an image formed by one of the plurality of channels by imparting any of an aberration and a diffraction effect into a light wavefront.
The multiplexing assembly can be configured to direct light in a variety of different manners. For example, the multiplexing assembly can simultaneously direct light from each of the plurality of channels onto the image sensor such that light from each channel forms an image on the sensor that fills a focal plane of the image sensor and overlaps with images formed by other channels.
A number of other optical elements can also be included in the imaging device. For example, in some embodiments the multiplexing assembly can be positioned between an optical element and an image plane of the device. In other embodiments, the device can further include a narcissus shield configured to any of partially and fully attenuate light passed therethrough. Moreover, the image sensor can be configured to detect infrared (IR) light and the narcissus shield can be positioned in combination with the multiplexing assembly near an aperture stop of the imaging device in front of at least one optical element. The above-mentioned narcissus shield can, in some embodiments, be configured to any of rotate and translate. In certain embodiments, the imaging device can further include a baffle configured to block stray light from joining light in at least one of the plurality of channels.
A variety of image sensors can be utilized with the imaging device. For example, in some embodiments the image sensor can be configured to detect any of ultraviolet (UV), visible, and infrared (IR) light.
In other embodiments, the imaging device can further include an imaging lens having a fixed effective focal length. Embodiments in which the imaging device further includes an imaging lens having a variable effective focal length are also contemplated. The imaging lens of the imaging device can, in some embodiments, include a plurality of discrete focal lengths. In other embodiments, the imaging lens can include a focal length that is continuously variable over a range of values. Variation of the focal length of the imaging lens can, in some embodiments, cause a projection of a center of the region imaged by each of the plurality of channels to remain fixed relative to the scene. In other embodiments, variation of the focal length of the imaging lens can cause a projection of a center of the region imaged by each of the plurality of channels to shift relative to the scene. In certain embodiments, one or more elements of the multiplexing assembly can be configured to be any of actively steered and phase controlled to move a projection of a center of the region imaged by each of the plurality of channels as the effective focal length is varied.
In other embodiments, the imaging lens can include a variable focal length afocal objective zoom lens configured to direct light into the multiplexing assembly. In some embodiments, the imaging lens can include a variable focal length object zoom lens and the multiplexing assembly can have a fixed focal length. The variable focal length objective zoom lens can be configured to form an intermediate image that is reimaged with the fixed focal length multiplexing assembly.
Moreover, the regions of the scene observed by the imaging device can be arranged in a variety of overlapping and non-overlapping configurations. For example, in some embodiments the plurality of regions of the scene can overlap one another. In addition, the plurality of regions of the scene can be observed from different perspectives. Further, the plurality of regions of the scene can partially overlap one another in some embodiments, and completely overlap with one another in other embodiments. In still other embodiments, the plurality of regions of the scene can be separated from one another.
Other arrangements are also possible. For example, in some embodiments the plurality of regions of the scene can be adjacent to one another. In other embodiments, the plurality of regions of the scene can be arranged to create a panoramic image of the scene.
Any of the features or variations described above can be applied to any particular aspect or embodiment of the disclosure in a number of different combinations. The absence of explicit recitation of any particular combination is due solely to the avoidance of repetition in this summary.
Certain exemplary embodiments will now be described to provide an overall understanding of the principles of the structure, function, manufacture, and use of the devices and methods disclosed herein. One or more examples of these embodiments are illustrated in the accompanying drawings. Those skilled in the art will understand that the devices and methods specifically described herein and illustrated in the accompanying drawings are non-limiting exemplary embodiments and that the scope of the present disclosure is defined solely by the claims. The features illustrated or described in connection with one exemplary embodiment may be combined with the features of other embodiments. Such modifications and variations are intended to be included within the scope of the present application. Further, in the present disclosure, like-numbered components of the embodiments generally have similar features, and thus within a particular embodiment each feature of each like-numbered component is not necessarily fully elaborated upon. To the extent features are described herein as being a “first feature” or a “second feature,” such numerical ordering is generally arbitrary, and thus such numbering can be interchangeable.
As noted above, optically multiplexed imaging is a developing field in the area of computational imaging that involves overlaying multiple images from different regions of a scene onto a single focal plane array or image sensor to form a multiplexed image. A combination of hardware and software processes can be used to disambiguate, or separate and reconstruct, the multiple de-multiplexed images. Optically multiplexed imaging can provide unique advantages over conventional imaging technologies. For example, optically multiplexed imaging systems can create higher resolution and wider field of view images than is possible with conventional imaging technologies because various fields of view are overlaid on one another at full resolution. Further, optically multiplexed imaging systems can be smaller, more efficient, and cheaper than conventional imaging systems of comparable capability because they can utilize a single optical system and focal plane image sensor or array where a conventional imaging system would require multiple sensors or arrays, along with attendant optical elements.
The Applicants of the present disclosure previously disclosed novel devices and methods for optically multiplexed imaging in U.S. patent application Ser. No. 14/668,214 (Atty. Dkt. No. 101328-347), entitled “Devices and Methods for Optically Multiplexed Imaging,” filed Mar. 25, 2015, the entire contents of which are incorporated by reference herein. In one embodiment disclosed therein, an optically multiplexed imaging device included a faceted reflective multiplexing assembly that divides the pupil area of an optical system into a plurality of sub-pupil contiguous multiplexed regions. Each sub-pupil region of the imaging optical system, referred to as channels, could be uniquely encoded to aid in disambiguation.
The teachings of the present disclosure improve the previously-described devices and methods by encoding one or more channels of an optically multiplexed imaging device or system in a manner that is dynamic, or variable over time. Moreover, in certain embodiments the devices and methods described herein can provide dynamically variable image encoding that is both rapid and precise to enable improved performance of the imaging devices and systems described herein. In some embodiments, rapid can mean variation at frequencies at or above a capturing frame rate of an image sensor or array, and precise can mean movement with precision that is less than an angular sampling of an image sensor pixel. The devices and methods for dynamically variable image encoding described herein can be applied to a number of different optical design architectures, as shown in
The devices and methods described herein can provide a number of advantages over prior imaging systems or methods. For example, dynamically variable encoding of one or more channels in an optically multiplexed system can be utilized to provide flexibility for the optically multiplexed imaging system to operate in different modes optimized for specific scene conditions and sensing objectives. By way of further example, an optically multiplexed imaging system can be configured to switch between operating in a first mode that can be suitable for capturing and efficiently disambiguating a sparse scene (i.e., a scene in which objects of interest are sparsely distributed in at least one dimension, such as time or space) and a second mode that can be suitable for capturing and efficiently disambiguating a rich scene (i.e., a more information-rich scene or one in which objects of interest are more numerous and/or closely grouped in at least one dimension). The ability to dynamically switch between operating modes can increase the efficiency of the imaging system and can find particular utility, for example, in surveillance imaging applications (e.g., the first mode can be utilized to view a scene until, for example, activity is detected, whereupon the system can switch to the second mode).
The devices and methods described herein can provide a further advantage of improved signal efficiency and robustness of disambiguation over prior imaging systems. This can enable, for example, efficient snapshot (i.e., single-frame) extended field of view disambiguation of sparse scenes. In addition, extended field of view imaging of a scene can be performed at rates adequate for motion video capture with the improved efficiency and performance of the devices and methods described herein.
The ability to dynamically encode one or more channels of an optically multiplexed imaging system at speeds at or above a capturing frame rate of an image sensor and with precision that is less than an angular sampling of an image sensor pixel can provide a number of advantages, such as the capability of recovering spatial resolutions finer than a pixel sampling. This can add a multiplicative factor to a resolution of an optically multiplexed imaging system, which already utilizes a single image sensor pixel to view multiple points in object space (i.e., in the scene being observed). The devices and methods described herein can also achieve temporal super-resolution due to the ability to rapidly and precisely vary encoding of one or more channels in the imaging device or system. More particularly, temporal information can be recovered at frequencies that exceed the image sensor or array frame rate.
Yet another example of advantages provided by the devices and methods described herein is lower computational complexity during image reconstruction or disambiguation. More particularly, the computational architectures described herein can significantly lower complexity of image reconstruction when compared to conventional techniques for directly solving for an image through, for example, a matrix inversion process. This can be accomplished in some embodiments by utilizing a decoding or disambiguation algorithm that is paired to the method of encoding used during multiplexed imaging. That is, encoding and decoding methods can be paired to provide advantages, such as less computationally intensive disambiguation, faster disambiguation, etc.
Turning to
Note that the various fields of view FOV 1 and FOV 2 shown in
The devices and methods described herein relate to encoding one or more of the channels in a dynamically variable manner using the one or more encoding elements 106. The concept is to encode one or more of the channels—and in some embodiments all of the channels—in a manner that is unique to each channel and dynamic (i.e., variable in time). In some embodiments, such encoding can also be performed rapidly (i.e., at frequencies at or above a capturing frame rate of the image sensor or array of the imager 104) and/or precisely (i.e., in a spatial context movement with a precision that is less than an angular sampling of an image sensor pixel).
Generally speaking, encoding according to the teachings provided herein can include applying a complex function to an electric field of incoming light. A complex function, as used herein, encompasses encoding that can modify the intensity, phase, and/or wavelength of the electric field. By way of example, intensity modulation can be achieved with an attenuating elements such as a physical or electro-optical shutter. Intensity modulation can also be accomplished using any number of spatial light modulating technologies, including, but not limited to, mechanical shutters (e.g., either fully attenuating/absorbing or partially transparent), micro-electromechanical systems (MEMS) such as digital micro-mirror devices (DMDs), eyelid arrays, or other MEMS light modulating arrays, as well as multiple liquid crystal-based technologies. Phase modulation can be achieved by physically or electro-optically changing the phase of light traversing the multiplexing assembly. This can include, for example, physically deforming or moving an optical surface using, e.g., a deformable mirror, a motion controller (e.g., a piezoelectric or other type of actuator), a MEMS device, or by rapidly inserting and removing an optical element encoded with a particular phase profile, such as a tilt to shift the image or a more complex aberration to spatially encode the point spread function ( ). Non-mechanical phase modification is also possible using, for example, a liquid crystal phase modulator. Wavelength encoding of the electric field can be accomplished by way of spectral filters, gratings, prisms, and/or other chromatically dispersive optical media.
The system 200 multiplexes images of FOV 1 and FOV 2 using one or more fold mirrors 206 and one or more beam splitters 208, as shown by light paths 210 and 212, respectively. Moreover, the fold mirror 206 can be coupled to an actuator 214 to tip, tilt, or otherwise move the fold mirror. The actuator 214 can be activated by a motor, piezoelectric mechanism, or any other known mechanism. Importantly, the actuator 214 can be capable of moving the fold mirror 206 rapidly and precisely, as described above. For example, in one embodiment the actuator 214 (or actuators) can rapidly tilt and settle the fold mirror 206 (or mirrors) by a known angle between frames to laterally shift one or more channels (i.e., images of FOV 1, FOV 2, etc.) of the multiplexed image. Encoding in this manner can require that the actuator 214 be capable of completing such a shift in the time between consecutive frame captures by the camera 202. Further, in some embodiments super-resolution imaging can be achieved with this method of encoding by precisely controlling the fold mirrors to sample multiple angles within the field of view of a single pixel. Accordingly, encoding in this manner can also require the actuator 214 be capable of tilting or otherwise moving the fold mirror 206 with a precision that is less than an angular sampling of a pixel in the image sensor or array of the camera 202. In an alternative embodiment, the actuator 214 can be configured to rapidly tilt the fold mirror 206 at a frequency faster than the frame rate of the camera 202 to spatially encode the point spread function by motion blur.
Moreover, encoding of the multiplexed channel images by rapid and precise movements of the fold mirror 206 or encoding element 314 can be varied over time. Variation can be spatial in nature, such as variation in the magnitude and/or direction of image shifting or the spatial structure of an engineered point spread function, or they can be temporal in nature, such as variation in the time delay between applying image shifts to one or more channels or the duration of applying attenuation to one or more channels. Such variation can occur at rates up to or exceeding the capturing frame rate of the system camera/image sensor or other detector.
While the embodiments shown in
In the embodiment shown in
Fold mirrors and pupil dividing faceted mirrors or other reflective elements are not the only possibilities for multiplexing assemblies.
In some embodiments, tiltable or otherwise movable fold mirrors can be utilized to perform sub-aperture multiplexing without a separate multiplexing assembly like the assembly 418 shown in
In a first embodiment of a method for encoding the multiplexed channels, the actuators 814a, 814b can be configured to rapidly tilt (or otherwise move) and settle the fold mirrors 806a, 806b by a known angle between frames to laterally shift each channel of the multiplexed image. Super-resolution capability can be implemented by further precisely controlling the fold mirrors 806a, 806b to sample multiple angles within a field of view of a single pixel (i.e., to move the fold mirrors to different positions with a precision that is less than an angular sampling of an image sensor pixel). In a second embodiment of a method for encoding the multiplexed channels, the actuators can be configured to rapidly tilt (or otherwise move) the fold mirrors at a frequency faster than a capturing frame rate of the camera 802 to spatially encode the point spread function by motion blur.
Note that in the above-described system 800, as well as other systems described herein, disambiguation can be accomplished even when one channel is not encoded. For example, in the embodiment of
The encoding element 916 can be, for example, an attenuator or a phase encoding element. In the case of an attenuator, a signal of a sub-set of channels can be modulated by attenuating the signal either partially or fully. A phase encoding element, on the other hand, can spatially encode the point spread function or laterally shift the image in a known manner. In some embodiments, the encoding element 916 can be inserted into a beam path of one or more channels at a frequency related to the frame rate of the camera to encode the one or more channels. This can be accomplished, for example, with an encoding element 916 that rotates about an axis to periodically place different regions of its area in the path of different channels. In other embodiments, the encoding element 916 can be configured to rapidly translate between two or more known positions that place different regions of its area in the path of different channels.
A multiplexing assembly 1120 in the form of a pupil dividing prism assembly can be positioned between the object plane and the negative lens 1130 in some embodiments. Prism elements included in the multiplexing assembly 1120 can include a single prism, a plurality of single prisms, or one or more achromatic prism groups. An encoding element 1116, such as an attenuator or a phase encoding element, can be positioned between the multiplexing assembly 1120 and the negative lens 1130. In many cases, the effective focal length of this optical design can generally be longer than the overall length of the lens. In addition, it can be desirable in such a configuration for the attenuator 1116 and multiplexing assembly 1120 to define a remote aperture stop of the system 1100. The architecture of the system 1100 can be well suited for use with a higher number of multiplexed channels because the reverse-telephoto lens design can be configured to position an aperture stop near the front of the system, a position that can be ideal for placement of a sub-aperture pupil dividing multiplexing assembly. Using a sub-aperture pupil dividing multiplexing assembly can make practical a larger number of multiplexed channels because sub-aperture pupil dividing multiplexing assemblies can be more compact than full aperture systems that include beam splitters and other components. The system 1100 can also be well suited for use with cameras 1102 that operate in the ultraviolet (UV) range, visible-light range, and short-wave infrared (IR) range (about 0.9 μm to about 2.5 μm), as well as the long-wave IR range (about 8 μm to about 14 μm) when using an uncooled microbolometer for an image sensor.
In the embodiment shown in
In the system 1400, the multiplexing assembly 1422 can be positioned proximate to an aperture stop of the system and between the afocal zoom assembly 1440 and the focusing lens 1404. As the afocal zoom assembly 1440 moves, or as particular elements within the zoom assembly move, the angular magnification between the object space of the fields of view (FOV 1 and FOV 2 in the figure) and the afocal space between the zoom assembly and focusing lens (or lens group) can change. As a result, the projections of the fields of view being imaged can shift as they are magnified (i.e., a center of each field of view being imaged can shift as the field of view is magnified). For example, the imaged fields of view can change from FOV 1 and FOV 2 bounded by solid lines to FOV 1 and FOV 2 bounded by dotted lines in the figure. Shifting the centers of the fields of view in this manner can maintain a constant relative overlap or separation of the imaged fields of view over the zoom range. This is in contrast to the system 1300 described above, where magnification of the fields of view being imaged can result in an overlapping view at sufficient magnification (e.g., if a magnified FOV 1 was shown in
Moving away from zoom lens embodiments,
As noted above with respect to
In an embodiment utilizing a mid-wave infrared (about 3 μm to about 5 μm) or long-wave infrared (about 8 μm to about 14 μm) image sensor with a cooled detector, an encoding narcissus shield can be utilized to attenuate one or more channels. Narcissus is a change in image detection resulting from radiation reflected from lens surfaces back onto the image sensor or detector. A Narcissus shield can prevent such radiation from reaching the image sensor.
The attenuating narcissus shield 2262 can have a concave surface facing towards the camera 2202, as well as partially or fully reflective sections that are pupil matched with the pupil dividing elements in the multiplexing assembly. These attenuating reflective sections can act as narcissus shields (also known as warm shields or warm stops). From the perspective of the cooled detector in the camera 2202, the attenuating reflective sections of the narcissus shield 2262 can appear to have a low temperature because they reflect light from the cold space (within the cold shield 2260) back to the image sensor or detector of the camera. This design can allow a detector sensitive to thermal background radiation to utilize attenuation-based temporal encoding without increased thermal background. Because encoding is performed inside the lens assembly a compact prism-based (or larger mirror-based) multiplexing assembly can be utilized at the relayed entrance pupil. The narcissus shield 2262 can be configured to rotate about an axis at a frequency related to the capturing frame rate of the camera 2202 to place regions with different attenuation characteristics into the beam path of different channels. Alternatively, multiple narcissus shields can be placed on a moving structure that periodically places shields within different attenuation patterns into the beam path to encode a plurality of imaging channels.
The above-described embodiments of optically multiplexed systems include a number of optical designs that can be used to implement the teachings of the present disclosure. As a summary and with reference to the embodiment of
In computational imaging, the cost of image reconstruction can be significant without carefully choosing encoding and decoding methods that pair to provide efficiency, especially when the image being reconstructed is large. The devices and methods described herein can include techniques for encoding multiplexed images and paired decoding methods that enable more efficient image reconstruction/disambiguation. As an example, consider a four-megapixel focal plane multiplexed six times. Using six frames to reconstruct this image can require a solution of a system of 24 million equations, with 24 million unknowns. A direct matrix inversion is computationally expensive, even after taking into account matrix sparsity and the inherent parallelism of this task. To reduce this computational burden, one embodiment of a paired method for encoding and disambiguating a multiplexed image can include shifting the image of one channel in each frame by an integer number of pixels in order to encode the multiplexed image. In such an embodiment, a difference can be taken between two multiplexed frames, which would leave only the moving channel, as the signal from all other channels would drop out. Decoding this channel can require only an appropriately chosen cumulative row and/or column sum of the difference frame, a task that is far more computational efficient than solving or all of the channels simultaneously.
In another embodiment of paired encoding and decoding/disambiguation methods for multiplexed imaging, a similar approach can be taken with respect to amplitude modulation. By way of example, an optically multiplexed imaging system can be configured to rapidly temporally encode each channel at a unique temporal frequency. Such encoding can include, for example, continuously shifting the image back and forth at a different rate for each channel or otherwise inducing a high frequency channel-dependent periodic variation in the image by means of rapid defocus, point spread function (PSF) engineering, or attenuation. Using this approach, each channel can be encoded at a separate frequency. Decoding the image can be achieved by decomposing each pixel's time series by frequency and reading off the image of each channel from its corresponding frequency bin. Such an approach has an advantage in that is can be implemented using a standard focal plane and off-board data analysis components. In other embodiments, the decoding can be performed on-chip with an advanced focal plane with in-pixel frequency discrimination to simultaneously disambiguate N channels at the full frame rate of the camera.
Still another embodiment of a method for encoding image channels in an optically multiplexed imaging system can include attenuating, shifting, and/or defocusing or otherwise point spread function encoding the image at each channel by a known, distinct frequency, and performing a computationally efficient reconstruction via a frequency analysis of the time series for each pixel of the image sensor or other detector. In some embodiments, such a method can include attenuating, shifting, defocusing or otherwise point spread function encoding the image at each channel by a known function of time, thereby measuring the channel images using a matrix with positive, bounded entries. Reconstruction methods that can be paired with such encoding methods can include recovering each channel image from the time series measured in each pixel using a matrix inverse.
A number of variations on the above-described method are possible. For example, in certain embodiments, the above-mentioned function of time for attenuating the image at each channel can include turning each channel on and off at a specific frequency per channel. In such an embodiment, the image reconstruction method can include computationally projecting each pixel's time series onto the corresponding channel frequencies. In other embodiments, the matrix inverse can be carried out within the logic of the pixel of each detector element. This can be accomplished, for example, by having counters that project the measured light onto the rows of the inverse matrix.
In still other embodiments, the above-described channel attenuator can include a light modulating array, such as a micro-electromechanical (MEMS) mirror array, and two focal planes can be used to measure two distinct time series per pixel. The two time series correspond to the two directions that light can be reflected off the MEMS mirror array or other light modulating array. A difference between the two measured time series can be utilized to instantiate a matrix with bounded entries that can be either positive or negative. This matrix can be computationally inverted to recover the images corresponding to each channel.
Another embodiment of a method for encoding and decoding multiplexed image channels can include spatially shifting all but one of the channel during a single integration period to blur the images of those channels. This can enable the image of the single stationary channel to be viewed on a blurry background that can be removed using known techniques.
Still another embodiment of a method for encoding and decoding multiplexed image channels can include continuously shifting all channels along different trajectories and shifting the charge or digital measurements on a focal plane array or other image sensor to follow one of the trajectories. This method can recover a single channel's image on a blurry background. A focal plane array or image sensor capable of multiple simultaneous measurement shifts could simultaneously acquire all images, each on a blurry background.
A still further embodiment of a method for encoding and decoding multiplexed image channels can include differentially rotating each channel's field of view such that image sensor motion can cause each channel's image to move in a different direction on a focal plane. An image sensor of focal plane array capable of charge shifting or digital measurement shifting could recover a channel's image on a blurry background by shifting measurements or charge in the direction along that channel's motion. Again, a focal plane array or other image sensor capable of of multiple simultaneous measurement shifts could simultaneously acquire all images, each on a blurry background.
As noted above, the devices and methods described herein for dynamically variable encoding of channels in an optically multiplexed imaging system can provide a number of advantages, including the ability to operate in a plurality of distinct sensing modes. For example, one such operating mode can be an object detection mode, which is optimized to detect unresolved objects in a sparsely populated scene. A sparsely populated scene is one that includes low levels of objects and background information. Sparse scenes can be intrinsically sparse (e.g., a star-scape) or may be sparse in a given representation (e.g., a time-lapse sequence may be temporally sparse in that the scene does not change much over time, making changes more easily identifiable). Accordingly, an object detection mode can be well suited for use with, for example, start tracking for attitude control and celestial navigation, astronomical observation, and targeting/tracking for surveillance or defense applications.
In an object detection mode, a number of pixels in the focal plane can exceed the number of objects in the scene. An optically multiplexed imager can therefore trade the pixel surplus to simultaneously measure multiple fields of view by uniquely spatially encoding the point spread function of each field of view. The dynamic variation in encoding described herein can allow the encoding to be activated, deactivated, and/or dynamically varied to optimize the sensor for signal collection and/or disambiguation. For example, the point spread function can be encoded to emphasize maximum signal-to-noise ratio for detection and tracking by concentrating light in a single pixel, or alternatively to emphasize maximum frame rate sparse scene disambiguation by channel-specific signal blurring. Furthermore, the ability to rapidly and precisely shift multiplexed channel images to provide for spatial super-resolution and/or to rapidly and precisely modulate multiplex channel images to provide for temporal super-resolution can allow the system to perform enhanced background reduction. If the background has higher spatial or temporal frequencies than the conventional sampling resolution of the camera, these frequencies can alias to cause spurious detections that can be suppressed with super-resolution techniques.
A second operating mode that can be interchangeably switched to using the devices and methods described herein is an imaging mode in which an extended rich scene can be observed and each image sensor pixel can view multiple relevant object points. Such an operating mode can be suited to, for example, use in commercially available cameras for still and motion imagery. In this mode, deterministic disambiguation of the image can require the optically multiplexed sensor to conduct a number of scene measurements equal to the number of channels (e.g., capture 4 frames for a 4-channel system). With conventional cameras the tradeoff is that snapshot imagery may not be possible with a multiplexed imager, however, this is unnecessary in many situations because most modem image sensors can collect a required number of samples at rates much faster than those required for motion imagery. In this operating mode, the dynamically variable rapid and precise encoding methods described herein can allow the encoding to be optimized as a function of frame rate for robust disambiguation of specific spatial frequencies and to achieve spatial and/or temporal super-resolution. The encoding can also be varied to change the computational requirements of image reconstruction and to take advantage of in-pixel computational capabilities of advanced focal plane arrays, as described above. When using an advanced camera with in-pixel frequency discrimination, rapid high frequency encoding can allow all N scene measurements to be conducted simultaneously, thereby producing so-called snapshot imagery in an optically multiplexed imaging device.
The devices, systems, and methods described herein can include repeatedly interchanging between operating in a plurality of imaging modes, such as the above-described object detection mode and imaging mode. Movement from one operating mode to another can be accomplished in a variety of manners. For example, in some embodiments variation in encoding can occur based on information gathered in the imaging system and can occur at a rate slower than the frame rate of the camera. By way of further example, in some embodiments a system operating in object detection mode can switch to imaging mode when activity is detected, such that the activity is captured with higher resolution, etc. In other embodiments, movement from one operating mode to another can occur in response to receiving a command, e.g., a command from a user to focus on a particular area or resume observing a large sparse area, etc.
Systems having the ability to operate in distinct imaging modes and to repeatedly switch between operating modes by dynamically varying multiplexed image channel encoding can have a number of applications. As noted above, systems operating in an object detection mode can have numerous applications in observational astronomy, for attitude control, and for targeting and tracking in defense applications. There are also a number of security applications that require surveillance of a large perimeter that could use systems operating in an object detection mode for motion tracking. Optically multiplexed imaging systems can be particularly suited to these applications because they can provide an extended field of view, improved resolution, dynamically tunable encoding for performance optimization, and opportunities for spatial and temporal super-resolution.
Applications for optically multiplexed imaging systems operating in the above-described imaging mode are also numerous. The ability to provide increased field of view and resolution is universally desirable in the camera market. Optically multiplexed imaging systems of the type described herein can be applied to many areas, including commercial photography, security and surveillance, and scientific imaging. Systems of the type described herein can thrive in applications where image sensors have a high cost per pixel due to the fact that a reduced number of sensors are utilized to image an extended field of view. Exemplary applications can include using photon counting detectors for low-light imaging, optical communication and active imaging (e.g. LIDAR, 3D LADAR or super-resolved imaging with structured illumination), and using infrared focal planes for surveillance, tracking, microscopy, spectroscopy, and in bio-medical applications.
Additionally, a novel characteristic of optically multiplexed imaging systems can be the ability to image a scene that has a continuous or discontinuous field of view with a different aspect ratio than the focal plane array. One example of this is an elongated field of view panoramic video camera. Another exemplary application can be a surveillance camera that can simultaneously look in multiple directions, such as down two hallways or around two sides of a building. Another exemplary application can be creating a multiplexed field of view configuration to efficiently translate from one aspect ratio to another. For example, a common 5:4 aspect ratio of infrared cameras can be converted to a popular 16:9 high definition format display by multiplexing the 5:4 image sensor in a 3×2 or 4×3 configuration. By way of further example, an efficient 2-channel multiplexing of a 1024×1024 camera image sensor could closely match the resolution of ubiquitous 1080p displays (1920×1080). For many uses, such infrared security cameras, this resolution and field of view increase can be invaluable.
Still another application area is stereo-vision or 3D imaging. This is an increasingly important field for robotic navigation, 3D entertainment, and virtual reality. For use in such environments, an optically multiplexed imaging system can be configured to observe the same scene form a different perspective, rather than increasing a field of view by observing different regions of a scene. The parallax between images can be used to passively detect object range. Conventional methods of doing this can lose resolution because images must be spatially separated on the focal plane rather than multiplexed to take advantage of the full resolution of the focal plane array or other image sensor.
Such systems can be well suited to use with advanced focal plane arrays that include on-chip processing capabilities. There is natural synergy when increases in computational demands for optically multiplexed imaging can be compensated for by capabilities of the focal plane array or other image sensor. For example, decreased signal and contrast that can be caused by aperture division and well-sharing can be compensated for by increased bit-depth and integration time. By way of further example, post-processing for image de-multiplexing can be performed on-chip to alleviate down-stream electronics requirements, and in-pixel frequency discrimination can be used for snapshot de-multiplexing, as described above.
One skilled in the art will appreciate further features and advantages of the disclosure based on the above-described embodiments. Accordingly, the disclosure is not to be limited by what has been particularly shown and described, except as indicated by the appended claims. All publications and references cited herein are expressly incorporated herein by reference in their entirety.
This disclosure claims the benefit of U.S. Provisional Application No. 62/165,642, entitled “Rapid and Precise Optically Multiplexed Imaging,” and filed May 22, 2015. The entire contents of this application are incorporated by reference herein.
This disclosure was made with government support under Air Force contract FA8721-05-C-0002. The government has certain rights in the invention.
| Number | Date | Country | |
|---|---|---|---|
| 62165642 | May 2015 | US |
