Patent Application
20100103309
The invention is generally in the field of compressed or compressive imaging.
Compressed imaging is a subfield of the emerging field of compressed or compressive sensing or sampling. Compressed imaging exploits the large redundancy typical to human or machine intelligible images in order to capture less samples than commonly captured. In contrast to the common imaging approach, in which a conventional image with a large number of pixels is first captured, and then, often, this image is loss-compressed digitally, the compressed imaging approach attempts to obtain the image data in a compressed way by minimizing the collection of redundant for some further task data. The further task may be visualization. In other words, the compressed imaging avoids collection of data which will not be of value for human viewing or for some machine processing. Thus, the compressed imaging uses sensing processes allowing production of only loss-compressed, when compared with conventional, images. This production step, if done, is called reconstruction. A few reconstruction methods are known. For examples, the following article may be consulted: E. J. Candes, J. Romberg and T. Tao, “Robust uncertainty Principles: Exact signal reconstruction from highly incomplete frequency information”, IEEE Transactions on Information Theory, vol 52(2), 489-509, February 2006 ([1]). This article is incorporated herein by reference. Image reconstruction from “incomplete” data is possible due that fact that common images are highly redundant, as we experience with conventional compression techniques (e.g. JPEG).
There is a need in the art for compressed imaging techniques. The inventor presents a new technique that can be applied for example in scanning, inspection, surveillance, remote sensing, in visible, infrared or terahertz radiation imaging. The technique may utilize at least one pixel sensor extending in one dimension (vector sensor), and a relative rotation between the imaged scene and the sensor (e.g. by moving (rotating) the sensor relatively to the imaged scene). It should be noted that the relative rotation between the image scene and the sensor is not necessarily obtained by rotation of the sensor (and of the associated optical elements, cylindrical lens or slit as will be described below). The image can be rotated optically using a prism and/or mirror, while the sensor is kept static; or both the image and the sensor are moved (rotated) one with respect to the other.
Pixels of the vector sensor(s) are typically arranged along a straight line, although this is not required. Different pixel subsets may be arranged along parallel lines and shifted by a non-whole number of pixels relatively each other and/or be sensitive to different wavelengths. The sensor is preceded by optics which projects on the sensor a signal (a field, for example the intensity field) indicative of the 2D Fourier transform of the object plane field along the dimension of the sensor (1D Fourier field). Due to the motion of the sensor and/or the image, a series of such 1D field or field strips is obtained, and due to the rotation, the series includes field strips extending in various directions. Thus, the strips can cover 2D Fourier space. However, the spatial frequencies of the object plane field, which contribute to at least one of the sensor measurements, are distributed non-uniformly in orthogonal spatial frequency coordinates (i.e. in the 2D Fourier space): spatial frequencies with larger magnitudes are separated by longer arcs, i.e. by larger spatial frequency distances, in the dimension of rotation. The full set of measurements may be augmented by one of reconstruction processes so that a total number of pixels, which will be shown in the reconstructed image, will be greater than the total number of pixels measured within the series.
It should be noted that using vector sensor(s) may be especially preferable if imaging is to be performed in those wavelength regions, in which matrix pixel sensors are expensive.
As well, matrix pixel sensors may be effectively utilized within the inventor's technique. If this is the case, optics is still setup to project on the matrix sensor a series of “one-dimensional” signals. Imaging can be performed not by all pixels of the matrix at a time, but by a rotating pixel vector, i.e. “vector trace”, within the matrix. Selection or definition of current read-out pixel vector can be done electronically. Imaging with a matrix sensor presents one of the ways to avoid physical rotation of the vector sensor; however, elements or parts of projecting optics still may need to be rotated. The imaging scheme relying on the use of a matrix sensor may help to save energy, increase sensor lifetime, and generate information-dense image data. These properties may be of high value in field measurements or in surveillance as they may relax memory and data transmission resources requirements and imaging system servicing requirements.
The following should be understood with respect to the motion of the vector pixel sensor and vector pixel trace in case of the pixel matrix sensor:
First, in those cases, in which the motion of the sensor relatively the imaged scene is reduced to simple rotation, the sensed spatial frequencies form a regular star in the 2D Fourier space. In some embodiments, the star is at least 16-pointed. The star may be at least 32-pointed. The star envelope is a circle.
Second, the motion may have components other than rotation. For example, the imaging system may be carried by an airplane. In this case, Fourier coefficients acquire a phase shift proportional to the airplane velocity. It should be understood that the unshifted phases can be restored if the motion is known.
Third, in case of an electronic control of read-out pixel vector, the length of the vector pixel trace typically varies for a given pixel non-circular matrix shape, depending on the direction or rotation angle of the pixel vector. In particular, vector trace length will vary with the most typical pixel matrix—rectangular, if all pixels are read-out along sensed directions. However, it should be understood, that not all pixels of a pixel vector, on which light is projected, have in fact to be read-out or kept in memory, in both cases—of the vector sensor(s) and of the matrix sensor. Hence, in either of these cases, star “rays” may be of close lengths or of significantly different lengths. For example, in some embodiments a ratio of the length of the shortest “ray” of the star to the length of the longest “ray” of the star is less than 0.65 or even 0.5, and in some embodiments this ratio is larger than 0.75 or even 0.9.
Additionally, irregularities of the star shape may be associated with variations in angular and radial sampling pitch. In other words, the pitches do not have to be constant. They may be selected to match specific application data acquisition goals, which for example may be to collect spatial frequencies more densely if sensor is oriented in a certain direction, so as to image specific object features. Non-regular (non equidistant) angular or radial sampling may permit better modeling of the acquisition process. For instance, the angular steps can be adapted to capture the Fourier samples on a pseudo-polar grid, which may simplify the reconstruction process and/or may improve its precision. For example, the grid may be selected for optimal presentation of image at the common rectangular grid.
In a broad aspect of the invention, there is provided an imaging system for use in compressed imaging. The system may include at least one pixel sensor having an array of pixels and an optical unit comprising an imaging optics for, projecting light indicative of an imaged scene of an object plane on the sensor, and may be be configured and operable to provide a relative rotation between the imaged scene and a sensor plane, sensed light being therefore indicative of Fourier transform of an object plane light field at various angles of the relative rotation.
The system may include at least one rotative optical element, such as prism and mirror(s), vector sensor and optics, laterally rotating and projecting light from an object plane on the sensor, and be configured to measure data indicative of Fourier transform of an object plane light field at various angles of the vector sensor rotation.
The system may include a pixel matrix sensor and optics, compressively projecting light information (i.e. visual information or image itself) from an object plane on a pixel vector of the sensor, and be configured to affect a direction of the light projection and measure data indicative of Fourier transform of the object plane light field by matching an orientation of the pixel vector within the pixel matrix and the direction of the light projection.
The system may include at least two vector sensors arranged in a staggered configuration. The system may include at least two vector sensors arranged in a stack configuration. The system may include at least two vector sensors with sensitivity peak wavelengths differing for more than 20% of a shortest of the sensitivity peak wavelengths.
The optical unit may include a slit. It may include a cylindrical lens and/or mirror. The optical unit may include a 4-f optical element arrangement. It may include a 2-f optical element arrangement. The system may include a source of radiation for directing emitted radiation onto the object plane, the source of radiation being configured for producing coherent or incoherent radiation. It may include at least one beam splitter and be configured as a holographic system.
The vector sensor may have a sensitivity peak between 90 GHz and 3 THz. The sensitivity peak may be in infrared range with a frequency higher than 3 THz. The peak may be in visible range.
The system may include a control unit configured to initiate measurements by said at least one sensor at predetermined angles of the relative rotation. Alternatively or additionally, the control unit may be configured to reconstruct an image from data measured by the sensor at various angles of its rotation or at various pixel orientations within the pixel sensor. A set of the various angles may be predetermined. The control unit may be configured to reconstruct an image using different optimization techniques such as minimization of total variation optimization technique or a l1 normalization optimization (e.g. minimization) technique from data measured by the sensor for various angles of its rotation, or a combined l1 and l2 optimization technique from data measured by the sensor for various angles of its rotation.
Reconstruction may utilize a maximum a posteriori estimation technique. Reconstruction may be done using a penalized maximum likelihood estimation technique.
In some embodiments, the system comprises a rotative mount associated with at least one of an object, the sensor and the optical unit for implementing said relative rotation.
In some embodiments, the optical unit comprises relay optics for rotating an image being projected relative to the sensor and object planes.
There is also provided, an imaging system for use in compressed imaging, the system comprising a pixel matrix sensor and optics compressively projecting light information, indicative of an image of an object, from an object plane on a pixel vector of said sensor, the system is configured to affect a direction of light projection; said light projection being a indicative of the 2D Fourier transform of the object plane field, and measure data indicative of Fourier transform of the object plane light field by matching an orientation of said pixel vector within the pixel matrix and the direction of the light projection.
In another broad aspect of the invention, there is provided a method for use in compressed imaging, the method including reconstructing an image from data indicative of Fourier transform of an object plane field, a set of spatial frequencies of the data having a star configuration in two-dimensional spatial frequency space, an envelope of the star being of a substantially circular shape.
The method may include reconstructing an image from data indicative of Fourier transform of an object plane light field, a set of spatial frequencies of the data having a star configuration in two-dimensional spatial frequency space, a ratio between a length of a shortest star ray a and a length of a longest ray being less than 0.65 or larger than 0.75.
The reconstruction may be done using minimization of total variation optimization technique or a l1 minimization technique.
In yet another broad aspect of the invention, there is provided a method for use in compressed imaging, the method including sequentially projecting light information, indicative of an image of an object from an object plane on various directions and/or various angles within a rotation plane of a rotative vector sensor and rotating the vector sensor so as to measure data indicative of Fourier transform of the object plane field by the sensor for the various directions of the projected light.
In yet another broad aspect of the invention, there is provided a method for use in compressed imaging, the method including sequentially compressively projecting light information, indicative of an image of an object, from an object plane on various directions within a pixel sensor plane and measuring data indicative of Fourier transform of the object plane field for the various directions by a pixel vector within the pixel matrix.
In order to understand the invention and to see how it may be carried out in practice, a few embodiments of it will now be described, by way of non-limiting example only, with reference to accompanying drawings, in which:
In this section, first, some mathematical ideas applicable within the invented technique are illustrated, and then examples of various suitable optical systems are presented.
Typically, for an image f[n,m] defined on N×N pixel (n=1, 2 . . . N, m=1, 2 . . . N) an equivalent number N×N of Fourier coefficients is needed for reconstruction of the image by direct means (e.g. by inverse Fourier transform). But, in the compressed imaging framework, satisfactory reconstruction can be obtained by using appropriate nonlinear recovery algorithms applied to only part of the conventionally needed. Fourier coefficients. As mentioned above, Fourier coefficients, which will be directly or indirectly measured by the moving sensor and its optical system, will be non-uniformly distributed in the Fourier space, and low spatial frequency components will be sampled denser than high spatial frequency components.
In
relatively the horizontal axis. Along each of the radial lines, frequencies of the set are distributed uniformly. The total number of frequencies on each radial line is 256; frequencies lie within a circle C in the illustration and satisfy the inequality |ω|≦ωmax.
When Fourier coefficients of the image are known at such a set, the image may be reconstructed. This is illustrated by
min∥D{circumflex over (f)}∥1 subject to {circumflex over (F)}(ω,θl)=F(ω,θl) for all {θl}l=0L−1,
where ∥•∥l denotes the l1 norm and D defines the finite difference operator.
In more detail, the following minimization was done by the inventor:
where the finite difference operator D was given by D{circumflex over (f)}[n,m]=√{square root over (|{circumflex over (f)}[n+1,m]−{circumflex over (f)}[n,m]|2+|{circumflex over (f)}[n,m+1]−{circumflex over (f)}[n,m]|2)} and {circumflex over (F)} denotes the Fourier transform of the reconstructed image {circumflex over (f)}. Basically, the reconstruction algorithm seeked a solution {circumflex over (f)} with minimum complexity—defined here as total variation
-and whose Fourier coefficients over the radial strips {{circumflex over (F)}(ω,θl)}l=0L−1 matched those found from the original image {F(ω,θl)}l=0L−1. The reconstruction criterion used by the inventor differed from the criterion used in [1] in that only spatial frequencies within the circle |ω|≦ωmax were used for reconstruction by the inventor. This criterion used by the inventor relates to the technique with which sets of Fourier coefficients can be obtained by compressed imaging.
In accordance with the technique of the invention, direct or indirect measurements of the Fourier transform of the object field may be done with a rotationally moving vector sensor and provide a suitable set of spatial frequencies and Fourier coefficients for satisfactory reconstruction.
Referring to
It should be noted that if the vector sensor is an intensity sensor, then measurements {gθ
Also, it should be remembered that system 100 is just a representative example. Other variations of the 4-f system, or equivalent systems can be utilized (see for example J. W. Goodman, “Introduction to Fourier optics”, chapter 8, or J. Shamir, “Optical systems and processing”, SPIE Press, WA, 1999, chapters 5 and 13 and chapters 5 and 6). As well, different implementations of the 1D Fourier transform may be used (see for example J. Shamir, “Optical systems and processing”, chapter 13). Generally, optics usable in the inventor's technique may include such optical elements as mirrors, prisms, and/or spatial light modulator (SLM).
Referring to
g(r=y′)=K∫f(x′,Myy′)dx′
where K is a normalization factor and My defined the lateral magnification along y′. This integral, which is proportional to the projection of f(x′,y′) on y′, can be recognized as a Radon transform. According to the “central slice theorem” the Radon transform is the 1-D Fourier transform of the slice F(ω,θl). Therefore, the intensity measured by sensor S is gθ
It should be noted herein that, although above description separates steps of object Fourier representation calculation and reconstruction, in practice these operations may be fused together: the Fourier calculation and reconstruction may be presented as a single operation, which will utilize the 2D Fourier transform inexplicitly. Further, this single operation may be described without reference to Fourier transform. It could be said, that the system in
Referring to
Similarly to the case with staggered sensors, the vector sensor can be replaced by multiple adjacent sensors sensitive to different wavelengths, which together with proper optical relay can implement a multispectral imaging system. In case of measurements with coherent light, the wavelength of coherent illumination may be tuned.
As well, a stack of (aligned) vector sensors can be used to collect more light even of the same wavelength. Since the projected signal is “one-dimensional”, aligned pixels will produce the same or close measurements.
Referring to
Referring to
The phase shift interferometer technique, or any other on-line or off-line holographic technique can be used [see for example J. W. Goodman, “Introduction to Fourier optics”, chapter 9]. By this method the sensor measures directly the (encoded) Fourier radial spectrum gθ(r)=ℑω{F(ω,θl}.
It should be noted with respect to holographic schemes, that in such measurements the well known property of conjugate symmetry of Fourier transform of real objects may be utilized: thanks to this property, only half of the complex Fourier coefficients need to be measured. This can be implemented, for example, by using a vector sensor of half size performing a rotational motion of 180° around an axis passing at one of its edges.
Holographic schemes, and more generally the technique of the invention, can as well work with various Fourier-related transforms, for example with the Fresnel transform.
As indicated above, relative rotation between an imaged scene (or an object, presented by object field f(x,y))) and the sensor plane may be obtained by rotation of the sensor plane (by angle θl), i.e. rotation of the sensor S (and its associated optics), and/or of the object and/or rotation of the image itself. For example, the image can be rotated optically using a prism and/or mirror using relay optics, while the sensor may be kept static. This is exemplified in
Reference is made to
Referring to
The reconstruction can be carried out by other optimization technique than the above-mentioned total variation minimization optimization technique. In general, any a-priory knowledge or assumption about the object features can be incorporated into used optimization technique. For common images, high quality results are expected from searches of reconstructed images with minimum complexity. For example, high quality reconstruction may be obtained by using l1 minimization techniques, or by using maximum entropy criterion, or maximum apriori methods with generalized Gaussian priors, or wavelet “pruning” methods. As well, the reconstruction may rely on the maximum a-posteriori estimation techniques or the penalized maximum likelihood estimation techniques.
The above-mentioned total variation minimization may be viewed as an l1 minimization of the gradient together with the assumption that the images to be captured are relatively smooth. Techniques of l1 minimization may be especially convenient, when they can be efficiently implemented by using “linear programming” algorithms—see E. J. Candes, J. Romberg and T. Tao, “Robust uncertainty Principles: Exact signal reconstruction from highly incomplete frequency information”; D. L. Donoho, “Compressed Sensing”, IEEE Transactions on Information Theory , vol 52(4), 1289-1306, April 2006; and Y. Tsaig and D. L. Donoho, “Extensions to Compressed Sensing”, Signal Processing, vol 86, 549-571, March 2006.
The described above compressed imaging technique may utilize also algorithms for motion estimation and change detection efficiently applied to the collected data. For example, “opposite ray algorithms” may be used involving complete rotations of the line sensor (i.e. rotations for 360° rather than for 180°). In a full rotation, two frames are captured. However, motion and change can be still be estimated with only half cycle rotation, by applying tracking algorithms on the data represented as sinogram.
Additionally, the following should be noted regarding the herein described compressed imaging technique. This technique can be applied for capturing not only still images, but also video sequences. As well, within this technique, color imaging and/or imaging in various spectral ranges is allowed.
The following is an example of a method of the invention for fast video acquisition and processing/motion detection. It should be understood that, according to the conventional approach in the field of video imaging, by capturing the projection of a 2D scene with linear sensors, a complete set of projections is acquired every time in order to reconstruct a single frame and then the same procedure is repeated by sampling another disjoint set of projections in order to reconstruct the next frame. The invention enables a much faster frame acquisition rate be using a sliding window over the set of projections and update the next frame by adding the next single new projection while omitting the oldest projection in the sequence of successively captured frames.
The above concept can be used for reconstructing a 2D image and then implementing a dynamic reconstruction of a 3D image from a 2D projection, by using a sliding window for the stream of 2D projections. Thus, in the static case, for an image f(j,k), where j,k=1, . . . , n, and for an encoded Fourier field g(r,θ) where r is a point of the θ projection, we have the following relation:
g(r,Θ)=∫∫f(x,y)δ(r−cos(Θ)x−sin(Θ)y)dxdy.
For a given set of L distinct angles A={θl}l=0L−1 in the range [0,π], the reconstruction approach used for the static case (i.e. the same reconstruction process as described above for the static case, is applied on sliding window of L projections) can be applied to reconstruct the object f from the projection set g(r,Θl) l=0, . . . , L−1.
In the dynamic case, the object field is presented as a function ft(j,k) at a certain time t, and the angle Θj is a periodical expansion of the set A defined above so that θj+L=θj.
As shown in
Then the k'th each reconstructed frame is obtained from the set:
{g(r,Θj)}, j=kL, . . . , (k+1)L−1.
The reconstructed frame rate is 1/(L·Δt), where Δt is the time between two consecutive projection acquisitions.
According to the technique of the invention, the output frame rate can be increased to 1/Δt. As exemplified in
{g(r,Θj)}, j=0, . . . , L−1.
The second frame is reconstructed from the set:
{g(r,Θj)}, j=1, . . . , L.
The k-th frame is reconstructed from the set:
{g(r,Θj)}, j=k−1, . . . , L+K−2.
Thus, a sliding window is applied across the set of projections. Starting from the second frame on every time the set of projections is updated by adding a single new projection and discarding the previous projection. For example, after reconstructing the k'th frame, from the set {g(r,Θj)}j=k−1L+k−2, where k>1, the next reconstructed frame is obtained by discarding projection g(r,Θk−1) and adding the single projection g(r,Θk+L−1).
Those skilled in the art will readily appreciate that various modifications and changes can be applied to the embodiments of the invention without departing from its scope defined in and by the appended claims.
The present invention claims priority from the U.S. Provisional Application No. 60/907,943, filed on Apr. 24, 2007, the disclosure of which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/IL2008/000555 | Apr 2008 | US |
| Child | 12605866 | US |
