Embodiments disclosed herein relate in general to imaging using a digital still or video camera, and in particular to color imaging (also referred to herein as “color photography”) using a monochromatic digital still or video camera having a clear image sensor, i.e. a sensor without a color filter array (CFA).
Color imaging with a digital camera (in both still and video mode) as known today is performed using a pixelated image sensor that has a CFA, most often a Bayer pattern of R (red), G (green) and B (blue) filters. For simplicity, such a camera will be referred to henceforth as “RGB camera”. The CFA pixels are arranged to match image sensor pixels. With the decreasing size of modern image sensor pixels (which approaches the wavelength of light), the signal level for a given photon flux per pixel decreases and the capture of each photon becomes crucial. Color filters prevent most (in some cases ca. 70%) of the photons from reaching the image sensor and therefore affect significantly such parameters as sensitivity and signal-to-noise ratio (SNR). Moreover, the fabrication of CFAs, which requires a number of masking and deposition stages, is costly.
A color image is essentially a weighted combination of RGB “band” images. Thus, a color image can be “constructed” (or “reconstructed”) to match an original imaged object if its RGB components are known. Each R, G and B band is itself a weighted combination of many separate spectral or hyperspectral (“HS”) images at distinct wavelengths (or bands) within the R, G or B band. As used herein, the term “hyperspectral” refers exemplarily to more than ca. 10 wavelengths. Consequently, a RGB color image may be reconstructed from spectral or hyperspectral image data, for example as described in D. H. Foster et al., “Frequency of metamerism in natural scenes”, Journal of the Optical Society of America A, 23, 2359-2372 (2006). However, such reconstruction is currently severely limited by the time and processing resources needed to acquire the HS data and by the time and processing resources needed to reconstruct the color image from the HS data.
PCT patent application PCT/IB2014/062270, filed 16 Jun., 2014 by the present inventors and titled “Apparatus and method for snapshot spectral imaging” teaches snapshot (single shot) HS imaging using a monochromatic digital camera that has a minimal hardware addition in the form of a restricted isometry property (RIP) diffuser element. The digital camera is adapted to provide a large number of spectral images in a snapshot. The spectral images are reconstructed from a single diffused-dispersed (DD) image, which is a single image obtained at the image sensor through the camera and the RIP diffuser. A hardware or software randomizer may be added to the camera with the RIP diffuser to provide a single diffused-dispersed and randomized (DDR) image at the image sensor. The reconstruction of spectral images from a single DD or DDR image described in PCT/IB2014/062270 is performed using compressed sensing (CS) algorithms. More specifically, PCT/IB2014/062270 teaches two dimensional (2D) CS-based spatial-spectral cube reconstruction (SCR) or “2D CS-SCR”.
There would be clearly a tremendous advantage in terms of both camera performance and image sensor fabrication costs if color images in both still and video mode could be obtained with a monochromatic digital camera having a “clear” image sensor that does not have color filters. Moreover, there is a need for and it would be advantageous to have monochromatic digital cameras and associated methods that can provide such images in real time and with restricted processing resources.
The present inventors have determined that a color image in still mode or a series of images (frames) in video mode can be obtained using a monochromatic still or video digital camera with a clear sensor. The color image is obtained in a single shot as with digital cameras having color sensors (in which the pixels are covered with a CFA, normally a Bayer-type RGB filter array). The description continues with detailed reference to a still mode digital camera, but is clearly applicable to video mode. A snapshot DD or DDR image is taken with the monochromatic digital camera. An “HS-originated color image” of size X×Y is reconstructed from K images of size X×Y at K spectral bands (which represent a “data cube” of size X×Y×K) obtained from the DD or DDR image. For simplicity, hereinafter “color image” is used insted of “HS-originated color image”. Some of the K spectral images used in the reconstruction of the color image are interpolated from directly reconstructed spectral images. Specifically, for a set of K spectral images used in color image reconstruction, R spectral images (R<K) are reconstructed directly from the DD image and K−R spectral images are interpolated. The direct spectral image reconstruction is performed preferably using 2D CS-SCR and may involve Bregrman iterations, as described exemplarily in PCT/IB2014/062270. The reconstruction of a K-band HS image J={Jk}, k=0, . . . , K−1, from the available set of bands Jr2={J2
Direct reconstruction of only R out of the K spectral images, followed by the interpolation of the remaining K−R images, provides significant time and computational resource savings, while not degrading significantly the quality of the color image. In particular, the time saving is expressed by the fact that the color image is obtained in a fraction of a second, i.e. in “real time”, thereby also allowing video production.
Since color images cannot be shown, such images are converted herein to grayscale images in all the drawings.
In some embodiments there is provided a method for obtaing a color image of an object or scene using a camera having a pixelated clear image sensor without color filters, the method comprising the steps of obtaining a DD image at the image sensor in a snapshot, processing the DD image to obtain K spectral images in K spectral bands where K≧3, and reconstructing the color image from the K spectral images. The color image may be a still color image or a video color image or color frame. In some embodiments, the step of obtaining a DD image may include obtaining a diffused-dispersed and randomized (DDR) image. The step of processing the DD or DDR image may include processing the DD or DDR image to provide R<K spectral images in R wavebands and using the R images to obtain K−R interpolated images, and the step of reconstructing the color image from the K spectral images may include recontructing the color image using the R spectral images and the K−R interpolated spectral images. The interpolated images may be obtained using a spline subdivision algorithm. The spline subdivision algorithm may be a binary spline subdivision algorithm. Each spectral image may include a set X×Y of multipixels, each multipixel including K wavebands, and the interpolation may be performed simultaneously on the entire multipixel set.
In some embodiments there is provided a camera for obtaining a color image of an object or scene, comprising a lens, a pixelated clear image sensor without color filters, diffuser means to obtain in a snapshot a DD image at the image sensor, and a processor configured to process the DD image into K spectral images in K spectral bands where K≧3, and to reconstruct a color image from the K spectral images. The color image may be a still color image or a video color image or color frame. In an embodiment, the diffuser means include a RIP diffuser. In an embodiment, a camera includes a randomizer used in conjunction with the diffuser means to provide a DDR image at the image sensor. The randomizer may be a hardware randomizer or a software-implemented randomizer. The processor may be further configured to process directly the DD or DDR image into R spectral images in R wavebands (where R<K), to interpolate from the R spectral images K−R interpolated spectral images and to reconstruct the color image using the directly processed R spectral images and the K−R interpolated spectral images. The processor configuration to interpolate K−R interpolated spectral images may include a configuration to interpolate the K−R spectral images using a spline subdivision algorithm. The spline subdivision algorithm may be a binary spline subdivision algorithm.
Aspects, embodiments and features disclosed herein will become apparent from the following detailed description when considered in conjunction with the accompanying drawings, wherein:
Note that while the description continues with specific reference to a DD or DDR image for which enabling details are provided in PCT/IB2014/062270, the methods described herein may be applied to DD images obtained in a snapshot using diffusing means other than a RIP diffuser or randomizers described in PCT/IB2014/062270.
Details of reconstruction step 106 are given in a flow chart
An HS image J is represented by a data cube C=(Pijk) in which i=1, . . . . V and j=1, . . . H are the spatial variables and k=0, . . . , k=K−1, is the spectral variable. The data cube is the collection of “multi-pixels” C={Mij}, i=1, . . . , V, j=1, . . . , H, where for a fixed ī,
Denote by {right arrow over (m)}={m[k]}, k=0, . . . , K−1, a single MP to be reconstructed from incomplete data. We illustrate exemplarily the reconstruction of a single MP using a binary spline subdivision algorithm.
Denote by Bp(t) the B-spline of order p on the grid{l}, which is supported on the interval (−p/2, p/2). An explicit expression for the B-spline is:
Assume, a data vector {right arrow over (m)}r2={m[2rl]}, l=0, . . . , K/2r−1 is available. To approximately reconstruct the MP {right arrow over (m)}, reconstruct a spline Sp(l)=m[2rl] of even order p, which interpolates the data vector {right arrow over (m)}r2 on the grid{l}, and approximates the vector {right arrow over (m)} by the vector {right arrow over ({tilde over (m)})} consisting of the spline values at diadic rational points:
In an embodiment, the reconstruction of the MP may be implemented as follows:
1. Assume j is the smallest natural number such that N=2j>K/2r. Denote the Discrete Fourier transform (DFT) of the sampled B-spline by
2. In order to use the DFT and to eliminate boundary effects, symmetrically expand the data vector {right arrow over (m)}r2 to the vector {right arrow over (x)}={x[l]}, l=0, . . . , N−1 whose length is N.
3. Calculate the DFT of the vector x:
4. Introduce the sequence ŷ of length 2rN
5. Calculate the inverse Discrete Fourier transform (IDFT) of the sequence ŷ
6. Symmetrically shrink the output vector y={y[l]} to the vector {right arrow over ({tilde over (m)})}={{tilde over (m)}[l]} whose length K is equal to the length of the MP {tilde over (m)}. The vector {tilde over (m)} is equal to the values of the spline Sp(t) (Eq. 2) and approximates the MP {tilde over (m)}.
The spline subdivision algorithm described above is able to operate on a set rather than on a single vector. Thus, the operations above are implemented on all the multi-pixels simultaneously. Subdivision of a data cube of size 1367×609×33 takes a fraction of second, i.e. can be considered “real time”.
To investigate an impact of reduction in the number of spectral bands used in the reconstruction on the quality of the color image, several computer simulations were run, and their results were compared with an original RGB image obtained directly from the spectral cube containing 33 original spectral bands (
Processor 610 is configured to perform all the functions described for processor 610 in PCT/IB2014/062270, and specifically 2D CS-SCR from the DD image. In addition, processor 610 is configured to perform interpolation of spectral images from directly reconstructed spectral images as explained above. Exemplarily, processor 610 is configured to perform the interpolation using a spline subdivision algorithm, and in particular a binary spline subdivision algorithm as described above. Processor 610 is further configured to reconstruct a color image using directly processed spectral images and interpolated spectral images. Optionally, apparatus 600 may include an added external (to the camera) digital processor 605 configured to perform some or all of the operations performed by processor 610 above.
The SW randomizer may be replaced by a hardware (HW) implemented randomizer inserted in the optical path between the RIP diffuser and the image sensor. Enabling details may be found in PCT/IB2014/062270.
While this disclosure describes a limited number of embodiments, it will be appreciated that many variations, modifications and other applications of such embodiments may be made. For example, while interpolation of spectral bands and reconstruction of a color image in the simulations above is done only in a last Bregman iteration, these operations may be implemented at the level of each Bregman iteration, starting with a first Bregman iteration. Moreover, Bregman iterations may be interchangeably combined with interpolation at the level of an entire set (or subset) of the iterations, either intermediate or last. In general, the disclosure is to be understood as not limited by the specific embodiments described herein, but only by the scope of the appended claims.
All references mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual reference was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present application.
This application is a 371 application from international patent application PCT/IB2014/063580, and is related to and claims priority from U.S. Provisional Patent Application No. 61/861,982 having the same title and filed Aug. 3, 2013, which is incorporated herein by reference in its entirety.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/IB2014/063580 | 7/31/2014 | WO | 00 |
Number | Date | Country | |
---|---|---|---|
61861982 | Aug 2013 | US |