Since its invention, multiphoton fluorescence microscopy has become a standard imaging technique used to study biological systems in part due to its relative simplicity and intrinsic optical sectioning. Examples of multiphoton fluorescence microscopic systems are described in International Patent Publication No. WO 2006/127967, filed May 25, 2006 by P. T. C. So et al., the entire content of which is incorporated herein by reference. For a multiphoton fluorescence microscope, the effective excitation point spread function (PSF) is proportional to the square of the light intensity. This optical sectioning capability has allowed such microscopes to uncover significant new structural and functional information in biological systems in diverse areas such as early cancer detection, brain activity, and cell signaling. Despite its utility, however, conventional multiphoton microscopy is slow because the diffraction-limited size of the laser beam that scans over a region of interest. The imaging speed is limited by the speed of raster scanning due to the signal-to-noise ratio requirements for a single pixel. This limited imaging speed hinders the study of fast dynamics of biological events, such as neuronal activity, that occur at a faster time scale than the raster rate of the scanning components.
Widefield temporal focusing microscopy (TFM) was developed over the last decade to overcome this limitation. While a point scanning method achieves local excitation by spatially focusing light, temporal focusing manipulates the femtosecond pulse duration, which is compressed only in the focal plane. Outside the focal plane, the stretched pulse duration reduces beam intensity, which correspondingly lowers the multiphoton excitation efficiency and results in fewer photons originating from out-of-focus locations. Another distinction of temporal focusing is the use of a camera or pixelated detector while the point-scanning method generally uses a photomultiplier tube. Point-scanning typically collects emission photons using a photomultiplier tube (PMT) regardless of the location of the originating scattering event and assigns the detected photons to one pixel. Conversely, temporal focusing allows for recording of images using a pixelated camera instead of just a single pixel at a time. However, scattered photons can contribute to photons counted in adjacent pixels resulting in degradation of the image. This degradation limits the use of temporal focusing in deep tissue imaging applications.
Systems and methods described herein provide improved imaging of deep tissue samples using line-scanning and wide-field techniques at high acquisition speeds. In various embodiments, the systems and methods employ multiphoton imaging systems and methods including temporal focusing and fluorescence imaging. In some embodiments, the systems and methods enable imaging deep inside scattering tissue at high speeds by line-scanning or projection of structured excitation patterns into a selected depth within the material being imaged and reconstructing an image from information acquired from the detected images to limit the influence of scattering. Advantageously, the present systems and methods can provide line-scanning or wide-field multiphoton microscopy imaging at high image acquisition speeds while maintaining optical sectioning in thick samples.
The systems of the present application employ a computer system having a system controller or data processor, an image processor, one or more memories that store processing instructions, image data, sample data, etc. The active components connected to the computer system include one or more detection or pixelated cameras, light sources, lens actuation, a sample translation device that can translate the sample in one, two or three orthogonal directions, a spatial light modulator, a digital micromirror device, a diffractive optical element, one or more mirrors, and a grating or filter device that separates a light beam into one or more spectral components. A display can be used to selectively display two- or three-dimensional images of processed data.
The systems and methods described herein can be used to image sections at different depths within tissue samples using illumination wavelengths with the infrared range, for example. For certain embodiments, an illumination pattern defined, for example, by a Hadamard matrix or by one or more scan lines can be used to optically section a tissue sample of an organ such as a mouse brain using a fluorophore inserted into the tissue to target selected features. A reconstruction method can be used as described herein to process the detected image data to form images of the fluorescence distribution in the tissue.
In particular, image data is acquired at each “scan” location or projected illumination pattern. The image data can then be reconstructed based upon the prior knowledge of the projected patterns.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
Systems and methods described herein provide improved multiphoton microscopic imaging methods that overcome limitations due to emission scattering. As compared to conventional point-scanning methods, the systems and methods described herein provide faster data acquisition without sacrificing signal-to-noise ratio in the final image. Systems and methods described herein spatially encode illuminated regions using structured illumination to illuminate a sample and decode spatial information in the sample from the resulting images. In some embodiments described herein, multiline angular scanning temporal focusing (masTF) can be used to increase imaging speed and reduce scattering using photon reassignment techniques. In some embodiments described herein, arbitrary pattern projecting wide-field temporal focusing (APP-WFTF) can be used to encode sections of the thick sample with arbitrary patterns to enable subsequent reconstruction. In some embodiments herein, De-scattering with Excitation Patterning—temporal focusing microscopy (DEEP-TFM) is employed that uses temporally focused, patterned excitation followed by widefield detection with computational imaging. DEEP can be used in combination with multi-photon (e.g., two-photon, three-photon, or higher number of photons) temporal focusing (TFM) to enable imaging at multiple scattering depths within tissue. These techniques provide images of a sample at high throughput with high signal-to-noise ratio at high resolution.
Examples of temporal focusing microscopes that are compatible with embodiments of some systems and methods described herein can be found in the published article entitled “Improvement of axial resolution and contrast in temporally focused widefield two-photon microscopy with structured light illumination” by H. Choi, E. Y. S. Yew, B. Hallacoglu, S. Fantini, C. Sheppard, and P. T. C. So, Biomedical Optics Express, vol. 4, no. 7, p. 995-1005, Jul. 1, 2013, “Wide-field three-photon excitation in biological samples” by Christopher Rowlands et al., Light: Science and Applications, vol. 6, e16255, May 5, 2017, and in “Scattering reduction by structured light illumination in line-scanning temporal focusing microscopy” by Y. Xue, K. P. Berry, J. R. Boivin, D. Wadduwage, E. Nedivi, and P. T. C. So, Biomed. Opt. Express, vol. 9, no. 11, p. 5654-5666, Nov. 1, 2018, the entire contents of the above publications being incorporated herein by reference in their entireties.
A temporal focusing microscopy system 200 with reduced scattering is illustrated in
Because scattered photons arise from scatterers nearby the object of interest, signal photons and scattered photons are recorded at the same time at nearby detection elements of the imaging device 203. This effect causes an apparent “broadening” of structural features of the sample on the imaging device and reduces resolution and contrast in the final image. Systems and methods described herein can account for this broadening effect and improve image resolution.
The structured illumination system 250 can couple multiphoton illumination and structured illumination onto a sample. In various embodiments described herein, the structured illumination system 250 can modulate light into illumination lines. For example, the structured illumination system 250 can include a scan line motion actuator such as a raster scanner 230. The raster scanner 230 can scan the light in one-dimension in some embodiments. In some embodiments, the raster scanner 230 can include a scanning mirror or an array of controlled mirror elements to scan one or more lines of illumination onto the sample. The structured illumination system 250 can include a cylindrical lens 240. In some embodiments, the cylindrical lens 240 can be positioned to focus the light in the same direction as the scanning direction of the raster scanner 230 without modulation in the other dimension. In such embodiments, the light forms a single illumination line. The illumination line can be projected onto a grating 254. In some embodiments, the grating 254 can stretch the duration of the pulse, which is a key component for temporal focusing applications. As described in greater detail below, the scanning direction (i.e., the long axis of the illumination line) can be rotated using a rotation element 255. The single line of light can be split into multiple lines using phase modulation components 210.
In typical approaches to point-scanning microscopy, the laser spot is rastered or scanned over the entire sample to induce scattering or emission of fluorescence photons from the sample as shown in
Theoretically, line-scanning temporal focusing two-photon microscopy has the same theoretical spatial resolution limit as point scanning two-photon microscopy. Scattered photons can reduce signal intensity and effectively add background noise. Point-scanning techniques that use a PMT do not account for the presence of scattering photons as all photons that reach the detector are summed into a single point signal. Moreover, the presence of unwanted scattered photons can reduce the resolution and signal-to-noise ratio (SNR) of line-scanning temporal focusing techniques as compared to the point scanning approach. In various embodiments described herein, the imaging device collects scattered photons from the sample at a plurality of light detection elements and performs an element-wise reassignment of photons in a direction perpendicular to the scanning direction.
The structured illumination system 250 can include phase modulation components 210 or other diffractive optical elements (DOE) to divide a single line into multiple illumination lines. Thus, the structured illumination system couples a plurality of scanning lines onto the sample. The use of multiple illumination lines can improve imaging speed while maintaining high spatial resolution. Imaging while using multiple scanning illumination lines can provide quasi-wide-field imaging. In some embodiments, the multiple illumination lines are parallel. Multiline projection effectively parallelizes the imaging process to improve overall imaging speed compared to single line-scanning temporal focusing. Diffractive optical elements can provide high diffraction efficiency, and this efficiency can enable better equalization of the intensities of light in each illumination line. In other embodiments, the structured illumination system 250 can include a spatial light modulator (SLM) 210 instead of DOEs. The SLM 210 can provide flexible modulation of the number of scanning lines and other parameters such as line spacing. In some embodiments, line spacing is selected so as to avoid cross-talk when lines are too dense (i.e., too closely spaced). In some embodiments, line spacing is selected based upon a desired acquisition speed by increasing the number of lines (i.e., avoiding making the lines too sparse).
In some embodiments, the number of illumination lines or the line spacing can be selected based on considerations such as a thermal damage threshold of the sample or an excitation saturation or photobleaching threshold of the sample (e.g., for one- or two-photon excitation). For example, the choice of fluorophore in the sample can assist the selection of the number of lines or line spacing. For bright fluorophores, low excitation intensity can be enough to achieve a high SNR image. Because the total power is typically fixed, the structured illumination system 250 can be set to parallelize or divide the optical power among more illumination lines that are more densely spaced (i.e., line spacing is smaller).
The structured illumination system 250 can be positioned such that the SLM 210 is located in the Fourier plane of the imaging plane (i.e., the sample plane). When the SLM 210 is placed in the Fourier plane, the phase shifting of the illumination light induced by the SLM 210 can scan the multiple lines in the image plane. Thus, the SLM 210 can act as a raster scanner in some embodiments where the refresh rate of the SLM is high enough. In such a design, an independent raster scanner 230 is not required. An important advantage is thus realized because the design is made more compact. In some embodiments, the refresh rate of the SLM 210 can be up to 1 kHz. In some embodiments that use other phase modulation components such as a DOE, the DOE 210 and raster scanner 230 can be placed at conjugate Fourier planes with respect to one another. The scanning speed of a system 200 including a DOE 210 and raster scanner 230 can be in a range from 1-30 kHz.
The computing device 150 can receive modulated images of the illuminated sample 201 from the imaging device 203. In some embodiments, the computing device 150 can include one or more processors 152, a memory 154, a controller 156, and a network connection 158. The processor 152 can include one or more central computing units (CPUs) and/or one or more specialized image processing units such as graphical processing units (GPUs). In some embodiments, the processor 152 can be configured to process images received from the imaging device and generate reconstructed images of the sample. As described above, the modulated image can include both real signal information and noise information caused by background scattering. Systems and methods taught herein can reduce the impact of background-scattered photons by reassigning the background-scattered photons in the perpendicular direction back to the excitation line to reconstruct a scattering-free image. This is accomplished through a reconstruction process. In the reconstruction, a preliminary step is to estimate the full-width at half-maximum (FWHM) of the scattered light. In some embodiments, an intermediate image is selected for this measurement that has good SNR. For example, an image including some can be selected as the intermediate image. The FWHM of the emission fluorescence from the line is measured in a direction perpendicular to the line direction in the intermediate image. In various embodiments, the FWHM of the emission fluorescence can vary depending upon the depth of imaging within the sample 201. In some embodiments, the sample will be too sparse or dim at certain depths to properly measure the FWHM. In such an instance, the FWHM can be estimated using measurements from adjacent depths.
The reconstruction technique can operate to ameliorate scattered photons in a direction perpendicular to the illumination line and parallel to the scanning direction. In some embodiments, the structured illumination system 250 can rotate the illumination lines through various angles during acquisition of intermediate images. By rotating the illumination lines, a single field of view (e.g., a slice of the sample) can be reconstructed in multiple directions to provide a final image having an isotropic lateral PSF. For example, pixels from reconstructed images obtained using illumination lines that scan along different directions can be stitched together to achieve the final reconstructed image.
In some embodiments, the structured illumination system 250 can include a rotation element 255. Several varieties of rotation elements 255 are illustrated in
By rotating the illumination light, the scanned illumination lines are rotated through different angles with respect to the imaging plane. In some embodiments, the rotation element 255 can rotate the illumination lines at multiple angles (i.e. [0, π/3, 2π/3] or [0, π/2]) to acquire multiple sets of intermediate images at different angles. Rotating the scanned illumination lines can help achieve an isotropic point spread function. The resulting intermediate images can enable reconstruction of a final image without the effects of scattering. In such an embodiment, the reconstruction process wherein pixels are reassigned can be repeated for each series of intermediate images obtained using scanned lines that are rotated through angles. In some embodiments, rotation of the illumination lines can be combined with multiple line projection to provide for scattering-free images at high resolution, high throughput, and high SNR in deep tissue.
Rotating the temporally focused multiple scanning lines is equivalent to rotating the conjugate back focal plane of the microscope. In some embodiments, the rotation element 255 is positioned on the common path of the illumination light and the returning emitted or scattered light from the sample. In these embodiments, the emitted or scattered light from the sample can be descanned and detected by the fixed pixels of the imaging device 203. An advantage of the descanned design is the large variety of imaging devices or detectors that may be used. In the descanned design, the position of the image (whether a single or multiple lines) does not change even as the illumination lines scan and rotate across the whole FOV. As a result, the imaging device 203 can be a camera or one-dimensional detector array such as a linear array of multianode PMTs. In some embodiments, the imaging device 203 can include a two-dimensional array of multianode PMTs. Multianode PMTs can have greater photon sensitivity and faster imaging speed than a camera in some embodiments. Thus, use of the descanned design can improve imaging speed and lead to higher SNR at the price of a more complicated system design. In some embodiments, the descanned design can utilize a second Dove prism coated at the fluorescence wavelength in the detection path that is synchronized with the Dove prism 255 located in the illumination path. In an alternative embodiment, a pair of broadband coated mirrors can be used instead of the Dove prism. In some embodiments, the imaging device 203 can acquire images and transfer the images to the computing device 150 at a sufficient speed to acquire an image for each excitation pattern.
In some embodiments, the rotation element 255 can be positioned only on the path of the illumination light as shown in
While line-scanning approaches can increase image acquisition speed (particularly for multiple lines) over typical point-scanning methods, the use of wide-field illumination (i.e., illumination of the entire slice of the sample at one time) can speed up acquisition still further. However, images obtained using wide-field methods can be noisy because of multiple scattering of the emission photons within the sample. In particular, the excitation light can be at a relatively long wavelength that can propagate through a scattering medium such as biological tissue without much disturbance to the light. However, the emission light wavelength is typically shorter as compared to the excitation light wavelength in temporal focusing multi-photon processes. Thus, the emission light that the microscope is trying to detect is strongly scattered as it propagates out of the medium. Systems and methods described herein can overcome image degradation due to emission light scattering in wide-field microscopes.
In accordance with some embodiments of this disclosure, a modified temporal focusing microscope (TFM) can project arbitrary excitation patterns onto the focal plane using a digital mirror device (DMD). Emission light from the modulated excitation is then detected by a camera. By using near-infrared (NIR) wavelengths for imaging, the excitation patterns maintain their fidelity despite traveling through scattering medium. However, the emission photons are scattered by tissues and the strength of scattering is strongly depth dependent. In particular, this assumption holds for most biological tissue. In practice, TFM images are minimally affected by scattering at or near the surface; as the imaging depth increases, scattering gradually degrades high-frequency information in the images. However, low frequencies in the images are retained for most depths even with wide-field detection. Single pixel detection approaches discard this low frequency information, and hence require a large number of excitation patterns. Systems and methods herein can then combine the information about the excitation patterns (through a calibration process) with the acquired images to computationally reconstructed a de-scattered image. In some embodiments, multiple pattered excitations (and images) are needed to de-scatter a single field of view (FOV). In some embodiments, the number of patterns and images depends on the loss of high-frequency information due to scattering, and hence on the imaging depth.
The excitation source 302 can include a pulsed laser in some embodiments. For example, the pulsed laser can provide femtosecond pulses (e.g., 100-fs temporal pulse width) at near-IR wavelengths (e.g., 800 nm). In various embodiments, the repetition rate of the pulsed laser can be in a range from 1 kHz to 10 MHz. The pulsed laser can emit illumination that is coupled to the sample and enables two-photon imaging of the sample. In some embodiments, the objective 306 can have a magnification of between 5× and 50×. In an exemplary embodiment, the objective 306 can have a magnification of 20×. The objective can have a numerical aperture of 1.0 in some embodiments.
The one or more beam-shaping or beam-steering optics 320 can expand the beam size in various embodiments. For example, the beam-shaping optics 320 can include a beam expander. The beam-steering optics 320 can direct the excitation light output to a diffractive optical element (DOE) 325 in some embodiments. For example, the diffractive optical element 325 can include a grating. The DOE 325 can diffract the beam with a specific diffraction angle for each color of excitation light so that the pulse width is broad at all points other than at the DOE 325 plane and the conjugate image plane or planes of the DOE 325. In some embodiments, the system 300 can include a 4-f lens system following the DOE 325. In some embodiments, the 4-f lens system can control the size of the field of view (FOV) at the sample plane.
In some embodiments, the 4-f lens system can include two lenses wherein a structured illumination system 350 is placed at the Fourier plane of the lens system as shown in one embodiment in
In some embodiments as shown in
In accordance with some embodiments, arbitrary pattern images can be generated using software implemented by the computing device 150. For example, patterns can be generated in Matlab by using the ‘rand’ function to produce 0 or 1 for each pixel. In some embodiments, patterns of 1024-by-1024 pixels that are resized by a factor of 8 can be used. This implementation defines a unit block of 8-by-8 pixels at the DMD 315, which corresponds to 60.8 μm for the length of one side. The corresponding size of the unit block at the sample plane is 0.83 μm, which can be close to the effective diffraction limit of the system in some implementations [λ/(2NA2)]. In this embodiment, the total number of patterns for each imaging session will be 256 to provide a complete basis set. In other embodiments, patterns of 1600-by-1600 pixels at the DMD 315 can be used to enlarge the field-of-view of the system 300 while employing a magnifying lens setup as described below in front of the imaging device 303. The exposure time of the imaging device 303 can be adjusted in some embodiments in the range of 100-500 ms per pattern depending upon the signal intensity of a given specimen. The EM gain of the camera can be set to be 3-100 depending on the signal intensity of a given specimen in some embodiments. In some embodiments, the imaging device 303 can acquire images and transfer the images to the computing device 150 at a sufficient speed to acquire an image for each excitation pattern.
The system 300 can include a second 4-f lens system in some embodiments. The second 4-f lens system can include a tube lens 308 and a high numerical aperture (NA) objective lens 306 to form patterned temporally-focused illumination light at a sample plane 301.
For each projected pattern, an image can be obtained using the imaging device 303. The imaging device 303 can include a plurality of detection elements in various embodiments. For example, the imaging device 303 can include a camera having an array of pixels preferably with 5 million or more pixel elements that can be individually addressed. The imaging device 303 can be a two-dimensional pixelated imaging device such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) pixelated imaging device. In some embodiments, the imaging device 303 can include a one-dimensional or two-dimensional array of multianode PMTs. The system 300 can include a filter 311 to reject light at the excitation wavelength in some embodiments. The images can be sent to the computing device 150 for further processing in some embodiments. For example, an image for each pattern can be saved in memory 154 or a database in the computing device 150 for post-processing.
The computing device 150 can be configured to control pattern projection of the structured illumination system 350, image acquisition using the imaging device 303, image storage, and/or image processing. For example, the computing device 150 can be configured to execute instructions to select a structured illumination pattern. For example, the structured illumination pattern can be selected based on a row of a Hadamard matrix in some embodiments. The computing device 150 can execute instructions to configure the structured illumination system 350 to modulate light from the light source 303 using the selected structured illumination pattern. The modulated light is used to illuminate the sample. For example, the computing device 150 can send the structured illumination pattern to the structured illumination system 350 for local configuration by the structured illumination system 350 or the computing device 150 can directly control the structured illumination system 350 to cause the structured illumination pattern to modulate the light.
The computing device 150 can execute instructions to receive image data corresponding to light emitted or scattered from the sample and received at the plurality of light detection elements of the imaging device 303. The computing device 150 can execute instructions to generate a reconstructed image of the sample by performing a processing operation on the received image data.
In some embodiments, the computing device 150 can execute instructions to receive intermediate image data corresponding to light emitted or scattered from the sample and received at the plurality of detection elements of the imaging device to form an intermediate image. The computing device 150 can execute instructions to collect additional intermediate images wherein each additional intermediate image corresponds to a different structured illumination pattern. The computing device 150 can reconstruct at least a portion of a depth-selective image of the sample by performing an element-wise reassignment or demodulation operation on the intermediate images using the known structure illumination patterns as a priori information as described in greater detail below.
In some embodiments, the excitation source 302 can include a regenerative amplifier (for example, the Legend Elite, Coherent, Santa Clara, Calif., USA) to produce an ultrafast pulsed laser beam with 800 nm center wavelength, 120 fs pulse width, 10 kHz repetition rate, and ˜8 mm beam diameter (1/e2). Beam shaping optics 320 such as a beam expander can magnify the beam to ˜32 mm and direct the beam to the DMD 315 (DLP LightCrafter 9000 EVM, Texas Instruments, Tex., USA). DMD 315 can be used as a diffractive element and pattern generator simultaneously in some embodiments. In some embodiments, the beam can be diffracted from the DMD 315 with an effective grating period of ˜190 lines/mm with an incident angle of 26.4°. Arbitrary patterns can be uploaded onto the DMD 315 using a control program such as the DLP LCR 9000 GUI provided by Texas Instruments. After the DMD 315, the beam can pass through a 4f-lens system 316 including two planoconvex lenses 307, 309. The first lens 307 can have a focal length f=250 mm (LA1461, Thorlabs, Newton, N.J., USA) and the second lens 309 can have a focal length f=125 mm (AC254-125-B-ML, Thorlabs, Newton, N.J., USA). The 4f-lens system 316 projects and magnifies the image of the DMD 315. The images formed by the 4f-lens system (e.g., L1 and L2) are relayed onto the sample plane 301 through the tube lens 308 (f=300 mm; AC508-300-B-ML, Thorlabs) and the objective lens 306 (water immersion 20×/1.0, Zeiss, Jena, Germany). In some embodiments, the system magnification is about 73× based on the focal lengths of tube lenses and the effective focal length of the objective lens. The geometric dispersion of the system can ensure that the pulse width is broad enough to minimize multiphoton excitation except for at the sample plane 301. In some embodiments, the location of objective 306 can be controlled using a positioner or translation stage. For example, an objective piezo positioner (MIPOS-500, Piezosystem Jena, Jena, Germany) can be used.
The two-photon excitation fluorescence from the sample plane 301 is collected by the objective lens 306 in an epi detection geometry and reflected by a dichroic filter 312 (Di03-R635-t3, Semrock, Rochester, N.Y., USA) towards the imaging device 303. A tube lens 317 (f=200 mm; PAC064, Newport, Irvine, Calif., USA) images the fluorescence signals onto the imaging device 303. In some embodiments, the imaging device 303 can include an EMCCD camera (iXon+, Andor, Belfast, Northern Ireland). For multicolor detection, three combinations of filter sets can be used; a blue channel centered at 460 nm (Semrock FF01-460/60-25 and Chroma E530SP-SPC), a green channel centered at 535 nm (Chroma ET535/70M and Chroma ET680SP-2P8), and a red channel centered at 605 nm (Chroma ET605/70M and Chroma E700SP-2P). In some embodiments, an achromatic doublet lens pair (1:2, MAP1050100-A, Thorlabs) can be used to expand the image size onto the camera. This arrangement can be particularly useful when the DMD 315 having 1024×1024 pixels is used for pattern generation. For patterns of even larger pixel size (1600×1600), a 1:1 achromatic doublet lens pair (MAP107575-A, Thorlabs) can be used to ensure the image fits onto the imaging device 303.
Data from the imaging device 303 can be transferred to the computing device 150 using a control program provided by the imaging device 303 manufacturer (e.g., Andor Solis, Andor, Belfast, Northern Ireland) or by control software implemented by the processor 152 of the computing device (e.g., control software implemented using LabVIEW 2015, National Instruments, Austin, Tex., USA).
The approach in systems 300 is to use a combination of wide-field temporal focusing (WFTF) and arbitrary pattern projection generated using the structured illumination system 350 (e.g., by a spatial light modulator (SLM) or a digital micromirror device (DMD)). Patterns can modulate spatial information at a focal plane deep in sample. Thus, spatial features of the object can be demodulated upon wide-field detection despite the presence of emission light scattering. Since scattering is more severe at deeper points in the sample, more patterns are required to obtain additional spatial information at deeper points. However, each pattern must be exposed to the sample for enough time to form an image, and a large number of patterns may lead to longer imaging times that can induce photodamage in the sample. To reduce the total imaging time, a pattern can be formed using multiple, identical sub-patterns which are termed herein as a pixel group. To further reduce the imaging time, systems and methods herein can exploit the sparsity of certain biological targets such as neurons by making compressed measurements as described in greater detail below. Post-processing of these images using demodulation can recover scattering-free images from wide-field excitation. An example encoded microscopy concept is described in “Encoded multisite two-photon microscopy” by M. Ducros et al., Proceedings of the National Academy of Science, vol. 110, no. 32, 13138-13143, August 2013, the contents of this publication being incorporated herein by reference in its entirety.
A mathematical model is provided of the process to generate modulated images for a FOV in the focal plane of the microscope. The following table provides a list of important symbols used in this section for convenient reference:
p,q,m,r
p,q,m
In some embodiments, systems and methods described herein encode spatial information using multiple modulation patterns replicated in the partitioned field of view. Let the imaged focal field of view be represented as X(i,j). For simplicity, assume X contains N×N pixels. Here, i, j∈{1, 2, . . . , N}, are, respectively, the row and the column pixel indices. Let the FOV be partitioned into (N/n)×(N/n) pixel groups where each pixel group contains n×n pixels (see
n=2m/2; m∈{0,2,4, . . . } [1]
In some embodiments, these constraints may be relaxed to a certain extent by selecting appropriate pixel groups that aren't exact squares or/and by selecting another appropriate basis.
Next, let xk,l denote an arbitrary pixel group whose starting pixels are k, l∈{1, n+1, 2n+1, . . . }. Then, by simple pixel indexing:
xk,l=X(k:k+n−1,l:l+n−1) [2]
Define, hm as the [2m×2m] (same as [n2×n2]) sized Hadamard matrix (
Hm,r(i,j)=hm,r(i%n,j%n) [3]
Here, % denotes the modulo operation (i.e., a % b is the remainder of a/b). The illumination pattern Hm,r is then projected to modulate the focal FOV as shown in
Yr(i,j)=X(i,j)×Hm,r(i,j)
In a real-world microscopy setup, Yr is subjected to scattering. Let, sPSF(i, j) denote the scattering point spread function (
Yr,s=Yr⊗sPSF [4]
Here ‘⊗’ denotes the 2D convolution operation. Finally, Yr,s is transmitted through the microscope's optics and is received at the detector of the imaging device. Let, mPSF(i, j) be the point spread function of the microscope. Then, the detected image (
Ŷr˜Poisson(Yr,s⊗mPSF) [5]
Here, ˜Poisson(⋅) denotes an instance drawn from the multivariate Poisson distribution, which includes scattering functions that capture the effect of noise added by the imaging device. An ensemble of n2 such images, (Ŷ1, Ŷ2, . . . , ), can be recorded, each image corresponding to a row of hm. To estimate X from (Ŷ1, Ŷ2, . . . ,
), a demodulation technique can be used.
In embodiments of systems and methods of the present application, noise-limited demodulation is possible for all pixels despite scattering and fixed partitioning of pixel groups. In some embodiments, a robust demodulation technique can be used that uses the ensemble of recorded images (Ŷ1, Ŷ2, . . . , ) to estimate the imaged FOV, X. Let {circumflex over (X)} be the estimate.
First, each pixel group can be treated as a macro pixel by summing the values of all of the pixels in the pixel group:
ŷk,l,r=Σi=kk+n−1Σj=ll+n−1Ŷr(i,j) [6]
For an arbitrary pixel group {circumflex over (x)}k,l and an arbitrary pattern Hm,r, the macro pixel value ŷk,l,r, can be approximated as:
ŷk,l,r=Σi=kk+n−1Σj=ll+n−1Hm,r(i,j){circumflex over (X)}(i,j) [7]
Using the pixel group notation, the expression is modified as:
ŷk,l,r=Σi=1nΣj=1hm,r(i,j){circumflex over (x)}k,l(i,j) [8]
Let {circumflex over (x)}k,l and hm,r be the [1×n2] sized vectored versions of [n×n] sized {circumflex over (x)}k,l and hm,r. Then,
ŷk,l,r=hm,r{circumflex over (x)}k,lT [9]
Here ‘aT’ denotes the transpose of a. Let ŷk,l=(ŷk,l,1, ŷk,l,2, . . . , ŷk,l,n
ŷk,lT=hm{circumflex over (x)}k,lT [10]
Here, hm is the full Hadamard matrix. By definition hm is invertible and, hence, {circumflex over (x)}k,l is given by:
{circumflex over (x)}k,lT=hm−1·ŷk,lT [11]
In the presence of scattering, ŷk,l contains leaked photons from adjacent pixel groups. Thus, the reconstruction from Equation 11 (either by matrix inversion or by using any other reconstruction algorithm) is very inaccurate near the edges of the pixel groups. This phenomenon is well illustrated in
Given this insight, consider a same [n×n] sized arbitrary virtual pixel group, {circumflex over (x)}p,q, that spans across the boundaries of the original pixel group partitions (see the shaded pixel group in
{circumflex over (x)}p,q={circumflex over (X)}(p:p+n−1,q:q+n−1) [12]
Here p, q∈(1, 2, . . . , N) are the indices of the starting pixel of the pixel group. The same mathematics developed above can be developed for such a pixel group with the exception of the hm,r (and its downstream versions). By the definition of the pixel group, the equivalent
By substituting from Equation 3,
Hm,r(p+i−1,q+j−1)=hm,r((p+i−1)%n,(q+j−1)%n)⇒
Thus, it can be seen that
{circumflex over (x)}p,qT=
Importantly, solving this inverse problem gives a noise limited estimate for the middle pixel of the pixel group, i.e. {circumflex over (x)}p,q(n/2, n/2), and hence, using Equation 12, for {circumflex over (X)}(p+n/2−1, q+n/2−1). Because p, q were arbitrarily selected, this result means that any arbitrary pixel in the image can be estimated to a noise limited accuracy despite scattering and fixed partitioning of the FOV. As an example,
Decomposing the imaging FOV into multiple virtual blocks followed by performing a local (e.g., 16×16) pixel Hadamard scan on each block can overcome limitations associated with the frame rate of single-pixel cameras, which is set by the number of illumination patterns needed for the image acquisition and is traditionally governed by the Nyquist rate. The local Hadamard scan can be followed by the measurement of the net fluorescence associated with each block on a binned group of camera pixels. The parallel nature of this architecture enables >80 frames per second acquisition rate independent of the FOV.
In some embodiments, additional denoising can be applied to the images before or after reconstruction. Image denoising works on the principle that the images contain sparse structures and the noise is drawn from a well characterized distribution. As an example, most images are sparse in the wavelet basis and most detected images contain noise drawn from the Poisson distribution. Therefore, once a sparse representation of an image and its noise distribution are identified, one could solve Equation 11 or Equation 15 as a convex optimization problem along with an additional regularization term such as the L1 norm. Alternatively, one could perform image denoising on the original intermediate images acquired by the imaging device prior to applying the reconstruction techniques of the systems and methods disclosed herein. In some embodiments, in order to make the maximum use of object structures that span multiple pixel groups considered herein, one could start with a sparse representation of the image (such as a wavelet representation) and perform an equivalent global optimization of Equation 11 or Equation 15).
The systems and methods described herein can employ a compressive sensing scheme to improve data acquisition speed further without significantly impacting image resolution. In some embodiments, a compressed-sensing-aided temporal focusing approach can help overcome depth limitations on wide-field temporal focusing and push the previous limits on wide-field imaging depth. In a compressive sensing framework, an N dimensional sparse signal, x∈RN, can be decomposed as a sparse vector a∈RN in a basis Was:
x=Wα [16]
According to compressive sensing theory, perfect reconstruction is possible by measuring M incoherent projections (yk, k=1, . . . , M) of x with a sensing matrix ϕ:
y=ϕx [17]
In exemplary embodiments, ϕk (a row of matrix ϕ) can be an illumination pattern that can be generated by the structured illumination device.
In various embodiments, arbitrary pattern projection temporal focusing microscopy methods can be combined with a compressive sensing framework to speed up image acquisition times even further. Because the modulation patterns used (e.g., derived from a Hadamard basis or random basis as described above) can be treated as incoherent measurement matrices, an incomplete sensing matrix can be used when the underlying image contains a sparse representation. In other words, one does not have to use all the patterns in such setting. Rather, a subset of patterns may be used for data acquisition and accurate reconstruction via the compressive sensing architecture. In some embodiments, the same algorithms for denoising mentioned above can be used for this purpose as well. Examples of algorithms and methodologies for denoising and compressed sensing that are suitable for use with the techniques described herein can be found in “Performance Bounds for Expander-Based Compressed Sensing in Poisson Noise” by M. Raginsky et al., IEEE Transactions on Signal Processing, vol. 59, no. 9, September 2011, “Wavelets, Ridgelets, and curvelets for Poisson noise removal” by B. Zhang et al., IEEE Transactions on Image Processing, vol. 17, no. 7, January 2008, and “Poisson Compressed Sensing” by R. Willett and M. Raginsky, available at http://willett.ece.wisc.edu/wp-uploads/2016/01/PCS.pdf, the entire content of all of these publications being incorporated herein by reference in their entireties. Compressive sensing approaches can be especially useful at larger imaging depths where the pixel group size has to be increased to cover the greater extent of scattering that is experienced at such depths. In such cases, the number of patterns needed is large, and the ability to use only a subset of them can be very useful to increase imaging speed. In some embodiments, the level of sparsity necessary to achieve the necessary level of accuracy in the reconstruction can be estimated.
Consider an imaging depth that is affected by scattering so completely that the entire area of the imaging device has to be treated like a single pixel group. In such a case, the number of patterns that are needed in the full basis set is equal to the number of pixels in the camera. As an example, for a camera with 256×256 pixels (i.e., at least 400,000 pixels), a total of 65536 patterns are needed.
Mathematically, the De-scattering with Excitation Patterning Temporal Focusing Microscopy (DEEP-TFM) imaging process can be modeled by the following equation.
Yt(x,y)=sPSF(x,y)*{(exPSF(x,y)*{tilde over (H)}t(x,y))·X(x,y)} (18)
Here x, and y are spatial coordinates; t is the time; exPSF(x, y), and sPSF(x, y) are the excitation and scattering point spread functions; {tilde over (H)}t(x, y) is the tth modulation pattern projected by the DMD. X(x, y) is the object being imaged; Yt(x, y) is the tth image acquired on the camera. The operators * and · represent spatial convolution and pixel wise multiplication, respectively. More particularly:
x∈{x′∈Z: −Nx/2≤x′≤Nx/2−1}
y∈{y′∈Z: −Ny/2≤y′≤Ny/2−1}
t∈{1,2, . . . ,Nt}
X(x, y): Imaged field-of-view of the sample
{tilde over (H)}t(x, y): tth spatial excitation modulation pattern
exPSF(x, y): Excitation point spread function, Σx Σy exPSF(x, y)=1
sPSF(x, y): Scattering point spread function, Σx Σy sPSF(x, y)=1
Yt(x, y): tth acquired image in no-noise conditions
·: Hadamard product
*: 2D convolution over x, y
g (kx, ky): Spatial Fourier transform of g(x, y).
kx, ky: Respective frequencies in x, and y.
Re-writing Equation 18 in the spatial Fourier domain gives:Yt(kx,ky)=
sPSF(kx,ky)·{(
exPSF(kx,ky)·
{tilde over (H)}t(kx,ky))*
X(kx,ky)} [19]
Assuming N pixels (both in the image, Yt, and the object, X), the above equation has 2N unknowns (N in X, and N in
sPSF). As written, each measurement (i.e. an image taken at time point t) appears to provide N equations. However,
sPSF acts as a low-pass filter, and for out-band frequencies (out of the frequency support of
sPSF), the right-hand side of Equation 19 is zero. Now, assume that
sPSF's frequency support has M pixels. Then, each measurement provides M independent equations. Thus, for the above system of equations to be solvable, we need Nt>2N/M measurements. For deep tissue imaging applications, the frequency support of
sPSF changes with imaging depth. As there's no scattering at the surface, at the surface M˜N; we only need Nt=O(1) measurements (here ‘O(·)’ represents the asymptotic ‘big-O’ notation). At deep locations within the tissue where there is no spatial information on the obtained images, M˜1. We hence need Nt=O(N) measurements. Please also note that because of the frequency domain convolution between
{tilde over (H)}t and
X (Equation 19), out-band frequencies (of the frequency support of
sPSF) in X are still sampled on to Yt as long as
{tilde over (H)} captures all possible frequencies of X. It can be shown that a random pattern of {tilde over (H)} satisfies this criterion (described below). Thus, an ensemble of O(2N/M) random patterns, {{tilde over (H)}t}, can be used to fully measure X in DEEP-TFM. Upon such measurement, some systems and methods herein record an ensemble of {Yt} images corresponding to {{tilde over (H)}t}; X can be reconstructed using {Yt} and {Ht} by solving Equation 18 or its corresponding frequency domain representation, i.e., Equation 19.
In practice we make noisy measurements of Y(x, y, t).
{tilde over (Y)}(x,y,t)=˜Poisson(Y(x,y,t)) [20]
Here, ˜Poisson(⋅) represents that the observations are drawn from a Poisson distribution according to a scattering function in the form of a shot noise model.
The imaging task is twofold: designing the modulation patterns {tilde over (H)}(x, y, t) so that Equation 20 constitutes a fully determined system and estimating X(x, y) from noisy observations {tilde over (Y)}(x, y, t).
In some embodiments, the design criteria for {tilde over (H)} can include the following considerations:
1. Select {{tilde over (H)}t(x, y)} such that ∀(x, y), {tilde over (H)}(x, y)∈{0,1}
2. Select {{tilde over (H)}t(x, y)} such that ∀(x, y), Σt(2{tilde over (H)}t(x, y)−1)=0
3. Select {{tilde over (H)}t(x, y)} such that ∀t and (kx, ky), {tilde over (H)}t(kx, ky)>0
Implementing the modulation patterns ({{tilde over (H)}t} in Equation 20) in some optical hardware has an important restriction. For certain embodiments such as those that use a digital micromirror device 315 for pattern projection, the elements of {tilde over (H)}t can only be binary entries, i.e. {tilde over (H)}t(x, y)∈{0,1} as embodied above in Design criterion 1. However, this condition causes Equation 19 to become noise unstable as described in greater detail in a journal article by Maxim Raginsky, et al. entitled “Compressed sensing performance bounds under Poisson noise.” IEEE Transactions on Signal Processing 58.8 (2010): 3990-4002, the entire contents of which in incorporated herein by reference.
Therefore,
{tilde over (H)}t=({tilde over (H)}t+1)/2
⇒Ht(x,y)∈{−1,+1} [21]
Substituting into Equation 20 gives:
2Y=sPSF*{(exPSF*(Ht+1))·X}
⇒2Y=sPSF*{(exPSF*Ht)·X+(exPSF*1)·X}
⇒2Y=sPSF*{(exPSF*Ht)·X+X}
⇒2Y=sPSF*{(exPSF*Ht)·X}+sPSF*X [22]
Please note that we have dropped the (x, y) indexing and written the variables in their proper matrix form for convenience. Here 1 is the matrix with all 1 entries.
Summing Equation 22 over all t∈{1, 2, . . . , Nt} gives
Σ∀tYt=sPSF*{(exPSF*Σ∀tHt)·X+NtX} [23]
Let's select {Ht} such that Σ∀t Ht=0 (Design criterion 2). Here 0 is the all 0 matrix of the same size as Ht. Then we can simplify Equation 23 as,
Σ∀tYt=Nt(sPSF*X)
Here
2Yt−
Ýt=sPSF*{(exPSF*Ht)·X} [25]
Using the convolution theorem, Equation 25 as represented in the spatial frequency domain gives:Ýt=
sPSF·{(
exPSF·
Ht)*
X} [26]
While sPSF acts as a low-pass filter, high spatial frequencies in
X can still be captured by the convolution with
exPSF·
Ht given that
Ht contains all possible frequencies as shown in
Next, we write Equation 25 as a linear system. The convolution operation can be constructed as a matrix multiplication, where one of the inputs is converted into a Toeplitz matrix.
VexPSF*H
Here, CexPSF is the equivalent ‘convolutional matrix’ of exPSF; VH
VÝ
Here, CsPSF is the equivalent ‘convolutional matrix’ of sPSF; VX is the column stacked version of X; VÝ
Similarly, the corresponding frequency domain form, i.e. Equation 26, can be written as,=
[29]
Expanding Equation 28 to include Nt patterns gives,
Here, 0 is the all zero matrix of the same size as CsPSF.
Similarly, the corresponding frequency domain form, i.e. Equation 29, can be written as,
Now, the Equations 30 and 31 are linear systems and can be used to solve for VX and iteratively.
Images were obtained of a 200 μm thick muscle tissue section stained for nucleus (blue channel, Hoechst 33342) and F-actin (red channel, Alexa Fluor 568 Phalloidin). The animal procedure (transcardial perfusion) was approved by the Massachusetts Institute of Technology Committee on Animal Care and meets the NIH guidelines for the care and use of vertebrate animals. Mice were deeply anesthetized with 1.25% avertin solution (350 mg/kg intraperitoneal) and transcardially perfused with phosphate buffered saline (PBS) containing 4% paraformaldehyde. After perfusion, thigh muscle was excised and post-fixed in 4% paraformaldehyde overnight. Muscle tissue was cryoprotected in 30% sucrose for 48 hours, embedded in Optical Cutting Temperature formulation (OCT, Tissue Tek), frozen at −20 degrees Celsius, and sliced at a thickness of 200 μm on a cryostat. Frozen sections were immersed in PBS for staining. Muscle slices were permeabilized in a solution of 1% Triton-X-100 in PBS for 20 minutes at room temperature with gentle shaking. Slices were then incubated in a working solution of the dyes (dissolved in PBS) for 20 minutes at room temperature with gentle shaking. Excess dye was removed by washing slices in PBS 3 times (6 minutes per wash, with gentle shaking at room temperature). Slices were then mounted on slides using Fluoromount-G or Vectashield as mounting media. Slides were coverslipped, and slides containing Vectashield as the mounting medium were sealed along the edges of the coverslip with clear nail polish. Slides were allowed to dry for at least 48 hours before imaging.
The FOV was nearly 150×150 μm2 with 256×256 pixels. All DEEP-TFM reconstructions in
Notably, DEEP-TFM and other systems and methods described herein are currently the only computational wide-field multiphoton imaging methods whose frame rate is independent of the size of the FOV. For example, millimeter-sized FOVs at diffraction limited resolution can be achieved with no sacrifice in speed. Systems and methods described herein uniquely provide flexible, depth-dependent imaging speeds wherein shallow imaging is almost single-shot while the deep imaging speed is depth-optimized. For instance, taken together, a speed-up of more than three orders of magnitude may be achieved over PSTPM for a volume of 256×256×156 px3 when the same acquisition time is used per measurement. In some embodiments, systems and methods described herein such as DEEP-TFM can employ modern compressive sensing theory with suitable image priors to achieve an additional ˜10× speed up.
Femtosecond laser pulses with high pulse energy (˜μJ−mJ) as used in TFM is what enables wide-field excitation. Depth resolution is achieved by controlling optical dispersion so that the pulse width rapidly broadens away from the focal plane resulting in low two-photon excitation efficiency out of plane. However, emitted photons, due their shorter wavelengths than excitation photos, are strongly scattered by the tissue. As a result of wide-field detection with an area detector, some scattered emission photons in TFM are assigned to incorrect pixels on the detector resulting in resolution degradation and signal-to-noise ratio reduction. As illustrated in
In some embodiments, calibration samples can be used to provide a baseline against which sample images can be compared. For example, a thin quantum dot layer can be used for the calibration of patterns for green (535 nm) and red channels (605 nm). A thin, fluorescent layer of green quantum dots (supplied by QDVision, Lexington, Mass., USA) dispersed in hexane (10 μL) are dropped onto a coverslip (thickness 170 μm) and allowed to dry. The coverslip is affixed to a glass slide and sealed by transparent nail varnish. Similarly, a thin DAPI solution layer can be used for the calibration of patterns for the blue channel (460 nm). Saturated DAPI solution in 1:1 mixture of deionized water and DMSO is dropped in a pre-holed spacer (120 μm thick, Secure-Seal Imaging Spacers, Grace Bio-Labs, OR, USA) onto a glass slide and a coverslip was placed on top of the spacer. The coverslip sealed using clear nail varnish.
A calibration process in accordance with embodiments described herein can give the ensemble of patterns, {{tilde over (H)}t}, used to modulate the spatial features. The imaging experiment gives, the ensemble of measurement images, {Yt}. Then the reconstruction of the de-scattered image, X, is possible from solving the set of Equation 18 equations (or the set of Equation 19 equations) as described above. However, the constituting set of equations in Equation 18 (and its corresponding frequency domain form in Equation 19) are not linear but rather quadratic with respect to the unknowns (X and sPSF). To solve this system, one could first assume a form for sPSF and then Equation 18 becomes a linear system that can be solved for X with commonly available linear-optimization methods. When a solution for X is found that can be substituted in Equation 19, a similar linear system is created that can be solved for sPSF (and hence for sPSF). Thus, a proper solution for X can be iteratively found. In some embodiments, only one iteration is performed assuming a canonical form for sPSF. This can result in visually accurate reconstructions. Some embodiments can employ a two-step iterative shrinkage/thresholding algorithm (TwIST) to solve the above linear equations.
In some embodiments, the systems 200, 300 can employ line-scanning and structured illumination (SI) techniques to increase spatial resolution and obtain axial sectioning capability. SI can increase lateral resolution or even achieve super-resolution in some instances. SI encodes sample information, shifting high frequency information, which is outside of numerical aperture originally, back to the range within the numerical aperture by using multiple intermediate images to reconstruct the final image. SI combined with temporal focusing can improve the axial sectioning capability of widefield and line-scanning two-photon microscopy in some embodiments. In some examples, SI can be modulated on the conjugate imaging plane or Fourier plane. For example, the digital mirror device 315 (DMD) placed at the conjugate imaging plane can work as a grating in the system 200, 300 while generating SI at the same time. A spatial light modulator (SLM) 210 placed at the conjugate Fourier plane can generate SI by holographic phase patterning.
In accordance with various embodiments described herein, HILo-Line-scanning temporal focusing microscopy (HiLL) can be used to obtain images with better resolution and contrast and improved axial sectioning capability. A HiLL system 400 is illustrated in
The HiLL system 400 is similar to the systems 200, 300 described previously with respect to
The excitation source 402 can be a laser in some embodiments. The laser can generate femtosecond pulses at a wavelength of 1035 nm (repetition rate 1 MHz, spectrum width ±5 nm), for example, the Monaco laser (Coherent Inc., CA, USA). The scanning mirror 405 (6350, Cambridge Technology, MA, USA) mechanically scans the beam along the y-axis. The cylindrical lens 407 (f=150 mm) focuses the beam into a line on the grating 454 (20RG1200-1000-2, Newport Co., CA, USA, 1200 grooves/mm). The incident angle θi is about 73°, so the 1st order diffraction angle is about 16°. The grating generates dispersion along the x-axis. The relay lenses 416 can include a first lens with focal length f=300 mm and a second lens with focal length f=75 mm.
To generate structured illumination, a grid 435 is placed on the conjugate image plane. In some embodiments, the grid 435 can be a Ronchi ruling (38-258, Edmund Optics, NJ, USA). In some embodiments, the grid 435 can be mounted on a magnetic mounting seat so that the grid 435 can be manually removed to enable uniform illumination. In some embodiments, the time to remove the grid 435 from the magnetic mounting seat can be negligible compared to the imaging time (e.g., 1.6 s per frame). The contrast of the stripes in the grid 435 is important for the HiLo process. Thus, a grid 435 can be chosen having 10 line-pair/mm, which is equivalent to a period of 2.88 μm on the image plane. This period is larger than the PSF.
The system 400 can include a tube lens 408. The focal length of the tube lens can be 300 mm in some embodiments. In the back aperture, the beam size is about 20×20 mm. The system 400 can overfill the back aperture so that the spatial resolution of HiLL microscopy is comparable with TPLSM. The FOV is about 250×250 μm2 in some embodiments. A second tube lens 417 can be included in the detection path. The focal length of the tube lens 417 can be f=350 mm. The system magnification can be about 40× based upon the objective magnification and the focal length of tube lenses. The image can be detected at an imaging device 403. In some embodiments, two imaging devices 403 can be employed simultaneously to work as, for example, red channel and yellow channel. In some embodiments, the imaging device 403 can be a two-dimensional pixelated imaging device such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) pixelated imaging device. In some embodiments, the imaging device 403 can acquire images at a sufficient speed to acquire an image for each excitation pattern.
The axial confinement of LineTFM is the same as that of TPLSM. Thus, it can be described as
where ZR represents the Rayleigh range of the beam and Δz refers to the displacement from the focus. The FWHM of TPE(z) defines the axial resolution of LineTFM. Structured illumination in HiLL can improve axial sectioning and reduce scattering. We calculated the SI intensity change along the z-axis according to the properties of a defocused optical system. In incoherent detection, the optical transfer function (OTF) of the system represents the contrast of stripes on the image plane by scanning a thin sheet of fluorescence along the z-axis. We measured the spatial frequency in the image plane Tg=2.88 which is converted to normalized SI frequency s=λ/(Tg·NA)=0.378. The OTF along z-axis is
Where A is the amplitude correction related to SI frequency s, and k is the wave vector. The SI pattern on the focal plane can be described as sinusoidal function. Both the lateral and axial periods of stripes are larger than the PSF of LineTFM. Because LineTFM overfills the back aperture of the objective, the resolution of HiLL is the same as LineTFM in our experiments. For a widefield temporal focusing system, if the axial width of the stripes is larger than the PSF of LineTFM generated by a higher frequency grid, the axial resolution of HiLL could be further improved but the contrast of the stripes would be lower.
The key of the HiLo technique is that SI only modulates objects that are in focus but not objects that are out of focus. The image acquired under uniform illumination is
U(x)=Iin(x)+Iout(x) [34]
where Iin and Iout are photons in focus and out of focus, respectively. The images acquired under sinusoidal SI are represented as
S(x)=A[Iin(x)(1+M cos(kgx))+Iout(x)],kg=2π/Tg [35]
Because Iout is not modulated, we can remove Iout by subtracting Equation 35 from Equation 34, that is,
This product is a low resolution version of Iin. A low-pass (LP) filter can be applied to D(x) on the Fourier plane, e.g., a Gaussian filter. Different filters can slightly influence the reconstructed image contrast. The cutoff frequency kc is smaller than 1/Tg; normally we choose half of the modulation frequency. So the low-passed image is
ILP(x)=−1{
{D(x)}×LP(kx)} [37]
To recover the high resolution part of Iin, a high-pass (HP) filter can be applied to U(x), because a single structured illumination image lost information smaller than the fringes. The HP filter satisfies HP(kx)=1−LP(kx). The high-passed image is
IHP(x)=−1{
{U(x)}×HP(kx)} [38]
The reconstructed image is a combination of ILP and IHP:
IHiLo(x)=ηILP(x)+IHP(x) [39]
where η adjusts the relative intensity of the two images to combine them seamlessly.
Dendritic branches are also illustrated at different depths.
The image stack in
Generating structured illumination is an important step in HiLL. The structured illumination can be generated using the grid 435 (e.g., a Ronchi ruling) in the conjugate image plane. Manual removal of the Ronchi ruling can be performed after finishing the structured illumination image stack. The manual method works well when the sample does not change during the imaging time (e.g., about 90 s for one image stack in some embodiments). In some embodiments described herein, the grid 435 can be associated with a motor controlled mount to enable the grid 435 to in and out of position in the optical path for each frame acquisition (e.g., about 1.6 s exposure time per frame in some embodiments).
HiLL can image the same sample much faster than TPLSM with similar signal to background noise and resolution. The imaging speed of HiLL is compared with TPLSM under the same pulse energy and the number of pulses of a diffracted limited spot. To visualize weak fluorescent structures, such as dendrites and spines, the imaging average power and exposure time can be selected to achieve sufficient SNR from weak fluorescent structures inside the sample while staying below the power threshold of two-photon excitation saturation.
An example of the speed-up obtainable using HiLL is described here. The excitation source of TPLSM is a Ti:Sapphire femtosecond laser. The average power out of the objective is about 40 mW, repetition rate is 80 MHz, and the dwell time of each pixel is 40 μs. The dwell time is longer than usual because the sample is dim. Thus, the maximum pulse energy for fluorophore such as most fluorescent proteins is about 40 mW/80 MHz=0.5 nJ without fluorescence saturation. The number of pulses is 40 μs×80 MHz=3200. The step size is 250 nm/pixel. A 256×256 μm2 field of view (FOV) requires about 1000×1000 pixels. So, the imaging time of one frame is 40×1000×1000=40 s.
The radial resolution of LineTFM is about 0.5 μm, and the FOV is about 250×250 μm2. The sampling rate of each diffracted spot is 3 pixels. Thus, LineTFM parallelizes N=250/0.5=500 diffracted limited spots in a single line. Pulse energy per line equals to 0.5 nJ×500=250 nJ. The repetition rate of the laser is 1 MHz, so the average power per line is po=250 nJ×1 MHz=250 mW. For the same number of pulses, the dwell time of a single line is 3200/1 MHz=3.2 ms. The scanning direction also has 500 diffracted limited spots, so the frame exposure time is 3.2 ms×500=1.6 s. We need to get one stack with uniform illumination and another stack with structured illumination, so the imaging time of HiLL is 3.2 s. Compared to the frame rate of TPLSM, HiLL is about 40/3.2=12.5 times faster.
The PSF of HiLL, LineTFM and line-scanning two photon microscopy (LineTPM) are measured, respectively, using 200 nm red fluorescent beads (Carboxylate-modified Microspheres, red fluorescent (580/605), ThermoFisher Scientific, MA, USA). The results are illustrated in
The lenses and gratings can be selected to achieve a sufficiently large FOV and overfill the back aperture at the same time. The beam shaping optics 420 can collimate the beam and expand it to the size of xo. After that, along the x-axis, the beam is collimated until the grating 454 adds dispersion to it. The incident angle to the grating 454 is θi. Along y-axis, the beam is focused by cylindrical lens 407, and the grating 454 is on the focal plane of the cylindrical lens 407. The beam size at back aperture and FOV are calculated using matrix methods.
After the grating 454, the input beam is [xin, αin]′, where
dλ is the spectrum width of laser, which is 10 nm here. d is the groove spacing of the grating 454, which is 1/1200 mm. θ is the diffraction angle of the grating, which is related to the incident angle by sin θi+sin θ=mλ/d, where m=1 for the 1st order diffraction.
Then the lens matrix is
and the propagation matrix is
In our setup, f3=300 mm, f4=75 mm, f5=300 mm. Thus, the size of beam along x-axis on the back aperture is:
xBA≈20 mm. For y-axis,
To match with x-direction beam size, we choose the focal length of the cylindrical lens 417 to be 150 mm, then yBA is about 20 mm on the back aperture. Both directions overfill the back aperture. The objective 406 can be a 20× objective (0.95NA from Olympus) in some embodiments, so the FOV is:
which is about 250 μm. yFOV can be easily adjusted by changing the voltage on the scanning mirror 405. We choose the focal length of tube lens 417 to be 350 mm according to the pixel size of the imaging device 303 to fulfill the Nyquist theorem.
The method includes illuminating the sample with the one or more spatially and temporally focused line beams of light including encoded structured illumination to cause light emission or scattering from the plane at the selected depth within the sample (step 2808). The method includes acquiring a second image including scattered or emitted light from the sample illuminated by the one or more line beams including encoded structured illumination (step 2810). The method includes extracting low spatial frequency information from the first image and high spatial frequency information from the second image (step 2812). The method includes combining at least the low spatial frequency information and the high spatial frequency information to reconstruct at least multiphoton image data related to the object at the selected depth (step 2814). The method includes assemble a plurality of reconstructed images acquired from different illumination angles to provide a reconstructed image having an isotropic lateral point spread function (step 2816).
While the present inventive concepts have been described with reference to particular embodiments, those of ordinary skill in the art will appreciate that various substitutions and/or other alterations may be made to the embodiments without departing from the spirit of the present inventive concepts. Accordingly, the foregoing description is meant to be exemplary and does not limit the scope of the present inventive concepts.
A number of examples have been described herein. Nevertheless, it should be understood that various modifications may be made. For example, suitable results may be achieved if the described techniques are performed in a different order and/or if components in a described system, architecture, device, or circuit are combined in a different manner and/or replaced or supplemented by other components or their equivalents. Accordingly, other implementations are within the scope of the present inventive concepts.
This application is a continuation of International Patent Application No. PCT/US2019/015272, filed Jan. 25, 2019, which claims priority to U.S. Provisional Application No. 62/622,692, filed Jan. 26, 2018, the entire contents of each application being incorporated herein by reference. This application is also related to U.S. patent application Ser. No. 16/258,307, filed Jan. 25, 2019, the entire contents of which is incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5034613 | Denk et al. | Jul 1991 | A |
9599805 | Chiang et al. | Mar 2017 | B2 |
9867525 | Yew et al. | Jan 2018 | B2 |
10042150 | Brown | Aug 2018 | B2 |
20080130093 | Silberberg et al. | Jun 2008 | A1 |
20080192231 | Jureller et al. | Aug 2008 | A1 |
20110025870 | Baraniuk et al. | Feb 2011 | A1 |
20140128743 | Yew et al. | May 2014 | A1 |
20150362717 | Chen et al. | Dec 2015 | A1 |
20200284724 | Dholakia | Sep 2020 | A1 |
Number | Date | Country |
---|---|---|
107505695 | Dec 2017 | CN |
2006127967 | Nov 2006 | WO |
2012135823 | Oct 2012 | WO |
Entry |
---|
Accanto et al., Multiplexed temporally focused light shaping for high-resolution multi-cell targeting. bioRxiv. Retrieved online at: https://www.biorxiv.org/content/early/2017/11/08/216135. 25 pages. Nov. 8, 2017. |
Alemohammad et al., Widefield compressive multiphoton microscopy. Opt Lett. Jun. 15, 2018;43(12):2989-2992. |
Candes et al., Stable Signal Recovery from Incomplete and Inaccurate Measurements. arXiv:math/0503066. 15 pages. Jun. 2005. |
Choi et al., 3D-resolved fluorescence and phosphorescence lifetime imaging using temporal focusing wide-field two-photon excitation. Opt Express. Nov. 19, 2012;20(24):26219-35. |
Choi et al., Improvement of axial resolution and contrast in temporally focused widefield two-photon microscopy with structured light illumination. Biomed Opt Express. Jun. 3, 2013;4(7):995-1005. |
Dai et al., Adaptive compressed 3D imaging based on wavelet trees and Hadamard multiplexing with a single photon counting detector. arXiv:1709.05961. 11 pages. Sep. 15, 2017. |
Ducros et al., Encoded multisite two-photon microscopy. Proc Natl Acad Sci U S A. Aug. 6, 2013;110(32):13138-43. |
Escobet-Montalban et al., Wide-Field Multiphoton Imaging Through Scattering Media. arXiv:1712.07415v1. 19 pages. Dec. 20, 2017. |
Escobet-Montalbán et al., Wide-field multiphoton imaging through scattering media without correction. Sci Adv. Oct. 12, 2018;4(10):eaau1338. 9 pages. |
Hirano et al., Recent advancements in structured-illumination microscopy toward live-cell imaging. Microscopy. Aug. 1, 2015;64(4):237-49. |
Isobe et al., Enhancement of lateral resolution and optical sectioning capability of two-photon fluorescence microscopy by combining temporal-focusing with structured illumination. Biomed Opt Express. Oct. 10, 2013;4(11):2396-410. |
Kurtz et al., Application of multiline two-photon microscopy to functional in vivo imaging. J Neurosci Methods. Mar. 15, 2006;151(2):276-86. |
Lum et al., Fast Hadamard transfers for compressive sensing of joint systems: measurement of a 3.2 million-dimensional bi-photon probability distribution. arXiv:1505.05431v3. 14 pages. Jul. 24, 2015. |
Meng et al., Fast two-snapshot structured illumination for temporal focusing microscopy with enhanced axial resolution. Opt Express. Sep. 18, 2017;25(19):23109-23121. |
Oron et al., Scanningless depth-resolved microscopy. Opt Express. Mar. 7, 2005;13(5):1468-76. |
Oron et al., Temporal focusing microscopy. Cold Spring Harb Protoc. Feb. 2, 2015;2015(2):145-51. |
Papagiakoumou et al., Temporal focusing with spatially modulated excitation. Optics express. Mar. 30, 2009;17(7):5391-401. |
Park et al. Enhanced Axial Resolution of Wide-Field Two-Photon Excitation Microscopy by Line Scanning Using a Digital Micromirror Device. Micromachines. Mar. 9, 2017;8(3):85. |
Raginsky et al., Compressed sensing performance bounds under Poisson noise. IEEE Transactions on Signal Processing. Aug. 2010;58(8):3990-4002. |
Raginsky et al., Performance Bounds for Expander-Based Compressed Sensing in Poisson Noise. IEEE Transactions on Signal Processing. Sep. 2011;59(9):4139-4153. |
Rodriguez et al., Resolution analysis in computational imaging with patterned illumination and bucket detection. Opt Lett. Jul. 1, 2014;39(13):3888-91. |
Rodriguez et al., Resolution analysis in computational imaging with patterned illumination and single-pixel detection. Proceedings of SPIE, The International Society for Optical Engineering. Second International Conference on Applications of Optics and Photonics. Vo. 9286, 7 pages, (2014). |
Rowlands et al., Wide-field three-photon excitation in biological samples. Light: Science & Applications. May 2017;6(5):e16255. |
Sie et al., Fast and improved bioimaging via temporal focusing multiphoton excitation microscopy with binary digital-micromirror-device holography. J Biomed Opt. Nov. 2018;23(11):1-8. |
Willett et al., Poisson Compressed Sensing. Defense Applications of Signal Processing. 6 pages. 2011. |
Zhang et al., Hadamard single-pixel imaging versus Fourier single-pixel imaging. Opt Express. Aug. 7, 2017;25(16):19619-19639. |
Zhang et al., Wavelets Ridgelets and curvelets for poisson noise removal. IEEE Transactions on Image Processing. 2008;17(7):1093-1108. |
Zhu et al., Simultaneous spatial and temporal focusing of femtosecond pulses. Opt Express. Mar. 21, 2005;13(6):2153-9. |
International Search Report and Written Opinion for Application No. PCT/US2019/015272, dated Apr. 25, 2019, 12 pages. |
Choi et al.. Depth resolved hyperspectral imaging spectrometer based on structured light illumination and Fourier transform interferometry. Biomedical optics express. Oct. 1, 2014;5(10):3494-507. |
Number | Date | Country | |
---|---|---|---|
20200342205 A1 | Oct 2020 | US |
Number | Date | Country | |
---|---|---|---|
62622692 | Jan 2018 | US |
Number | Date | Country | |
---|---|---|---|
Parent | PCT/US2019/015272 | Jan 2019 | US |
Child | 16844323 | US |