Patent Application
20180268185
The use of digital devices to collect and store irradiance patterns representative of the scene, optics, etc. has provided a convenient means for storing and processing images. Consequently, image processing algorithms combined with specialized optics are increasingly used to decode the sampled patterns into a form conveying the most relevant information to the user. Examples include, but are not limited to, using algorithms to remove optical aberrations and extract depth information encoded into the collected irradiance pattern.
Embodiments include a method for detecting the number and location of particles. The method may include imaging a plurality of particles using an imaging system having a point spread function, wherein the imaging returns an image; estimating a plurality of point spread function dictionary coefficients of the image using a point spread function dictionary, wherein the point spread function dictionary includes a plurality of point spread function responses that each correspond with a different particle position; and determining the number and/or location of particles in the image from the point spread function dictionary coefficients.
In some embodiments, the method may further comprise determining a two-dimensional location of each particle in the image from the point spread function dictionary coefficients. In some embodiments, the method may further comprise determining a three-dimensional location of the particles in the image from the point spread function dictionary coefficients.
In some embodiments, the point spread function may change as a function of the axial location of the plurality of particles. In some embodiments the number of particles in the image may be determined from a number of non-zero point spread function dictionary coefficients. In some embodiments, the number of particles in the image may be determined from a number of point spread function dictionary coefficients above or below a threshold value.
In some embodiments, the point spread function dictionary may include a plurality of point spread function responses that correspond with a plurality of three-dimensional spatial position of the particle. In some embodiments, the point spread function dictionary may include a plurality of point spread function responses that correspond with a plurality of two-dimensional spatial position of the particle. In some embodiments, the point spread function may include a double helix point spread function. In some embodiments, the point spread function may include an astigmatic point spread function. In some embodiments, the point spread function may produce lobes in the image that are displaced based on the axial position of the plurality of particles. In some embodiments, the plurality of coefficients of the image of the particles are estimated using either or both a Matching Pursuit algorithm and a Convex Optimization algorithm. In some embodiments, the plurality of coefficients of the image of the particles may be estimated using an optimization algorithm.
Embodiments include a computer-implemented method for reconstructing images from compressively sensed data that includes expressing an image as a set of point spread function dictionary coefficients each one corresponding to a particular element of a point spread function dictionary, wherein each particular element of the point spread function dictionary may be an image of an imaging system point spread function associated with a particular spatial position within a particular field of view. In some embodiments, each one of the set of point spread function dictionary coefficients may be calculated based on an optimization algorithm selected from the list consisting of Convex Optimization; Matching Pursuit; orthogonal Matching Pursuit; maximum likelihood estimation; Bayesian information criteria, maximum-likelihood principle; maximum-posterior estimator principle; maximum entropy principle; and minimum discrimination.
Embodiments include an imaging system. The imaging system may include a plurality of optical elements, an imager, a memory, and a processor. The plurality of optical elements may be configured to image a plurality of particles, wherein the optical elements generate a point spread function, and wherein the point spread function changes as a function of the axial location of the plurality of particles. The imager may be configured to image the plurality of particles located onto an image plane. The memory may include a point spread function dictionary that includes a plurality of point spread function responses that each correspond with a different particle position. The processor may be configured to estimate a plurality of point spread function dictionary coefficients of an image of the plurality of particles using the point spread function dictionary and determine the number and/or position of particles in the image using the point spread function dictionary coefficients.
In some embodiments, the processor may be further configured to determine a two-dimensional location of the plurality of particles in the image from the point spread function dictionary coefficients. In some embodiments, the number of particles in the image may be determined from the number of non-zero point spread function dictionary coefficients.
In some embodiments, the number of particles in the image may be determined from a number point spread function dictionary coefficients above or below a threshold value. In some embodiments, the point spread function produces lobes in the image that are displaced based on the axial position of the plurality of particles. In some embodiments, the point spread function dictionary may include a plurality of point spread function responses that correspond with a three-dimensional spatial position of the particle. In some embodiments, the point spread function dictionary may include a plurality of point spread function responses that correspond with a two-dimensional spatial position of the particle. In some embodiments, the plurality of point spread function dictionary coefficients of the image of the particles are estimated using a Matching Pursuit algorithm.
In some embodiments, the plurality of point spread function dictionary coefficients of the image of the particles are estimated based on an optimization algorithm selected from the list consisting of Convex Optimization; Matching Pursuit; orthogonal Matching Pursuit; maximum likelihood estimation; Bayesian information criteria, maximum-likelihood principle; maximum-posterior estimator principle; maximum entropy principle; and minimum discrimination.
Embodiments include a method for detecting the number and/or location of particles. The method may include imaging a plurality of particles using a point spread function that changes as a function of the axial location of a particle, wherein the imaging returns an image; estimating a first plurality of point spread function dictionary coefficients of the image using a Matching Pursuit algorithm with a first point spread function dictionary that includes a plurality of point spread function responses corresponding to a different particle position; determining a first region within which the particles are located in the image from the first plurality of point spread function dictionary coefficients; estimating a second plurality of point spread function dictionary coefficients of the image using a Convex Optimization algorithm with a second point spread function dictionary that includes a plurality of point spread function responses corresponding to different particle position; and determining a location of the particles in the image from the second plurality of coefficients.
In some embodiments, the number of point spread function responses in the second point spread function dictionary may be greater than the number of point spread function responses in the first point spread function dictionary. In some embodiments, the spatial resolution of the point spread function responses in the second point spread function dictionary may be greater than the spatial resolution of the point spread function responses in the first point spread function dictionary.
In an aspect, a method for resolving at least one of a number and location of particles within a scene as described in at least one of the specifications and figures is disclosed.
In an aspect, a method for determining at least one of a number and location of particles located closer than the diffraction limit as described in at least one of the specifications and figures is disclosed.
In an aspect, a computer-implemented method for localizing labels within a scene is disclosed. The method may include expressing an image of the scene as a set of coefficients, each one corresponding to a particular element of a point spread function dictionary, wherein each particular element of the point spread function dictionary is an image of an imaging system point spread function (point spread function) associated with a particular 2-D or 3-D spatial position within a particular field of view, wherein the image of the scene is a superposition of the elements of the point spread function dictionary with the corresponding coefficients.
In an aspect, a computer-implemented method for reconstructing images from compressively sensed data is disclosed. The method may include expressing an image as a set of coefficients, each one corresponding to a particular element of a point spread function dictionary, wherein each particular element of the point spread function dictionary is an image of an imaging system point spread function (point spread function) associated with a particular spatial position within a particular field of view.
A further understanding of the nature and advantages of various embodiments may be realized by reference to the following figures. In the appended figures, similar components or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. When only the first reference label is used in the specification, the description is applicable to any one of the similar components having the same first reference label irrespective of the second reference label.
Embodiments of the present disclosure are directed towards various measurement techniques and/or processing algorithms to determine a number and/or location of particles. As used herein a particle may include a molecule, atom, small object, reflector, cell, scatterer, absorber, emitter, reflector, etc.
In some embodiments, this may be done with precision. In some embodiments, this may be done for particles that may be densely packed. In some embodiments, a super-resolution image may be collected in a fraction of the number of frames of conventional systems.
Embodiments of the present disclosure may be used in a number of applications. For example, embodiments of the present disclosure may be used in fluorescence imaging, astronomical observations, military applications, or scientific measurements. Still other applications are possible as well.
With the advent of digital computers, the purpose of optical imaging systems has changed. A primary goal may no longer be to produce the best image. Rather, a primary goal may be to produce the most useful data, which can be processed using computational methods to directly provide answers to various questions such as: What objects make up the scene under observation? How many of each type of object are there? What are their locations? How much noise is present in the data, and what would the data look like if the noise were not present? It is contemplated that the various measurement techniques and processing algorithms described herein may be capable of providing answers to questions of these sorts.
For example, one field in which the above sorts of questions, and the various measurement techniques and processing algorithms described herein, may be particularly applicable is the field of super-localization microscopy. In this paradigm, the object is assumed to consist of a collection of point sources which are observed using an imaging system. Classical descriptions of optical systems suggest the resolution capability of the system to be limited by the wavelength of light and the numerical aperture. With the best lenses available today, this works out to be approximately half a wavelength. This limit arises from the fact that diffraction does not allow light to be focused to or on an infinitesimally small spot. The pattern that arises from propagation through an optical system is called the system's point spread function, also known as the system's impulse response. If one observes an isolated point source, and the system point spread function is known, the image of the point spread function can be fit to “high” precision, and thus the location of that particle can be known or derived with a precision that is orders of magnitude better than the diffraction limit.
Microscopy techniques such as Photo-Activated Localization Microscopy (PALM) and Stochastic Optical Reconstruction Microscopy (STORM) leverage this idea. In these cases, super-resolution may be obtained by precisely localizing labels in the sample using knowledge of the system's point spread function. To reconstruct a clear image, labeling density should be defined so that details of the object may be identified. However, localization techniques dictate the individual particles to be well-separated from neighboring particles. To address this issue, experimental techniques employed in PALM and STORM enable the experimenter to excite a “small” subset of the particles per frame. However, many localizations from numerous frames may be combined to make or form a dense, super-resolved image. Enhanced spatial resolution may thus be achieved at the expense of temporal resolution. The present disclosure addresses these and other issues. In particular, the various measurement techniques and processing algorithms described herein may enable localization of significantly denser scenes of particles, which allow a super-resolution image to be collected in a fraction of the number of frames.
The reconstruction techniques of the present disclosure are based upon the realization that all the images collected consist of collections of images of the system's point spread function corrupted to a certain degree by noise. It is contemplated that an image may be decomposed using a point spread function dictionary or database that includes shifted point spread functions, and thus we determine the locations and strengths of the sources in the scene, while omitting the noise at this stage. An example of the point spread function dictionary or database of the present disclosure is discussed in further detail below in connection with at least
With respect to the MP technique, the image is decomposed by iteratively selecting the “brightest” sources in the scene, until all sources have been retrieved. In general, the source locations and intensities may be found by projecting the data onto a basis set that consists of all possible particle images. In example embodiments, the “brightest” source will have the largest coefficient when it is projected onto the basis element that corresponds to its image; this location is stored, and that component is subtracted from the image. This continues until all sources have been found, and only background noise remains.
In some embodiments, the 3-D positions of particles may be estimated using a wide-field fluorescence image using the double-helix point spread function shown in the figure. The imaging system 300, for example, can be composed of a sample located at an objective focal plane 310 of a microscope objective 305, a lens 315 with an image plane 318 (e.g., a conventional microscope), another lens 320, and an optical signal processing section. The signal processing section can be essentially a 4f imaging system with a reflective phase-only spatial light modulator 325 placed in the Fourier plane. In some embodiments, an tube lens 315 (e.g., a achromatic lens) can be placed at a distance f from the microscope's image plane 318 producing the Fourier transform of the image at a distance f behind the lens. The phase of the Fourier transform can be modulated by reflection from the liquid crystal spatial light modulator 325. The spatial light modulator can include a double-helix point spread function. Because the spatial light modulator 325 can be sensitive only to vertically polarized light, a vertical polarizer 330 can be placed immediately after the spatial light modulator 325 to block any horizontally polarized light not modulated by the spatial light modulator 325. The final image can be produced by another achromatic lens 335 (e.g., f=15 cm) placed at a distance f after the spatial light modulator 325 and recorded with a camera 340 (e.g., an EMCCD camera).
Additionally or alternatively, the spatial light modulator 325 may include or be replaced by a phase mask, an amplitude mask, or a phase/amplitude mask located in a Fourier plane or other plane. Moreover, in some embodiments, the imaging system 300 may be reconfigured with a refractive spatial light modulator. In some embodiments, the spatial light modulator 325 may produce a spatially varying phase delay, have a continuously varying phase, and/or may include a piecewise constant phase, a piecewise linear phase function, a continuous piecewise linear phase function or a convex piecewise linear phase function, and/or a discontinuous piecewise linear phase function.
In some embodiments, the imaging system 300 may include a refractive, reflective, or diffractive optical element composed of at least one linear phase domain (phase ramp) such that each linear phase domain is organized to generate a linear phase delay to light passing through one part of the objective. Each linear phase domain, for example, may have equal or different linear variations, including equal or different slope orientations. When the spatial light modulator 325 is loaded with a double-helix point spread function phase-mask, the Fourier transform of the sample image can be multiplied by the double-helix point spread function transfer function. Equivalently, every object point may be convolved with two lobes. In some embodiments, the angular orientation of the two lobes may vary depending on the axial location of the object above or below focus. For example, the two lobes may be aligned horizontally when the particle is in focus. As the particle is moved towards the objective, the double-helix point spread function lobes may rotate in the counterclockwise direction. On the other hand, if the particle is moved away from the objective, the lobes may rotate in the clockwise direction.
Alternatively or additionally, a lateral point spread function may be used causing the lobes to be displaced horizontally depending on the axial location of the particle. For example, the two lobes may be aligned vertically when the particle is in focus. As the particle is moved towards the objective, a first lobe may move to the left and a second lobe may move to the right. On the other hand, if the particle is moved away from the objective, the first lobe may move to the right and the second lobe may move to the left. In some embodiments, the point spread function may produce a transverse profile composed of multiple patterns. Each of these patterns may retain its fundamental shape with defocus yet each pattern may move in different trajectories during defocus.
In some embodiments, the point spread function may produce two image lobes that are separated from each other along a straight line. The two lobes may be separated from each other (e.g., the line may extend or contract) based on the defocus of the particles. For example, the two lobes may be separated from each other in the opposite direction. In some embodiments, the point spread function may produce two lobes that move along two different straight lines as the particle moves relative to the objective lens from a positive defocus position to a negative defocus position and vice versa. In some embodiments, the two straight lines may be parallel to each other.
In some embodiments, the point spread function may have at least one helix with infinite offset radius that degenerates into a straight line. In some embodiments, the point spread function may have two straight lines with an axis of symmetry coinciding with the optical axis of the system. In some embodiments, the point spread function may have at least a helix with null (zero) pitch such that the helix degenerates into a circle or an ellipse. In some embodiments, the point spread function may generate a conical surface in three-dimensional space. In some embodiments, the point spread function may have at least one helix degenerating into a conical section curve.
In some embodiments, a maximum of a point spread function may describe a curve in three-dimensional space that turns around an axis at a constant or continuously varying distance (offset) while moving in the direction parallel to the axis. In some embodiments, a maximum of the point spread function describes a curve in three-dimensional space similar to a helix with varying offset from a helix axis and/or a pitch, and when both the axis and offset are constant, the point spread function describes a helical curve.
In some embodiments, the imaging system 300 may encode spectral and/or polarization information about each particle and/or the image.
In some embodiments, the imaging system 300 may include a converging lens such as, for example, a kinoform.
In some embodiments, the imaging system 300 may be configured to obtain at least two images with different focus settings. In some embodiments, the imaging system 300 may be configured to work either in a wide field mode or a confocal mode.
When a sample comprises multiple sparse particles at different 3-D positions, the detected double-helix point spread function image may exhibit two lobes (with different angular orientations) for each particle. The transverse (x-y) position of a particle may be estimated from the midpoint of the line connecting the positions of the two lobes, and the axial position can be estimated from the angle of the line connecting the two lobes using a calibration plot that maps angles to axial positions.
Some embodiments described herein may include a mechanism that produces a spatially varying phase difference imposed among different components of the light coming through different parts of the objective 305 to generate a three-dimensional point spread function image. Such a mechanism may be placed within the optical path of the objective 305 and/or the camera 340 and the image plane 318.
An example calibration plot of angle versus z position is shown in
At block 510 a model of the imaging system may be created. In some embodiments, the model of the imaging system may be created prior to imaging. In some embodiments, the model of the imaging system may occur during calibration, during set up, during development, and/or during manufacturing. In some embodiments, the model may be created from experimental measurements, theoretical calculations, and/or a combination of both. In some embodiments, the model may vary depending on the type and/or the characteristics of the point spread function of the imaging system.
In some embodiments, a model of the system may be a mathematical description of the response of the system that relates output signals to input signals. The model may be the result of a theoretical calculation of the response based on the individual components and obtained through geometrical optics, wave optics, and/or rigorous electromagnetic calculations. The model may be obtained experimentally and/or may be composed of the measurements performed of the system response, possibly with interpolation between measurements and possibly with processing to reduce the influence of noise. In some embodiments the model may include a model of the noise which can also be based on measurements or theoretical.
At block 515 a point spread function dictionary may be created having a number of different entries where each entry corresponds with the point spread function for a particle at different locations. The point spread function dictionary can be created from the model created in block 510. Each point spread function dictionary element may include a 2-D cross section of the point spread function corresponding to a different discrete particle location in a full 3-D space. Each point spread function dictionary element may correlate a point spread function response Examples of a point spread function dictionary are illustrated in
The step size between adjacent elements within a point spread function dictionary, in some embodiments, may be an important consideration. In some embodiments where localization precision is important, small steps may be used. In some embodiments where computational efficiency is important, large steps may be used. In some embodiments, steps having sub-pixel resolution may be used. In some embodiments, the point spread function dictionary may extend further in one direction than another. In some embodiments, the dictionaries may be over complete, meaning there are more elements in the point spread function dictionary then there are in pixels being imaged (D>N). In some embodiments, block 505 and block 510 may be performed together. In some embodiments, block 510 and block 515 may occur during calibration, during setup, during development, and/or during manufacturing.
At block 520, coefficients of the point spread function dictionary elements that best represent the data may be estimated. Any number of techniques may be used to estimate these coefficients such as matching techniques, optimization techniques, fitting techniques, etc. These techniques may include, for example, a Matching Pursuit technique and/or a Convex Optimization technique. Other techniques may use the maximum-posterior estimator, maximum entropy, minimum discrimination, least squares, linear programming, convex quadratic minimization with or without linear constraints, quadratically constrained Convex-quadratic minimization with or without convex quadratic constraints, conic optimization, geometric programming, second order cone programming, semidefinite programming, Bayesian estimation, generalized Bayesian estimation, empirical Bayesian estimation, or recursive Bayesian estimation, and/or entropy maximization. In some embodiments, a combination of different techniques may be used. Moreover, in some embodiments, the estimation may be fully parametric, semi-parametric, and/or non-parametric. In some embodiments, the estimation may include information theoretic statistical inference, such as minimum description length.
At block 525, the number of particles may be determined from the point spread function dictionary coefficients. The estimates found in block 520 may be used to determine the number of particles that are present in the image. For instance, the estimates provided in block 520 may determine point spread function dictionary elements that are local or global maxima or minima. The number of these point spread function dictionary elements may correspond with the number of particles (or elements) in the image.
At block 525 the locations of the particles may be determined from the point spread function dictionary coefficients. Each point spread function dictionary element corresponds with a particle location in three-dimensional space. Thus, once the point spread function dictionary elements having local or global maxima or minima are identified, the location of the particles can be determined.
At block 620, rough coefficients of the point spread function dictionary elements that best represent the data can be estimated using a first technique such as, for example, the Matching Pursuit technique. The Matching Pursuit technique, for example, can use a coarse point spread function dictionary. A coarse point spread function dictionary, for example, may include fewer point spread function dictionary elements than the number of particles in the image.
At block 625, the rough number of particles and/or the rough location of the particles can be estimated from the rough coefficients determined above.
At block 630, from the rough number of particles and/or the rough location of the particles, the Convex Optimization technique may be used in the area around the rough locations of the particles. The Convex Optimization technique may use, for example, an over-complete point spread function dictionary that includes more point spread function dictionary elements than pixels in the image.
Processes 500 and/or 600 may be repeated any number of times using imaging subsystems having different point spread functions and different point spread function dictionaries. A number of different images may be returned. In some embodiments, these images may be statistically or mathematically combined to create a single image prior to determining the number and/or location of particles. In some embodiments, the number and/or location of particles in each image may be determined for each image and the average (or another function) location and/or number of particles may be determined. A first example of an iterative process for finding the locations of a number of sources is shown in
The original image is shown in
In this example, the simulation parameters were defined to be comparable to “typical” experimental conditions: emission wavelength 660 nm; imaging system NA 1.45; magnification 100×. These simulation parameters are similar to those used for the first example shown in
In some embodiments, the computational effort of performing the localization may be far less than in other methods, yielding faster processing times. The algorithm may be efficient and robust such that it may be implemented in hardware, allowing for the possibility of real-time super-resolution reconstructions at video rates. A further advantage of the systems, devices, and methods may be that background noise in the data is not reconstructed using the point spread function dictionary; the reconstruction produced is “noiseless” (see e.g.,
With respect to the CO technique, rather than decomposing the data by iteratively choosing the set of coefficients to describe the scene, CO may arrive at an “optimal” and/or “approximate” representation of the image by solving for all the coefficients in parallel. This is possible by enforcing the assumption that the image is sparse in “point spread function space.” This means that although the image may be defined as a large set of pixel values, it can be efficiently expressed as a much smaller number of coefficients that correspond to point spread functions. In the example of
The CO technique is conceptually similar to the MP technique. Although CO may have the potential to find a more optimal, and thus more correct, decomposition of the true image, this, however, may come with a computational cost. For this reason, it may be inadvisable to use CO on “standard” data sets. However, CO may be optimally used in performing the localization of data collected in a compressive manner. By combining the collection efficiency of compressive sensing with the methods, of the present disclosure, ability to handle high-density scenes, total observation times for a particular set of data may be reduced substantially or drastically.
As mentioned above, the sparsity assumption may allow embodiments described herein such as, for example, the CO technique, to discover the source locations in an image. In particular, the sparsity assumption when used in combination with compressive sensing (CS) may yield localization and super-resolution results. Simulations indicate this point spread function dictionary may be employed to achieve arbitrary localization precision. When used in combination with a microscope equipped with a point spread function (e.g., as shown in
In some embodiments described herein, a signal may be recovered using far fewer measurements than Shannon's sampling theory predicts. This may allow for stable reconstruction of large signals using “small” numbers of measurements.
Embodiments described herein may include a transform from “image space” to “measurement space.” This may be written mathematically as
M=CS×I equation 1
sizes: [K×1]=[K×N]×[N×1].
I is the image expressed as a column vector, CS is the measurement matrix, and M is the vector containing the set of K measurement values.
For an image with N pixels resulting from an s-sparse signal (a signal that can be represented by s non-zero coefficients in some basis), the number of required samples for stable recovery is:
K=s log N/s. equation 2
The matrix CS may spread some information about the coefficients of the underlying signal into many measurements. For this reason, the measurement matrix may be populated with random values from a Gaussian distribution or a set of random 1s and 0s. To reconstruct the image from the measurements, another transform may be introduced. This is the transform from “sparse space,” also referred to as “feature space,” to image space. Here, M may be re-written as:
M=CS×T×x, equation 3
[K×1]=[K×N]×[N×N]×[N×1].
T is a matrix representing the image transform, and x is the s-sparse vector containing the coefficients that describe the image I. Further, T is a square matrix, and may represent any signal decomposition, such as a Fourier basis or a wavelet basis. Given this assumption about the vector x, the reconstruction problem may be formulated as convex optimization. An algorithm is written to maximize the sparsity of x while enforcing that the measurements are accurate; in mathematical terms:
{tilde over (x)} is the variable that is being solved for, and ε is the l2 error threshold at which it will be decided that a solution has been found (in the absence of noise, ε=0).
An image transform may be selected to perform feature-extraction on an image such as, for example, an image made from fluorescence microscopy data, in accordance with some embodiments described herein. In general, in some embodiments the goal of fluorescence microscopy may be to accurately and/or precisely determine the location of particles within the volume of a sample. By selecting an image transform that goes from feature space to image space, an algorithm (e.g., a CS algorithm) may be used to solve for the desired measurements. When using an imaging system with the capability of producing a #-D point spread function (e.g., a 3-D point spread function) such as the double-helix point spread function or the lateral point spread function, this idea may be extended to 3-D super-resolution.
In some embodiments, the basis set for the CS recovery algorithm may be selected to take advantage of all possible assumptions. And, in some embodiments the basis may not form an orthonormal set. There is freedom in selecting a basis set, because any restrictions imposed by the recovery scheme may apply to the product of the random measurement matrix and the basis set. The basis set, for example, may be chosen to obey the Restricted Isometry Property (RIP), shown below as:
(1−δs)∥x∥22≤∥Ax∥22≤(1−δs)∥x∥22 equation 5
The main implication of this property is that each subset of s rows from the full set of K total rows approximate an orthonormal set.
In fluorescence microscopy experiments, in which the goal is to determine the location of particles in a sample, the images consist of a sparse set of images of the systems point spread function. Thus, it is contemplated that a choice of basis elements would be images of the system point spread function shifted to different locations in the field of view. Smaller shifts in the point spread function may result in more precise localization results. However, this increased precision may come with a cost of increased computational complexity and a less strict adherence to the RIP. An example of the basis elements of images of the system point spread function is shown in
As mentioned above in connection with
In general,
In some embodiments a point spread function dictionary may include a 2D point spread function corresponding to lateral and/or axial shifted locations of the point source. If, for example, the system is shift-invariant for a given depth, all 2D point spread functions may be equal except for a transverse shift. The dictionary entries may be generated by first selecting a 3D window size and making a list of discrete 3D locations to fill this volume. A model is then used to generate the corresponding image for each point source location in the dictionary. The model may be either an ideal point spread function, or it may be the result of calibration data from the specific imaging system. The step size of the locations in the dictionary may be used to determine the localization precision. Thus, in some embodiments, the step size may be very fine. In some embodiments, the step size may be related to the limit of localization precision achievable in a given system, as calculated by the Cramer-Rao Lower Bound (CRLB). In some embodiments, the effective pixel size (physical pixel size divided by magnification) in single-particle detection systems may be designed to sample the point spread function at approximately Nyquist sampling, which may result in effective pixel sizes on the order of, for example, 160 nm. The CRLB, on the other hand, can be significantly lower (typically 10 to 20 nm). As a result, the dictionary may result in sub-pixel shifts. Additionally, since the extent of the point spread function is significantly larger than the step size, there may be a high degree of similarity between adjacent dictionary elements. Hence, by definition, the dictionary may be coherent.
Furthermore, the requirement for 3D information may result in many dictionary elements at the same transverse location. In some embodiments, the point spread function (e.g., the double helix point spread function) may enable an extended depth of field (DOF) as compared to a standard aperture by approximately a factor of two. In some embodiments a 1.45 NA objective may be used with a DOF of approximately 2 m. However, since the CRLB is larger in the axial dimension than in the transverse dimensions, the axial step size can be also slightly larger than the transverse step size.
Because the dictionary contains discrete locations, the results may have a quantization error. Even if the localization algorithm returns the correct dictionary element for every particle in a scene, there will still be a distribution of errors spanning a range between d/2 to +d/2, where d is the step size. The standard deviation of a uniform distribution with a range of d is d/√{square root over (12)}. The standard deviation may be chosen to characterize the error because it can be directly compared to the CRLB, which can be stated as a standard deviation. In some embodiments, the standard deviation may be used to quantify the magnitude of the error in simulations.
In some embodiments, the goal is not to generate a reconstruction of each image. Although a reconstruction image may be obtained, the desired information may be actually in the indices of the large coefficients in the sparse reconstruction. Each dictionary element may have a physical meaning: the presence of a large coefficient in the solution means there is a particle at the corresponding location. To keep track of this, each dictionary may carry with it a LUT to convert the index of a dictionary element to a physical location. The LUT is a D-by-3 matrix, where D is the number of dictionary elements. There are three columns because the solution space is 3D. Each row contains the (x,y,z) values describing the location of a particle 8 that would generate the point spread function in the corresponding index of the dictionary. The LUT may be easily generated alongside the dictionary, and may provide a simple way to obtain a physical interpretation of the coefficients in the solution.
Each element of the dictionary may be normalized such that its L2 norm is unity. When this is the case, the product of the coefficient and the dictionary element may match the intensity of that component in the image. The corresponding number of photons may be calculated by multiplying the coefficient with the L1 norm of that particular basis element. This is only true, of course, if the data has previously been converted from camera counts to photons. A matrix containing the L1 norms for each dictionary element may be calculated beforehand and stored with the dictionary for fast conversion from coefficient values to photon counts. It may be necessary to calculate photon counts frequently, since one of the stopping criteria for the iterative method requires knowledge of the number of photons.
In some embodiments, the coefficients for representing the image could be found by the inner product of the image and each individual dictionary element. The use of small sub-pixel shifts means adjacent dictionary elements have a high degree of similarity; they cannot be orthonormal. Therefore, the coefficients may be found using Matching Pursuit (MP) and Convex Optimization (CO).
MP is similar to finding coefficients for an orthonormal dictionary. In one iteration of MP the image may be projected onto the dictionary, just as if the dictionary were orthonormal. However, instead of keeping all the resulting coefficients, only the largest coefficient is stored as part of the solution. Next, that element is subtracted from the image. Iterations may continue until a stopping criterion is met.
Implementation of the Matching Pursuit (MP) algorithm may use any type of stopping criteria. One stopping criterion, for example, may limit the number of iterations (i.e. the maximum number of particles the algorithm will return for a given scene). Simulations indicate the reliability of the algorithms decreases as the particle density increases. Therefore results that are likely to be erroneous may not be recorded. Not only is a limit placed on the maximum iterations, but the order in which results are subtracted from the scene is also recorded. With this information, the performance of the iterative algorithm may be observed as the density increases in an experimental setting.
Another stopping criterion, for example, may be the minimum estimated number of photons from a particle. Even when all the particles in a scene have been accounted for and subtracted, the remainder will still have positive values due to noise (particularly background noise). This could lead to spurious particles in the reconstruction. To avoid this, MP ends iterations if the photon count of the strongest remaining component is below a threshold. This threshold can be determined beforehand based on the particles used in the experiment, or the threshold can be decided afterwards based on the data set. In this second case, a preliminary threshold must be set to a very low value. Once MP is complete, the threshold can be raised and the weakest particles are excluded from the results.
Another stopping criterion, for example, may involve placing a limit on the value of pixels in the remainder image. As above, when all actual particles in a scene are accounted for and subtracted, there will still be a remainder due to noise. Once the maximum pixel value in the remainder image drops below the expected value of background in the data set, the subsequent localizations are likely a result of noise only. Therefore, the iterations are stopped when the maximum of the remainder image drops below a threshold. Typical background levels can be predicted accurately from examining statistics of the whole data set. Furthermore, this criterion is usually set slightly below the background level (by perhaps 75%) to avoid false negatives.
Many other embodiments of MP (or other iterative algorithms) may employ a stopping criterion that compares the reconstructed image to the raw data. Iterations stop when the error between the two has reached an acceptable level. In some embodiments, a limit on the error between the reconstruction and the original data would be meaningless in the case of a scene that contains no particles.
Estimation using CO, on the other hand, may attempt to arrive at a solution for the significant coefficients in parallel. Although the estimation method is quite different, the use of the algorithm is similar with the inputs are a window of data and a dictionary, and the output is a list of coefficients that describe the data. The reconstruction problem may be formulated as a convex problem in which the variable to be optimized is the sparsity of the coefficient vector (quantified as the L1 norm). This can be phrased mathematically as:
minimize∥x∥1subject to∥Ax−b∥2≤ε,
where x is the set of coefficients, A is the dictionary, and ε is an error bound related to the total intensity in the image. In this formalism, the image b has been folded into a column vector.
In some embodiments, the set or a set of coefficients of the “point spread function dictionary” of the present disclosure may be found based on a number of various criteria that may or may not include other prior information about the scene or the imaging system. For instance, one class of optimality criteria is the class of information theoretic criteria, which are based on a probabilistic formulation of the problem. One particular example is the maximum-likelihood principle for estimation of a set of parameters (the coefficients in this case). Other examples are the maximum-posterior estimator principle, maximum entropy, or minimum discrimination principles. Other estimators may also be used.
For example, as shown in
The computer device 1900 is shown comprising hardware elements that may be electrically coupled via a bus 1902 (or may otherwise be in communication, as appropriate). The hardware elements may include a processing unit with one or more processors 1904, including, without limitation, one or more general-purpose processors and/or one or more special-purpose processors (such as digital signal processing chips, graphics acceleration processors, and/or the like); one or more input devices 1906, which can include, without limitation, a remote control, a mouse, a keyboard, and/or the like; and one or more output devices 1908, which can include, without limitation, a presentation device (e.g., television), a printer, and/or the like.
The computer device 1900 may further include (and/or be in communication with) one or more non-transitory storage devices 1910, which can comprise, without limitation, local and/or network accessible storage, and/or can include, without limitation, a disk drive, a drive array, an optical storage device, a solid-state storage device, such as a random access memory, and/or a read-only memory, which can be programmable, flash-updateable, and/or the like. Such storage devices may be configured to implement any appropriate data stores, including, without limitation, various file systems, database structures, and/or the like.
The computer device 1900 might also include a communications subsystem 1912, which can include, without limitation, a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and/or a chipset (such as a Bluetooth™) device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities (e.g., GSM, WCDMA, LTE, etc.), and/or the like. The communications subsystem 1912 may permit data to be exchanged with a network (such as the network described below, to name one example), other computer systems, and/or any other devices described herein. In many embodiments, the computer device 1900 will further comprise a working memory 1914, which may include a random access memory and/or a read-only memory device, as described above.
The computer device 1900 also can comprise software elements, shown as being currently located within the working memory 1914, including an operating system 1916, device drivers, executable libraries, and/or other code, such as one or more application programs 1918, which may comprise computer programs provided by various embodiments, and/or may be designed to implement methods, and/or configure systems, provided by other embodiments, as described herein. By way of example, one or more procedures described with respect to the method(s) discussed above, and/or system components might be implemented as code and/or instructions executable by a computer (and/or a processor within a computer); in an aspect, then, such code and/or instructions can be used to configure and/or adapt a general-purpose computer (or other device) to perform one or more operations in accordance with the described methods.
A set of these instructions and/or code might be stored on a non-transitory computer-readable storage medium, such as the storage device(s) 1910 described above. In some cases, the storage medium might be incorporated within a computer system, such as the computer device 1900. In other embodiments, the storage medium might be separate from a computer system (e.g., a removable medium, such as flash memory), and/or provided in an installation package, such that the storage medium can be used to program, configure, and/or adapt a general-purpose computer with the instructions/code stored thereon. These instructions might take the form of executable code, which is executable by the computer device 1900 and/or might take the form of source and/or installable code, which, upon compilation and/or installation on the computer device 1900 (e.g., using any of a variety of generally available compilers, installation programs, compression/decompression utilities, etc.), then takes the form of executable code.
It will be apparent to those skilled in the art that substantial variations may be made in accordance with specific requirements. For example, customized hardware might also be used, and/or particular elements might be implemented in hardware, software (including portable software, such as applets, etc.), or both. Further, connection to other computing devices such as network input/output devices may be employed.
As mentioned above, in one aspect, some embodiments may employ a computer system (such as the computer device 1900) to perform methods in accordance with various embodiments described herein. According to a set of embodiments, some or all of the procedures of such methods are performed by the computer device 1900 in response to the processor(s) 1904 executing one or more sequences of one or more instructions (which might be incorporated into the operating system 1916 and/or other code, such as the application program 1918) contained in the working memory 1914. Such instructions may be read into the working memory 1914 from another computer-readable medium, such as one or more of the storage device(s) 1910. Merely by way of example, execution of the sequences of instructions contained in the working memory 1914 may cause the processor(s) 1904 to perform one or more procedures of the methods described herein. The terms “machine-readable medium” and “computer-readable medium,” as used herein, may refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented using the computer device 1900, various computer-readable media might be involved in providing instructions/code to the processor(s) 1904 for execution and/or might be used to store and/or carry such instructions/code. In many implementations, a computer-readable medium is a physical and/or tangible storage medium. Such a medium may take the form of a non-volatile media or volatile media. Non-volatile media may include, for example, optical and/or magnetic disks, such as the storage device(s) 1910. Volatile media may include, without limitation, dynamic memory, such as the working memory 1914. Example forms of physical and/or tangible computer-readable media may include a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, a RAM, a PROM, EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and/or code.
Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor(s) 1904 for execution. By way of example, the instructions may initially be carried on a magnetic disk and/or optical disc of a remote computer. A remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and/or executed by the computer device 1900.
The communications subsystem 1912 (and/or components thereof) generally will receive signals, and the bus 1902 then might carry the signals (and/or the data, instructions, etc. carried by the signals) to the working memory 1914, from which the processor(s) 1904 retrieves and executes the instructions. The instructions received by the working memory 1914 may optionally be stored on the storage device(s) 1910 either before or after execution by the processor(s) 1904.
In one example, a first optical subsystem 2010-1 may include a double-helix point spread function and may produce image frame 2015-1. A second optical subsystem 2010-N may include a lateral point spread function and may produce image frame 2015-N. The number and/or location of the particles 2005 may be estimated from each image frame. The results of these two image frames may be combined (e.g., averaged, etc.) to estimate the number and/or location of the particles 2000. Any number of optical subsystems 2010 may be used with any number or combination of point spread functions.
The methods, systems, and devices discussed above are examples. Various configurations may omit, substitute, or add various method steps or procedures, or system components as appropriate. For instance, in alternative configurations, the methods may be performed in an order different from that described, and/or various stages may be added, omitted, and/or combined. Also, features described with respect to certain configurations may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Also, technology evolves and, thus, many of the elements are examples and do not limit the scope of the disclosure or claims.
Specific details are given in the description to provide a thorough understanding of example configurations (including implementations). However, configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the configurations. This description provides example configurations only, and does not limit the scope, applicability, or configurations of the claims. Rather, the preceding description of the configurations will provide those skilled in the art with an enabling description for implementing described techniques. Various changes may be made in the function and arrangement of elements without departing from the spirit or scope of the disclosure.
Also, configurations may be described as a process which is depicted as a flow diagram or block diagram. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure. Furthermore, examples of the methods may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a non-transitory computer-readable medium such as a storage medium. Processors may perform the described tasks.
Furthermore, the example embodiments described herein may be implemented as logical operations in a computing device in a networked computing system environment. The logical operations may be implemented as: (i) a sequence of computer implemented instructions, steps, or program modules running on a computing device; and (ii) interconnected logic or hardware modules running within a computing device.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
| Number | Date | Country | |
|---|---|---|---|
| 61791392 | Mar 2013 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14777361 | Sep 2015 | US |
| Child | 15856539 | US |
