The present invention relates to improving the spatial resolution of images captured using multi-beam scanning confocal imaging systems by developing the mechanisms required in a variety of multi-beam confocal scanner formats that enable the data capture requirements of the prior art calculations to be met.
Attempts at improving spatial image resolution beyond the Abbe resolution limit in wide field and confocal imaging systems have used structured illumination methods combined with manipulation of the Fourier representation of those images (Mueller and Enderlein, Physical Review Letters 104, 198101 (2010)). These attempts have been targeted at traditional single beam scanning confocal systems and have not been applied to multi-beam confocal scanning systems because the latter systems do not inherently contain the necessary elements or the precision of step by step scanning or synchronisation between elements to obtain the data required for the calculations.
In the prior art a single point scanning confocal system is made to scan a focused laser illumination beam step by step over a sample; and an image of a region of interest (ROI) Spq is captured at each stationary beam scan position producing an array of pixel intensities I(Spq) for each scan position Rj,k. The distance between spatially adjacent scan positions on each coordinate axis is substantially less than the native optical resolution of the confocal system and typically a sub-multiple of the pixel dimensions of the detector. The data from the multiple ROIs is processed to produce an effective image Ieff(Rjk)=summation of intensities for all p,q of I(Rjk−(Spq/2), Spq); the effective image Ieff(Rjk) is further processed by Fourier transformation, filtered with a weighting function equivalent to an enhanced point spread function (PSF), then reverse Fourier transformed into a spatial equivalent to produce a final image which exhibits enhanced spatial resolution. Full details of this data processing are given in the reference (Mueller and Enderlein).
There is a class of confocal imaging systems which use multi-beam scanning techniques to increase the speed of image capture over the traditional single beam scanners. Such systems use multiple parallel beams that scan the sample simultaneously. Multi-beam scanners come in a variety of forms, for example, those based on Nipkow spinning discs (for example U.S. Pat. No. 5,428,475), 1-D array (sometimes called swept field) (for example U.S. Pat. No. 6,856,457) or 2-D array (for example PCT WO 03/075070) scanners.
Nipkow disc based systems may use a single disc, spinning on its central axis, and contain an array of apertures placed in a conjugate image plane such that the illumination source is divided into multiple scanning beams and in which the return fluorescent or reflected light that passes through the same, or an identical set of apertures (as in tandem scanning confocal systems, for example U.S. Pat. No. 4,802,748), forms a confocal image. An enhancement of the basic Nipkow disc arrangement patented by Yokogawa Electric Corporation, Japan, adds a second disc to the first such that the second disc contains an array of microlenses, positioned such that each aperture has an associated microlens. This second disc is inserted between the first disc and the light source, such that each of the microlenses collects the illumination light from a larger area and focuses it into its associated aperture, thus improving the efficiency of the illumination path. A dichromatic mirror, or beam splitter, placed between the two discs separates the emission, fluorescent or reflected, light from the illumination light and reflects it out from between the two discs.
Swept field scanners use a 1-dimensional array of illumination apertures, with or without an associated microlens array, which is scanned rapidly back and forth across the sample while a piezo transducer oscillates the scanning lines of the one dimensional array along the axis of the array with an amplitude equal to the separation of the apertures in the 1-D array. The period of the piezo oscillation is typically half that of the framing rate of the image capture device, such that each half period scans the whole sample field of view.
2-D array scanners use a single oscillating galvanometer mirror to scan a 2-D array of illumination apertures, with or without an associated microlens array, back and forth across the sample; each sweep completing a full scan of the sample.
The fluorescent light returned in both the swept field and 2-D array scanners is de-scanned by the scanning mirror, separated from the illumination light by a suitable dichroic mirror and focused on to an array of imaging apertures matching the pattern positions of the array of illumination apertures in a conjugate focal plane. These systems also make use of microlens arrays positioned in the illumination path such that the focus plane of the points of light so generated are focused as diffraction limited spots in the conjugate sample plane with or without passing through apertures in the illumination path.
Multi-beam scanning systems offer several useful advantages over single spot scanners, for example, faster scanning of the sample area enabling higher image capture rates to be achieved; and reduced photo-bleaching of the sample material with consequent reduction in phototoxic effects especially when applied to the imaging of living cells in life science research. Consequently improved spatial image resolution in these systems is also highly desirable. This invention shows how multi-dimensional confocal scanners can be adapted to meet the stringent scanning requirements and collect and process the data into the form required by the prior art for two-dimensional image resolution enhancement.
This invention provides for improving the spatial resolution of images captured using multi-beam confocal scanning systems by developing the mechanisms required in a variety of multi-beam confocal scanner formats that enable the data capture requirements of the prior art calculations to be met.
Each of the beams in a multi-beam scanning system can be considered as an individual single beam scanner, provided that there is sufficient spatial separation between any pair of adjacent beams and the beams are focused into diffraction limited spots in the sample focal plane. These conditions have to be met for successful normal operation of such confocal systems and therefore can be assumed to exist in all practical multi-beam confocal scanning systems.
Each beam, or focused scanning spot of illumination, forms a special case of a structured illumination in which all Fourier components supported by the imaging optics are present (Mueller and Enderlein). In normal operation, the scanning of these multiple beams or spots is continuous and the fluorescent, or reflected, light returned from the illuminated spots in the sample focus plane is focused on to a set of confocal apertures arranged such that every illumination spot has its own confocal aperture in a conjugate focal plane and the light which passes through the confocal apertures forms a confocal image in yet another conjugate focal plane. Typically, a 2-dimensional detector, for example, a CCD camera, is placed in this focal plane (or in an optically relayed duplicate of it) to record the confocal image. Such confocal images are substantially limited in resolution by the Abbe resolution limit.
If the array of illumination spots is made to sequentially step through a scan pattern rather than scan continuously, and an image is captured at each step position, each of the images so captured can be divided into an array of sub images or ROIs, a sub array or ROI being centered on the current image position of each of the illumination spots.
The present invention defines the means by which this image data may be collected for processing by the methods defined in the prior art by Mueller and Enderlein in that the intensity data contained in each ROI from a series of images that together form a complete scan of the sample is processed to create an intermediate image having an improved resolution. The intermediate image is Fourier transformed, filtered with a weighting function based on an improved point spread function (PSF) and reverse Fourier transformed to create the final image containing further resolution enhancement.
The requirement of the prior art is to scan the sample point by point with increments smaller than the optical resolution limit of the optical system in two orthogonal directions in the sample focal plane of the confocal scanning microscope system. This requirement is not met by current multibeam confocal scanning microscope systems. The present invention describes methods by which the various forms of multibeam confocal scanning systems may be modified to achieve this requirement.
1-D or swept field confocal scanners require additional means to produce the required positional accuracy of the scanning mechanisms on both axes and to obtain the level of synchronisation required between the scanning mechanisms and the image capture by the 2-D detector.
Nipkow disc and 2-D array confocal scanners require additional means to move the scan orthogonal to their basic scanning direction in order to generate the additional data required for the calculations required to create the improved resolution images. Such additional means can take various forms and has to deflect the scan in sub optical resolution dimensions with a total range of movement of the same order of magnitude as the native optical resolution of the target system.
The inclusion of devices to move both the microlens and pinhole arrays by equivalent amounts provides one method which suits both the 1-D and 2-D array scanners.
Moving the scan of a Nipkow disc scanner in a radial direction across the sample field of view can be achieved by a variety of methods, for example moving the center of rotation of the rotating disc(s) using displacement devices such as a piezo displacement device or a linear motor. An alternative solution is to deflect the scanning beams and this can be achieved by inserting an optical window into the optical path between the confocal scanner and the microscope and tilting the window such that the scanning beams are deflected by refraction at the air/window and window/air interfaces by an amount determined by the angle of tilt and the thickness of the window. Yet another solution is the insertion and removal of a tilted, or tilting, window in the optical path to provide the necessary displacements orthogonal to the scan direction. Due to the reciprocal nature of the optical path, the fluorescent, or reflected, light returned from the sample on the microscope is deflected by an identical amount and hence no change in position of the microlens or pinhole arrays relative to the optical path is required providing that compensation means are included for chromatic aberrations.
1-D and 2-D array scanners may also use the technique of including a tilting optical window in the optical path between the confocal scanner and the microscope to shift the scanning beams.
Another solution requires that the array of points in the spinning disc and in the 2-D array scanner are arranged such that subsets of scanned points within the scan pattern are displaced by sub native optical resolution distances orthogonal to the scanning direction, in this case no additional displacement mechanism is required, it remains to identify the scan positions of the relative subsets of points so that the appropriate ROI images can be used in the subsequent calculations. This solution requires no additional optics in the beam path and is computationally efficient when the sub-optical resolution scanning increment is arranged to match a sub-multiple of the detector pixel size.
In the Nipkow disc type of confocal scanner, the rotational position of the discs must be known to a sufficiently precise degree such that the relative image points required by the prior art calculations can be identified with sufficient accuracy. The angular position can be identified by techniques such as adding an encoding mechanism to the rotation of the discs. This can be a separate encoding module attached to the shaft or its motor drive, or an encoding pattern (optical, mechanical, magnetic, etc.,) placed directly on one or both discs, and combined with a suitable encoder detection device.
In the 2-D array scanner additional positional accuracy can be obtained by implementing such methods as closed loop control techniques to the drive of the scanning mirror. It is also necessary to ensure that the drive signal to the scanning mirror has the necessary resolution and accuracy to enable sufficiently small incremental scanning steps.
Using the above techniques the required image data can be collected by methods such as stepping the scan, position by position, over the sample area and capturing an image at each step; alternatively, images may be collected while scanning ‘on the fly’ by strobing the illumination and/or gating the detector at each required scan position.
The ability to collect the intensity data in parallel from multiple ROIs when using multi beam scanners significantly improves the speed of image (i.e. data) capture. The ability of modern computer systems with parallel operation of multiple and multi-core CPUs and multi-threaded software provides suitable environments for efficient use of the increased data capture rates and offers the potential for real time imaging with enhanced spatial resolution.
In a typical Nipkow disc configuration (
Sub native optical resolution spacing between the scanned tracks of the microlenses and pinholes can be obtained by using techniques such as that described in
For a 1-D array scanning confocal configuration the scans are illustrated in
In a 2-D array scanning confocal system, the sub native optical resolution increments along the normal scanning direction can be set by precise control of the image scanning galvanometer. If only a single pattern of microlenses and pinholes is used for scanning then an additional mechanism is required to shift the scan in sub native optical resolution steps in the direction perpendicular to the normal scan direction. This can use techniques such as an additional galvanometer mirror 520 (FIG. 5B) or an adjustable tilting thick optical window 510 (
An alternative method requires a change to the microlens and pinhole arrays such that the pattern of microlenses and pinholes is extended, either to include additional pinholes that interleave with the existing pattern or including additional pattern repeats 410, 411, 412 (
Each of the embodiments requires a means to synchronize the image capture to a stationary pattern of illumination spots in the sample. In the case of spinning disc configurations it is not trivial to drive the disc to a specific rotation angle and to step it to an adjacent angle quickly and accurately due to the inertial mass of the disc and its drive motor, etc., so a means of apparently stopping the disc motion is required. This may take the form of a stroboscopic control of the light source, where the illumination source, typically but not exclusively a laser, is turned on and off extremely rapidly. A laser light source may be modulated by direct modulation of a laser diode, by passing the laser beam through an electro-optic or acousto-optic device such as an AOM or AOTF, or a high speed shutter. An alternative method is to illuminate the sample continuously and to gate the detector, such that the detector exposure time is very short and effectively ‘stops’ the disc motion. The cue for each image capture is derived from the rotational position of the discs provided by an encoder device attached directly to the shaft of the discs or to the drive motor, or via an encoder track (magnetic or optical) formed on at least one of the discs and a suitable detection device that reads the encoded track.
The 1-D array scanner can be driven step by step on each axis, using any of the scan moving mechanisms detailed in the embodiments previously described, thus the image capture can be readily synchronized with these movements. It is also possible to continuously scan with the 1-D array scanner and use stroboscopic illumination or detector gating as described for the spinning disc embodiments.
The 2-D array scanner can also be driven step by step on each axis, using any of the scan moving mechanisms detailed in the embodiments previously described, thus the image capture can be readily synchronized with these movements. It is also possible to continuously scan with the 2-D array scanner and use stroboscopic illumination or detector gating as described for the spinning disc embodiments.
Each embodiment contains an electronic or computer subsystem that coordinates the movement of the scan, control of the illumination, the capture of the images, and the data processing steps.
The data processing steps are described with reference to
The sample in the field of view has coordinates j,k corresponding to the coordinates j,k in the prior art reference.
The multibeam spatial filter has coordinates of x,y for the positions of the microlens or pinhole elements in the filter.
The spatial filter scan position coordinates are u,v
For each scan position u,v an image is captured in which an array of ROIs T(x,y) is recorded. Each ROI has dimensions p,q and is centered on the coordinates x,y.
For each of the multi-beam scanner types, the ROIs centered on each beam position are captured simultaneously and the intensity data from the ROIs is loaded into the appropriate elements of the four-dimensional array I(R(u+x,v+y),Spq) according to the scan position u,v and ROI center position x,y after testing that the values of u+x and v+y lie within the image field of view dimensions j and k respectively. Such computational processing may be carried out in parallel, thus improving the speed of creation of the array I(R(u+x,v+y),Spq) which is equivalent to the array I(Rjk,Spq) of the prior art reference. This equivalent array is processed according to the prior art to produce an image with enhanced spatial resolution.
In the case of spinning disc scanners the ROIs are centered on positions r,phi in the spatial filter. The position of the spatial filter is given by d, theta. These positions are converted from polar to rectangular coordinates before testing if they lie within j,k.
| Number | Date | Country | Kind |
|---|---|---|---|
| GB1107556 | May 2011 | GB | national |