1. Field of the Invention
This invention relates in general to the field of microscopy and, in particular, to a novel approach for providing multi-spectral illumination for pathologic analysis of tissue slides.
2. Description of the Prior Art
Changes in the cellular structure of tissue are used to detect pathologic changes, such as to assess the progress of precancerous conditions and to detect cancer. A tissue sample removed from a patient is typically sectioned and fixed to a slide for staining and microscopic examination by a pathologist. The morphology of the tissue (the visually perceptible structure and shape of features in the tissue) is analyzed to provide a qualitative assessment of its condition and to identify the presence of pathologic changes that may indicate progression towards a malignancy. For many decades, this visual procedure has been the diagnostic mainstay of pathology.
With the advent of computers and sophisticated digital imaging equipment, researchers have extended the realm of histopathology through the use of mechanized procedures for diagnostic and quantitative investigation. In such mechanized procedures, histopathologic sections and/or cytologic preparations are imaged with a microscope, and the images are digitized, stored, and analyzed for nuclear-placement patterns (histometry) or for the spatial and statistical distribution patterns of nuclear chromatin (karyometry). Karyometric assessment is always preceded by image segmentation, whereby each nucleus in an image is identified, outlined, isolated and stored as a separate image.
Each stain provides information on the localization of particular molecules in tissue or associated with different structural elements of the tissue section (e.g., nuclei). Therefore, a detailed knowledge of every stain's spatial distribution offers a distinct perspective about the biological material under review. Accordingly, localization of diverse stains in a single of tissue is a challenge of growing significance.
The conventional approach has been to employ at most two stains (e.g., hemotoxylin and eosin) in every tissue section because the human eye is often not capable of distinguishing pattern produced by additional stains. Increasing the number of stains can be problematic also for computerized analysis because of overlap between the stains' absorbance spectra. A conventional imaging system that captures a red, green, and blue (“RGB”) image of the tissue section does not provide, for example, information that may be used unambiguously to separate the absorbance contributions of each stain at a detector pixel. Therefore, the use of more than two stains in the same tissue sample would not necessarily provide additional information with conventional microscopes even through mechanized analysis.
A desirable approach would be to stain the tissue sample with additional stains and collect more than three spectral channel images (colors) at every pixel, thereby providing sufficient spectral sampling to uniquely and unambiguously separate the absorbance contributions of each stain to each pixel of the detector. However, this approach has been difficult to implement with good results because of practical tradeoffs between source brightness, detector sensitivity and resolution, optics magnification, speed of acquisition, and computational requirements.
Systems for brightfield and fluorescence multispectral recording of specimens have been described (see U.S. Pat. No. 6,690,466 and U.S. Pat. No. 6,373,568 and are commercially available. Three different approaches are used to collect the multispectral information. For example, spectral filtering is used in the Nuance™ system sold by Cambridge Research and Instrumentation, Inc. (“Cri”), of Woburn, Mass.; scanning optical-path difference (OPD) in the interferometer SpectraView® sold by Applied Spectral Imaging, Inc., of Vista, Calif.; and diffraction grating in the LSM 510 META microscope manufactured by Carl Zeiss, Inc., of Germany. CRi's system has a spectral source constructed of multiple-color LEDs that can be individually controlled, thus allowing for an illumination source of adjustable spectrum. The light source can be used to either illuminate the sample with a sequence of “pure wavelengths” using one type of LED at a time or using a mixture of LEDs simultaneously.
All existing systems have in common the use of commercial microscope objectives and remain constrained by the field-of-view/numerical-aperture trade-off associated with these optical elements. These systems are primarily intended for collecting images of a single objective field of view rather than whole-slide scanning. Accordingly, these microscope optics enable two main image-acquisition approaches, by the so-called “step-and-repeat” and “push-broom” scanning modes. In the step-and-repeat approach, individual image fields (also known as “tiles”) are recorded and the stage bearing the microscope slide is advanced to the next field, as illustrated in
Pushbroom scanning is based on the use of a linear detector array covering the diameter of field of view of the microscope objective. The approach results in a swath that could cover the entire length of a slide, to be repeated, after a step-over, typically 20-30 times to cover a 20 mm-wide slide. This approach is illustrated in
A problem with both these image-acquisition approaches is that they require an overlap between the individual tiles or swath scans to ensure complete imaging coverage of the region of interest. Thus, stitching of image tiles or swaths and additional computation are required to determine the extent of redundant image data. Therefore, a more straightforward approach to generating multi-spectral images of tissue slides would represent a useful advance in the art. The present invention utilizes an array microscope to advantageously improve multi-spectral imaging of tissue slides.
In essence, the invention lies in the use of an array microscope that makes it possible to scan a whole sample slide in rapid sequence at different illumination wavelengths to record multiple spectral images of the same sample. As a result of this approach, full spatial resolution of the image sensor is realized at each color because pixels are not shared between spectral bands. The image quality and focus can be optimized separately at each color, instead of relying on conventional chromatic correction, which is not sufficiently perfect when implemented with commercially affordable optical elements, such as those made of plastic materials. Image-sensor response can be carefully calibrated separately for each color.
As a critical aspect of the invention, the various images collected at different wavelengths must be produced with the same magnification in order to ensure registration of all images. This is achieved by adjusting the position of the object and/or detector for each wavelength so that they are placed at the optimal conjugate distances from the objective array.
Image spectral analysis is carried out by combining the images captured at each wavelength to produce a vector of spectral responses for each detector pixel, thereby yielding spectral information that is not available when broadband illumination is used. On the basis of this information and the corresponding greater spectral resolution, the contemporaneous effects of a larger number of stains on a tissue sample may be isolated and displayed for improved pathological analysis. The summation of spectral images may be weighted in some fashion (such as by using a filter vector with positive and/or negative weights) to further improve the diagnostic significance of the spectral resolution produced by the invention.
Various other advantages will become clear from the description of the invention in the specification that follows and from the novel features particularly pointed out in the appended claims. Therefore, to the accomplishment of the objectives described above, this invention consists of features hereinafter illustrated in the drawings, fully described in the detailed description of the preferred embodiments, and particularly pointed out in the claims. However, such drawings and descriptions disclose only some of the various ways in which the invention may be practiced.
Co-owned International Application PCT/US02/08286 and U.S. patent application Ser. No. 10/158,626, herein incorporated by reference, describe a new approach to microscopy. An array microscope is disclosed that comprises a plurality of optical imaging elements configured to image respective sections of an object and disposed with respect to an object plane so as to produce at respective image planes respective images of the respective sections of the object measurements. The object may be illuminated in a variety of ways, i.e., by trans-illumination, epi-illumination, or epi-fluorescence, as these terms are understood in the art.
As illustrated in
Because of the staggered arrangement of the rows of lenses, the continuous strip covered by the linear scan of each optical system 16 is substantially free of overlap with continuous strips covered by other optical systems. Thus, at each acquisition frame each system 16 projects image data for a small section of the sample object 20 directly onto a pixel detector 24 and the sets of individual intensity data are then combined to form a composite image of the entire object by hardware and/or software manipulation. The details of implementation of array microscopes are disclosed in copending Ser. No. 10/158,626.
The use of array microscopes is based on the realization that small optical systems can provide good-quality, better-than RGB spectral resolution imaging with acceptable magnification. Accordingly, each individual optical system in the array is designed to perform such a function and a plurality of systems is packed together as closely as possible within the constraints of the physical size of each component. A typical individual microscope system used in an array microscope is shown in
According to the present invention, the array microscope of Ser. No. 10/158,626 is used to sequential, whole-slide images of the sample using different spectral sources, such as light diodes (LEDs). A collection of LED light sources, such as available from the Philips Lumileds Lighting Company of San Jose, Calif., is used to trans-illuminate in sequence the same glass slide. The glass slide is held in place, preferably using vacuum, during each scan in the sequence. Scanning can be performed in the same direction (only one direction of motion is necessary) or in two opposite directions. The latter approach results in a slightly faster operation of the scanner. With reference to the scanning direction of the imaging system of the invention, as described and claimed, the term “linear” is intended to cover a straight as well as a curvilinear path during which each objective of the microscope array acquires image data (light intensity) corresponding to a respective straight or curvilinear continuous strip of the object.
The process is preferably initiated by using the array microscope for a rapid, low-resolution scan of the glass slide with broadband light to assess generally the tissue and determine a region of interest. The resultant region, normally rectangular in shape, defines the length and width of the subsequent scans at each wavelength. Thus, the preferred configuration of the system is set up for scanning the same area of the slide at each wavelength of illumination.
As illustrated in
In multi-spectral imaging for computational analysis of information obtained from tissue slides stained with more than two colors, it is critical to have identical transverse magnification for all sampled wavelengths, so that the separate color images may be combined for meaningful analysis. The important point is to be able to measure the contribution of each wavelength to the measurements acquired at each pixel of the detector, thereby producing a color vector for each pixel that subsequently can be used to analyze the slide with respect to each stain used in the pathological process. Due to the unavoidable chromatic aberration associated with optical elements, the system of the invention needs to be corrected to guarantee a substantially constant transverse magnification at each wavelength used in the sequential scan.
In the microscope of the invention, which utilizes an array of custom-made objectives in the configuration of
As one skilled in the art would readily understand, the small amount of variation in transverse magnification produces a small difference in principal ray height on the image surface for different wavelengths at different field positions. A plot of this difference is shown in
In order to achieve the substantially constant magnification reflected in
The approach illustrated above has several advantages. Very significantly for the purposes of pathology analysis, full spatial resolution of the image sensor is realized at each color because pixels are not shared between spectral bands. Furthermore, image quality and focus can be optimized at each separately, instead of relying on the imperfect chromatic correction provided even by an apochromatic microscope objective. Image-sensor response can be carefully calibrated for each color separately and image spectral analysis can be performed by storing weighted whole-slide image at each wavelength. Then, all spectral images may be summed at every pixel to provide a spectral filter vector with positive and/or negative weights as deemed necessary achieve desirable results. For example, a separate set of gains and offsets can be used for each color channel in order to equalize the variations among image-sensor detector responses and illumination variations. The gains and offsets can be modified by the introduction of a multiplicative factor that weighs a particular spectral image. The summation of the weighted, gain-and-offset-corrected, images results in a projection of each pixel's spectrum onto a vector, such as a particular principal component.
It is noted that these techniques of image manipulation are old in the art of spectral analysis. The contribution of this invention lies in an approach and tool for producing registered spectral images of the sample object with grater spectral resolution than heretofore possible.
In operation, the array microscope is first focused on the tissue mounted on the glass slide. This is done by performing a complete scan using at least one of the available illumination wavelengths, such as green, for example. A conventional autofocus approach may be used, focusing the array at each scan position to acquire and record the best-focus position according to the topography of the tissue as viewed by the objective, the tissue being mounted on the glass slide under the transparent cover glass or thin plastic tape typically used for pathology microscopy. A procedure for producing such a best-focus map is described, for example, in copending Ser. No. 10/431,937, hereby incorporated by reference.
The array microscope next scans the glass slide using one illumination color at a time. During each scan, the optics roll and pitch angles and the height of the array of objectives relative to the slide are adjusted to maintain focus at all individual objectives in the array on the basis of the information acquired during the initial auto-focus scan. In addition, for each illumination wavelength, the optics' object and image distances are adjusted, according to the invention, in order to maintain as much as possible a constant magnification throughout the field of view. This is achieved by shifting the trajectory obtained from the initial autofocus operation outlined above by an additional amount up or down according to the conjugates associated with each illumination color shown in Table 2.
As a result of this multi-spectral scanning process, the array microscope produces a series of registered spectral images. The registration is achieved, as explained, by an optical design that minimizes spectral dependence of distortion and a mechanical design that holds the glass slide securely in place during scanning. Based on current performance, the digital slide scanner described herein is capable of capturing each illumination-color image in approximately 18-19 seconds, covering a sample area of 15 mm by 15 mm.
Thus, it has been shown that the microscope array of the invention can be used advantageously to provide an improved tool for fast and reliable mechanized analysis of pathology slides. The improved spectral resolution produced by the invention makes it possible to stain tissue samples at the same time with more than two colors (the practical limit of prior art approaches), thereby enabling a more precise segmentation and identification of items of interest in the samples. This level of better-than-RGB spectral resolution, which is not visible in color images, also enables mechanized analysis that would not be possible by visual inspection of a pathologist, thereby improving the diagnostic efficacy of histopathology.
While the invention has been shown and described herein in what is believed to be the most practical and preferred embodiments, it is recognized that departures can be made therefrom within the scope of the invention. Therefore, it is not to be limited to the disclosed details but is to be accorded the full scope of the claims, including any and all equivalents thereof.
This application is based on Provisional Application No. 60/753,879, filed Dec. 23, 2005, and is a continuation-in-part application of U.S. Ser. No. 10/637,486, filed Aug. 11, 2003, which is a continuation application of PCT/US02/08286, filed Mar. 19, 2002, and claims the benefit of priority of U.S. Provisional Application No. 60/276,498, filed Mar. 19, 2001, under 35 U.S.C. Sect. 119.
Number | Name | Date | Kind |
---|---|---|---|
5052238 | Sewell | Oct 1991 | A |
6373568 | Miller et al. | Apr 2002 | B1 |
6396053 | Yokoi | May 2002 | B1 |
6690466 | Miller et al. | Feb 2004 | B2 |
7369307 | Wolleschensky | May 2008 | B2 |
7586530 | Onozawa et al. | Sep 2009 | B2 |
7697382 | Mizutani et al. | Apr 2010 | B2 |
7697808 | D'Urso et al. | Apr 2010 | B2 |
20020036824 | Sasaki | Mar 2002 | A1 |
20050036197 | Awamura | Feb 2005 | A1 |
20090109527 | Sasaki et al. | Apr 2009 | A1 |
20090296207 | Goelles et al. | Dec 2009 | A1 |
Number | Date | Country | |
---|---|---|---|
20070153370 A1 | Jul 2007 | US |
Number | Date | Country | |
---|---|---|---|
60753879 | Dec 2005 | US | |
60276498 | Mar 2001 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 10637486 | Aug 2003 | US |
Child | 11644564 | US | |
Parent | PCT/US02/08286 | Mar 2002 | US |
Child | 10637486 | US |