Patent Application
20190178867
The present invention relates generally to image analysis methods for the assessment of tissue samples. More specifically, the present invention relates to image analysis methods for the evaluation of tissue objects within a tissue sample without directly staining for those tissue objects.
Tissue samples are generally preserved, embedded in a block of paraffin, cut into thin sections, and one section placed on a glass slide. This section is then further prepared and stained for viewing. Stains can be either chromogenic or fluorescent, which are visible in a brightfield and a fluorescence microscope, respectively. The staining aids in viewing the tissue slide, which otherwise is so thin as to be nearly transparent, but can also be used to tag specific components of the tissue, for example highlighting cells that express a specific protein.
It is difficult to use many different stains. With a brightfield microscope, we are typically limited to two or three chromogenic stains (not only because of the difficulty of combining the stains, but mostly because of the difficulty of distinguishing the colors on the slide). In digital imaging, using an RGB camera, no more than three stains can be separated consistently. This limit is flexible if stains are not collocated, or if using more complex imaging methods.
With a fluorescence microscope it is easier to distinguish more dyes, because fluorescent dyes can be separated by their wavelengths. However, fluorescence staining and imaging has other problems that make it less suitable for pathology in the clinic, such as reduced stain longevity compared to chromogenic staining.
Reducing the number of stains used for an assay makes the assay less expensive, easier to use in the clinic, and more robust. There is a struggle in the industry to reduce the total number of stains in an assay while still increasing the amount of information gathered from the assay.
Tissue preparation and staining is well established in the field, and of common knowledge to one of ordinary skill in the art. Typical methods for tissue staining involve using either chromogenic or fluorescent stains on a single tissue section.
Both brightfield and fluorescent imaging is well understood by one of ordinary skill in the art. It is standard practice to use high-powered fields when using traditional microscopy and whole slide imaging with a digital pathology workflow.
Many methods are proposed in the literature for image alignment or registration, which can be broadly separated into rigid and elastic methods. Rigid registration allows only for a rotation and translation to match one image to the other, whereas elastic registration deforms one of the images to match the other. For alignment of whole-slide images of consecutive tissue sections, elastic registration methods are commonly applied. Rigid registration is not favored in the art as it typically does not provide the same quality of alignment compared to an elastic alignment for consecutive tissue sections.
As an alternative to aligning two images based on their individual pixels, it is possible to detect, for example, cells in both images, then align the cells based on their relative positions. This can be accomplished with efficient point cloud registration algorithms that have been developed primarily within the robot vision field. This registration process leads to a one-to-one assignment of cells in one image to cells in the other image. This assignment can then be used to transfer cell information obtained in one image to cells in the other image, or it can be used to derive a rigid or elastic transformation of one image to match the other. However, in the industry, as alignment is typically used for images of multiple tissue sections, the transfer of information is limited in scope, as the distance between tissue sections is such that at most a small fraction of cells will show in both sections. This lack of correspondences can make this method very inaccurate for consecutive sections, and useless if the distance between sections increases.
In accordance with the embodiments herein, a method for detecting and identifying tissue objects without direct staining is disclosed. The method described herein generally utilizes digital image analysis of a pair of images, one brightfield and one fluorescence, of a stained tissue section. The tissue section must be stained with a chromogenic stain and a fluorescent stain that stains tissue objects, such as immune active cell clusters, imaging the stained tissue section in both brightfield and fluorescence image modalities, aligning the digital images using any of number of image alignment techniques, and analyzing the aligned images, such that staining from the fluorescence image can be used to identify cells within the brightfield image.
In the following description, for purposes of explanation and not limitation, details and descriptions are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments that depart from these details and descriptions without departing from the spirit and scope of the invention.
For purpose of definition, a tissue object is one or more of a cell (e.g., immune cell), cell sub-compartment (e.g., nucleus, cytoplasm, membrane, organelle), cell neighborhood, tissue compartment (e.g., tumor, tumor microenvironment (TME), stroma, lymphoid follicle, healthy tissue), blood vessel, and lymphatic vessel. Tissue objects are visualized by histologic stains which highlight the presence and localization of the tissue object. Tissue objects can be identified directly by stains specifically applied to highlight that tissue object (e.g., hematoxylin to visualize nuclei, IHC stain for a protein specifically found in a muscle fiber membrane), indirectly by stains applied which non-specifically highlight the tissue compartment (e.g., DAB staining), or are biomarkers known to be localized to a specific tissue compartment (e.g., nuclear-expressed protein, carbohydrates only found in the cell membrane).
For the purpose of definition, patient status includes diagnosis of disease state, disease severity, disease progression, and therapy efficacy. Other patient statuses are contemplated.
In an illustrative embodiment of the invention, as summarized in
In the above embodiment, the steps of chromogenic staining, fluorescent staining, brightfield imaging, and fluorescence imaging can be performed in any logical order to obtain the disclosed invention. For example, the fluorescent staining may be followed immediately by the fluorescence imaging, and then the same tissue sample could be stained with the chromogenic dye and the brightfield image taken. Or the order could be reversed, such that chromogenic staining and brightfield imaging are followed by fluorescent staining and fluorescence imaging. It is also understood that the tissue sample could be stained with both chromogenic and fluorescent dyes, then both brightfield and fluorescence images could be acquired. However, it is understood that the illogical order of imaging before staining with appropriate dyes for that imaging modality is not part of the present invention.
In other embodiments, a dye that is visible in both imaging modalities, such as Fast Red, could be used to more easily identify tissue objects within both the brightfield and fluorescence digital images for alignment. A pair of dyes could also be used that stain for the same type of tissue objects, such as the cell nuclei, with one dye visible in each imaging modality, such as Hematoxylin and DAPI.
In another embodiment, the brightfield digital image and the fluorescence digital image can be aligned using common element between the two images and then the aligned image is analyzed to identify the tissue object in the brightfield digital image, as shown in
In other embodiments, additional steps can be added to the previously disclosed method to allow for using the identified tissue objects to determine patient status for the patient from which the tissue sample was taken. This is performed by creating a score based on the identified tissue objects, then using that score and an established scoring scheme to determine that patient's status related to disease state, disease severity, disease progression, and/or therapy efficacy, along with other potential statuses related to diagnosis, prognosis, and treatment.
Additional embodiments include correcting for color cross-talk in the fluorescence digital image such that each color channel within the digital image contains information related to only a single dye. This additional step is particularly useful when multiple fluorescent dyes are present in the tissue sample, as the emission spectra of most fluorescent dyes have a long tail that can overlap with the emission of another fluorescent dye, thus contaminating the color channel associated with the overlapped dye.
One application of this method to identify tissue objects without directly staining the tissue object is in the use of computer aided diagnostics, such that a computer performs the diagnostic determination of the tissue sample for the patient. The core concept there is to teach the computer to distinguish specific tissue objects based on their morphology (their appearance in the brightfield image) while avoiding the need for a specific stain. This process requires lots of examples of the cells of interest, to allow the computer to generalize from these examples and be able to distinguish them in other settings. By automating how such examples are generated, this method reduces the need for hundreds, if not thousands, of hours of human pathologists to hand identify the tissue objects. Some tissue objects, such as specific immune cells, are notoriously difficult to identify without specific staining. The process thus is: (i) obtain tissue samples, prepared, imaged and analyzed as described in this invention, to derive a large set of example brightfield images of tissue objects stained with chromogenic dyes but not a dye that can be used to identify them; (ii) train a machine learning system the appearance of these cells; and (iii) apply the machine learning tool to brightfield images of tissue to identify those cells. Note that, by using the fluorescence modality to identify the cells of interest for the training set, the brightfield image appearance of these cells is not affected by the specific dye, which would introduce a bias in the machine learning tool.
This application is a continuation-in-part (CIP) of U.S. Ser. No. 15/396,552, filed Dec. 31, 2016, and titled “METHODS FOR DETECTING AND QUANTIFYING MULTIPLE STAINS ON TISSUE SECTIONS”; the contents of which are hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15396552 | Dec 2016 | US |
| Child | 16271525 | US |
