Patent Application
20210110519
The present invention relates generally to image analysis methods for the assessment of tissue samples. More specifically, the present invention relates to image analysis methods for the evaluation of multiple biomarkers across multiple tissue sections of the same tissue sample.
Tissue samples are generally preserved, embedded in a block of paraffin, cut into thin sections, and one section placed on a glass slide. This section is typically then further prepared and stained for viewing. Stains can be either chromogenic or fluorescent, which are visible in a brightfield and a fluorescence microscope, respectively. The staining aids in viewing the tissue slide, which otherwise is so thin as to be nearly transparent, but can also be used to tag specific components of the tissue, for example highlighting cells that express a specific protein.
It is difficult to use many different stains on a single slide. There exist some methods that allow the evaluation of multiple biomarkers on the same slide, but workflows are limited to two or three different stains. The limitation comes both from the difficulty of combining the stains without cross-reactivity, and from the difficulty of distinguishing the colors on the slide.
Reducing the number of stains used for an assay makes the assay less expensive, easier to use in the clinic, and more robust. There is a struggle in the industry to reduce the total number of stains in an assay while still increasing the amount of information gathered from the assay.
Tissue preparation and staining is well established in the field, and of common knowledge to one of ordinary skill in the art. Typical methods for tissue staining involve using either chromogenic or fluorescent stains on a single tissue section.
Both brightfield and fluorescent imaging is well understood by one of ordinary skill in the art. It is standard practice to use high-powered fields when using traditional microscopy and whole slide imaging with a digital pathology workflow.
Many methods are proposed in the literature for image alignment or registration, which can be broadly separated into rigid and elastic methods. Rigid registration allows only for a rotation and translation to match one image to the other, whereas elastic registration warps one of the images to match the other. For alignment of whole-slide images of consecutive tissue sections, elastic registration methods are necessary. A rigid registration cannot handle changes in morphology from one section to the next, nor the different stretching, tears and folds that can occur in each of the tissue sections during the mounting process. However, elastic registration is computationally expensive and not precise enough for the use cases covered by the present invention.
As an alternative to aligning two images based on their individual pixels, it is possible to identify specific objects of interest in the two images, and then match up these objects. In the case of muscle fibers, serial sections will show the same set of fibers with very similar local configuration. And there are other tissue objects that also are large enough to show in multiple serial sections. This fiber to fiber matching can then be used to transfer per-fiber information obtained in one image to fibers in the other image, or it can be used to derive an elastic transformation (warping) of one image to match the other.
The fiber to fiber matching can be accomplished with point cloud registration algorithms that have been developed primarily within the robot vision field. However, these algorithms are typically designed for point clouds representing surfaces in three dimensions, and they derive a transformation that matches a surface in one point cloud set to the same surface in a second point cloud set. These algorithms don't typically match up individual points directly. Consequently, the resulting matching is not necessarily optimal.
In accordance with the embodiments herein, a method for aligning digital images of two tissue sections from the same tissue sample is disclosed. The method described herein generally utilizes digital image analysis of tissue objects within the digital images of the two tissue sections. Generally, each tissue section is processed prior to digitization in such a way to ensure that tissue objects within the section can be properly identified. Once the tissue objects within each digital image have been identified, the image analysis features of the tissue objects are extracted from the digital image. These image analysis features are then used to determine tissue object correspondences between the two digital images, such that tissue objects in the first digital image correspond with tissue objects in the second digital image. These correspondences are then used to align the two digital images.
In the following description, for purposes of explanation and not limitation, details and descriptions are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments that depart from these details and descriptions without departing from the spirit and scope of the invention.
For purpose of definition, a tissue object is one or more of a cell (e.g., immune cell, muscle fiber), cell sub-compartment (e.g., nucleus, cytoplasm, membrane, organelle), cell neighborhood, a tissue compartment (e.g., tumor, tumor microenvironment (TME), stroma, lymphoid follicle, healthy tissue), blood vessel, a lymphatic vessel, vacuole, collagen, regions of necrosis, extra-cellular matrix, a medical device (e.g., stent, implant), a gel, a parasitic body (e.g., virus, bacterium), a nanoparticle, a polymer, and/or a non-dyed object (e.g., metal particle, carbon particle). Tissue objects are visualized by histologic stains which highlight the presence and localization of a tissue object. Tissue objects can be identified directly by stains specifically applied to highlight the presence of said tissue object (e.g., hematoxylin to visualize nuclei, IHC stain for a protein specifically found in a muscle fiber membrane), indirectly by stains applied which non-specifically highlight the tissue compartment (e.g., eosin staining), are biomarkers known to be localized to a specific tissue compartment (e.g., nuclear-expressed protein, carbohydrates only found in the cell membrane), or can be visualized without staining (e.g., carbon residue in lung tissue).
For the purpose of this disclosure, patient status includes diagnosis of inflammatory status, disease state, disease severity, disease progression, therapy efficacy, and changes in patient status over time. Other patient statuses are contemplated.
In an illustrative embodiment of the invention, as summarized in
In a further embodiment shown in
In another embodiment, the image analysis features include morphometric features, localization features, neighborhood features, and staining features of the tissue objects within the tissue sample. Morphometric features are features related to the size, shape, area, texture, organization, and organizational relationship of tissue objects observed in a digital image. For example, and not limitation, morphometric features could be the area of a cell nucleus, the completeness of biomarker staining in a cell membrane, the diameter of a cell nucleus, the roundness of a blood vessel, lacunarity of biomarker staining in a nucleus, etc.
Localization features are features related to position of a feature in the tissue section, spatial relationships of tissue objects relative to each other, relationship of image analysis features between tissue objects in the tissue section, and distribution of image analysis features within a tissue object. Location can be determined based on an absolute (e.g., x and y location based on pixel dimensions of image, μm from center of image defined by pixel dimensions of image) or relative (e.g., x and y position of cells relative to a tissue feature of interest such as a vessel, polar coordinates referenced to the center of mass of a tumor nest) coordinate system (e.g., x-y-z coordinates, polar coordinates). Location for specific image objects can be defined as the centroid of the object or any position enclosed by the object extending from the centroid to the exterior limits of the object.
Neighborhood features are features related to tissue object morphology within a distance of an anchor tissue object, tissue object staining within a distance of an anchor tissue object, and morphology and/or staining between tissue objects within a distance of an anchor tissue object. For example, and not limitation, neighborhood features could be the average size or area of cells within 100 microns of an anchor cell or the quality or quantity of staining of cell nuclei within 500 microns of an anchor cell nucleus.
Staining features are features related to stain appearance, stain intensity, stain completeness, stain shape, stain texture, stain area, and stain distribution of specified immunohistochemistry, in situ hybridization, and immunofluorescence stains or dyes or amount of a molecule determined by mass spectroscopy based imaging methodologies. Staining features are evaluated relative to tissue objects (e.g., average staining intensity in each cell in an image, staining level in a cell membrane, biomolecule expression in a nucleus).
In an additional embodiment, the tissue object correspondences can be determined using tissue object morphology, such as morphometric features, or using spatial relations between neighboring tissue objects, such as localization and neighborhood features. When determining what are neighboring tissue objects, a number of different methods can be used. Physical proximity using a Delaunay triangulation, a maximum fixed distance, or a maximum number of nearest objects can be used to determine if a tissue object is a neighboring tissue object to an object of interest.
In a further embodiment, the tissue sections may be stained for biomarkers. If the tissue sections are stained for a biomarker, the biomarker can be quantified for each tissue object. This biomarker quantification data would be specific to the digital image that contains the tissue object of interest. The biomarker quantification data from the two digital images can be combined using the tissue object correspondences, or even once the digital images are aligned. This can lead to a version of multiplexing, where each tissue section is stained in such a way to highlight a different biomarker, yet both the biomarker information from each tissue section is easily interpreted into the other tissue section.
With the combined biomarker quantification data, a patient status for the patient from whom the tissue sample was taken can be determined. Patient status includes diagnosis of disease state, disease severity, disease progression, and therapy efficacy. Other patient statuses are contemplated.
Unlike the current state of the art, where tissue alignment is largely restricted to consecutive sections, a further embodiment of the present invention can be used to align consecutive serial sections, non-consecutive serial sections, and even non-serial sections. The present method provides for ample correction of error, both systemic and human error, as consecutive serial sections are not always available for analysis. The present invention provides for a way to correct that issue.
In another embodiment, as shown in
Similarity scores can be calculated using analysis of the image analysis features of the two tissue objects that may be matched, analysis of the relative spatial location of the two tissue objects that may be matched within their respective tissue sections, a combination of these methods, or other methods. A most likely matching tissue object may be one that is spatially closest to the same location as the tissue object of interest based on the neighboring matches or one that has the best similarity score in a small region around that point. This small region would typically be no more than 200 μm across, though could be enlarged to accommodate larger tissue deformations incurred for example by tears. The combination of these measures may be used as well to determine the most likely matching tissue object. Other methods are contemplated to determine a most likely matching tissue object.
Matching requirements are typically based on spatial ordering of the new match and previously matched neighboring tissue objects, a preset minimum similarity score, a maximum local spatial displacement, or some combination of these factors.
