Cell Detection Studio: a system for the development of Deep Learning Neural Networks Algorithms for cell detection and quantification from Whole Slide Images

Abstract

The invention is made out of methods for the development of Deep Neural Networks for cell detection and quantification in Whole Slide Images (WSI): 1. Method to create generic cell detector that detects the centers and contours of all cells in a WSI.2. Method to create algorithms to detect cells of specific categories and that can classify between various types of cells of different categories.3. Method for efficient cell annotation with online learning.4. Method for efficient cell annotation with active learning.5. Method for efficient cell annotation with online learning and data balancing.6. Method for auto annotation of cells7. Cell Detection Studio: a method to create an AI based system that provides pathologists with a semi-automatic tool to create new algorithms aiming to find cells of specific categories in WSI digitally scanned from histological specimen

Description

BACKGROUND OF THE INVENTION

Field of the Invention

The invention relates to the application of methods of image processing, computer vision, machine learning and deep learning to create new algorithms for the detection of specific types of cells in Whole Slide Images (WSI) obtained by scanning the biopsies with a digital scanner.

In pharma research and medical diagnosis, the detection and quantification of specific types of cells, e.g. lymphocytes, is important. The usual practice is that the pathologist views the slide under a microscope and roughly estimates the number and density of the cells of interest. The availability of high resolution digital scanners for pathology that produce digitized WSI allows the development of state of the art Computer Vision and Deep Learning methods for cell detection and quantification. Different applications require the detection of different cells. Each new cell detection algorithm usually requires two major efforts: the first is the annotation of the cells of interest by an expert pathologist, and the second is the development of specific computer vision and deep learning algorithms tailor made for the detection of the specific cells of interest. Both efforts require dedicated expert teams and resources.

The invention provides a tool to be used by pathologists that allows them to create new algorithms for specific cell detection. The invention also provides a tool for rapid annotation of image patches taken from WSI, as well as a visualization tool for the cells detected.

Description of the Related Art

The ability to automatically detect certain types of cells in pathology images and to localize them is of significant interest to a wide range of pharma research and clinical practices. Cell detection is a common task that is routinely performed by pathologists, who examine slides under a microscope and provide an estimation of the quantity and density (or other attributes) of the cells based on their empirical assessments. These assessments are generally time consuming and tedious and are prone to fatigue induced errors.

For example, the presence of tumor-infiltrating lymphocytes (TILs), have become a central research topic in oncology and pathology. Immunohistochemical staining (IHC) is a technique that allows to target specific cell types, including lymphocytes, by attaching a colored label to a specific antigen in (subcompartment of) a cell. In this way, immune cells can be distinguished from other type of cells.

Accurate detection and assessment of presence of lymphocytes in cancer could potentially allow for the design of new biomarkers that can help monitor the rapid progression of a tumor. Moreover, automated tools to quantify the immune cells density and their localization in the proximity of tumor cells might help to predict the presence and development of metastases and overall survival of cancer patients. In addition, it allows personalized treatments that can significantly benefit the patients.

Given the very large amount of lymphocytes (≈100,000) in a single cancer tissue specimen, manual assessment at whole-slide image level is a very tedious, time-consuming, and therefore unfeasible task. Moreover, manual assessment suffers from intra- and inter-observer variability. Consequently, a method for automatic detection and quantification of immune cells is of great research and clinical interest.

Moreover, once a cell detection capability is available various quantitative attributes such as cellular morphology, size, shape and texture can be calculated.

The task of cell detection is a very popular topic in digital pathology. Computer-aided methods provide faster image analysis and can significantly improve the objectivity and reproducibility of cell detection. Moreover, the basic science researchers and clinician scientists can be released from boring and repeated routine efforts. Several approaches have been proposed for automatic cell detection on different types of digitized microscopical specimens and for various types of stained specimens. In many cases, detection algorithms are based on morphological operations, region growing, analysis of hand-crafted features and image classifications.

Cell detection and localization constitute several challenges. First, target cells are surrounded by clutters represented by complex histological structures like capillaries, adipocytes, collagen etc. In many cases, the size of the target cell is small, and consequently, it can be difficult to distinguish from the aforementioned clutter. Second, the target cells can appear very sparsely (only in tens), moderately densely (in tens of hundreds) or highly densely (in thousands) in a typical WSI. Additionally, significant variations in the appearance among the targets can also be seen. Moreover, due to the enormous variability (cell types, stains and different microscopes) and data complexity (cell overlapping, inhomogeneous intensities, background clutters and image artifacts), robust and accurate cell detection is usually a difficult problem that requires a dedicated R&D effort of experienced algorithms developers.

Cell detection methods have evolved from employing hand-crafted features to deep learning-based techniques. Traditional computer vision based cell detection systems adopt classical image processing techniques, such as intensity thresholding, feature detection, morphological filtering, region accumulation, and deformable model fitting. Deep neural networks recently have been applied to a variety of computer vision problems, and have achieved better performance on several benchmark vision datasets. The most compelling advantage of deep learning is that it has evolved from fixed feature design strategies towards automated learning of problem-specific features directly from the training data. By providing massive amount of training images and problem-specific labels, users do not have to go into the elaborate procedure for the extraction of features. Instead, a deep neural network (DNN) is subsequently optimized using a mini-batch gradient descent method over the training data, so that the DNN allows autonomic learning of implicit relationships within the data.

In order to develop deep learning neural network based cell detection algorithms it is required to first annotate thousands of cells within WSI and then develop a specific cell detection deep learning algorithm. Then, there should be a dedicated R&D effort for the development of the neural network for the detection of the specific cells. This is a major effort that is not readily available for every pathology lab.

This invention is not another cell detection algorithm. This invention provides a “do it yourself” tool for pathologists in order to create new cell detection algorithms suited to the problem at hand without a need to annotate thousands of cells and without the need for a dedicated research and development effort.

In this invention the use of the following components represent a novel contribution to the state of the art and constitute a Cell Detection Studio Framework:

• • 1. Generic Cell Detection.
  • 2. Effective Cell Annotation Tool using online learning, active learning and data balancing techniques.
  • 3. Cell Categories Classifier.
  • 4. Auto Annotation Tool.

SUMMARY OF THE INVENTION

The present invention is a method for automated detection of a cells categories in histological specimens, comprising: providing a specimen-stained slide; obtaining a scanned image of the slide with a digital scanner; detecting all cells in the slide; generating image patches that contain various types of cells; annotating those image patches according to categories; creating a cell classifier for the cell categories annotated; apply the generated algorithm on new whole slide images; detect centers and contours of cell categories of interest; generate a report on various categories of cells attributes.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read with the accompanying drawings in which:

FIG. 1 shows a flow chart summarizing the method of one embodiment of the invention, illustrating the steps from the digital Whole Slide Image to the generation of Cell Categories Detector by a pathologist for the analysis of Cell Categories of interest.

FIG. 2 shows a flow chart for the Generic Cell Detector method.

FIG. 3 shows a block diagram of the Unet neural network architecture that is used for the generic cell detector.

FIG. 4 shows a flow chart for the training of a CNN for different cells categories, where the cells to be categorized are selected from the generic cell detector described in FIG. 2.

FIG. 5 shows a block diagram of the CNN used for Cell Categories classification.

FIG. 6 shows a flow chart for effective cell annotation using online learning.

FIG. 7 shows a flow chart for effective cell annotation using active learning.

FIG. 8 shows a flow chart for effective cell annotation using data balancing strategy for active learning.

FIG. 9 shows a flow chart for a suggested auto annotation for cells.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

FIG. 1 illustrates a flowchart representation of an embodiment of a method of the invention for Cell Detection Studio. The aim is to provide pathologists and researchers with a tool to create a Deep Learning based cell detection tool according to their requirements. In which digital pathology slides are available A1. The pathology slides are generated from tissue biopsies. These tissue biopsies are sectioned, mounted on a glass slide and stained to enhance contrast in the microscopic image. For example, in some embodiments of the invention, the histological specimen slices may be stained, by H&E, Giemsa or immune histochemical staining to enhance certain features in the image. The slide is then loaded on a digital scanner for digitization and the result is a digital file containing the image information.

The resultant image is then submitted to a generic cell detection algorithm A2 that aims to detect all cells in the slide, of any category. A detailed description of the generic cell detector is given in the text for FIG. 2 and FIG. 3.

The result of A2 is a list of all the centers and contours of the cells present in the slide. The image patches extraction module A3 aims to extract crops that surround every cell selected. The size of the image patches can be set as a parameter and its default value is 32. The image patches created are then submitted to interactive image annotation using a GUI (graphical user interface) application A4. Each time a single image crop is presented to the annotator, and using a keyboard press or mouse click, or touch screen tap, the annotator chooses one of the possible categories, each containing a specific type of cell or background. A classification CNN is trained on all the available annotated image patches to create a cell categories classifier A5. Online learning is used to update the cell classification neural network as more annotations become available as described in the text for FIG. 6. Efficient annotation is enabled by active learning described in the text for FIG. 7, and active learning with data balancing strategy described in the text for FIG. 8. Moreover, efficient annotation is further enhanced using an auto annotation scheme when feasible as described in the text for FIG. 9. Once a Cell Categories Classifier is generated the user can test it on new slides and obtain visual and quantitative evaluation of the quality of the cell categories detector generated A6. The quantitative evaluation can be ROC, AUC or other standard metrics used in deep learning. When the user is satisfied with the quality of the generated cell detector it can be applied to new digital pathology slides A7, where the centers and contours of the cell categories that are of interested only are located A8 and a report that provides information on attributes of these cell category can be obtained A9. These attributes can be but are not limited to: number, density, area, location and perimeter.

FIG. 2 illustrates a flowchart representation of an embodiment of a method of the invention for the Generic Cell Detection. In which the input is an annotated dataset of cells of any kind B1. A neural network based on the Unet architecture that is presented in FIG. 3 is trained to obtain cell segmentation B2, where the output are the contours of the segmented cells B3. Data augmentation methods are used to add variability of scale, aspect ratio, staining and other variations to generalize the cell segmentation neural network and make it robust to this possible variability B4. The result is a generic cell detector that can segment all cell types in a whole slide mage B5.

FIG. 3 illustrates a flowchart representation the invention for a Unet based Generic Cell Detector.

The input to the generic cell detection block is patches from whole slide images. The output of the generic cell detection block is contours and centers of all detected cells in each patch. The generic cell detection can be any method for nuclei detection. One method for doing this is using a neural network for semantic segmentation, e.g Unet. In this case Unet is trained on a dataset of cells and outputs the body and the contour segmentation for each cell. The dataset is consisted of annotated cells where each cell has a marked polygon around its border.

In case of a segmentation network based on Unet, the architecture is consisted of an Encoder and a Decoder. The Encoder has 5 convolutional blocks, each with a kernel of 3×3 and a stride of 2. The Decoder has 5 convolutional blocks, each with a kernel of 3×3, and an up sampling ratio of 2. Each decoder block performs concatenation with features from a layer from the Encoder. The last layer in the decoder has an output size of 3: One for the background class, one for the cell body class, and one for the cell border class.

FIG. 4 illustrates a flowchart representation of an embodiment of a method of the invention for the training of a CNN for the classification of different cells categories.

The input is digital slides that contain cells of categories that are of interest to the user C1. First, the generic cell detection algorithm whose generation is described in the text for FIG. 2 is applied to the slides C2. The result is a list of the centers and contours of all the cells in the slide C3. Next, image patches are generated around each cell center as crops of, e.g. 32×32 C4. The user now annotates the image patches using a GUI to the cell categories of interest or background C5. This annotation is used to train a CNN for cell categories classification C6.

FIG. 5 illustrates a flowchart representation of an embodiment of a method of the invention for CNN for the classification of different cells categories.

The CNN architecture relies on the standard VGG16 architecture (other variants could be equivalently used), with 7 convolutional blocks (each containing a Convolutional layer, followed by a Rectified Linear Unit and a Batch Normalization unit), and then 3 fully connected layer, the first two of them followed by ReLU layers, and dropout layers. The last layer in the network outputs a score for the presence of the cell category in the input image patch (and equivalently a score for the lack of presence of a cell category in the input image patch).

FIG. 6 illustrates a flowchart representation of an embodiment of a method of the invention for a Cell Detection Algorithm Generation using Active Learning. In which digital slides are submitted D1, and the generic cell detector described in FIG. 2 is applied to provide the centers and contours of all the cells in the slides D2. Images patches around the centers of all the cells detected are created D3 and are submitted to annotation by the user D4. Once a sufficient number of image patches is available, according to a predefined threshold, a detection algorithm for specific cell categories is trained using this data D5. The deep learning neural network is then continuously trained in on the evolving dataset, as more annotations become available. During the training the number of annotated cells in each cell category constantly changes and this can affect the loss of the neural network. The weights have to be constantly updated as the database constantly changes during online learning. Therefore the loss is calculated as weighted cross entropy where the weights can be set using methods for class balancing, e.g. median frequency balancing. In addition, during the generation of the algorithm the architecture of the neural network, the number of layers, the number of parameters and the connection between layers may change. The user can test the generated cell categories detector generated on new slides and obtain visual and quantitative evaluation on the quality of the cell categories detector generated. The quantitative evaluation can be ROC, AUC or other standard metrics used in deep learning. When the user is satisfied with the quality of the generated cell detector, the Cell Detection generation process is completed. The user can resume the process with new data or modified annotations.

FIG. 7 illustrates a flowchart representation of an embodiment of a method of the invention for effective cell annotation using active learning. In which digital slides are submitted E1, and the generic cell detector described in FIG. 2 is applied to provide the centers and contours of all the cells in the slides E2. Images patches around the centers of all the cells detected are created E3 and are submitted to annotation by the user E4. Once a sufficient number of image patches is available, according to a predefined threshold, a detection algorithm for specific cell categories is trained using this data E5. The current cell categories classifier is then applied to image patches that were not annotated yet by the user E6. These image patches are then ranked according to active learning methodology E7. Each rank has a life time of a pre-defined parameter that can be configured. When the life time elapses the rank is then reset to a default value. The active learning algorithm chooses the best images to be annotated in order to make the annotation process most efficient. This ranking can be based on the acquisition function (should be minimized) or using an ensemble of models. If the ensemble of algorithms is in disagreement this means that the algorithm is less confident for that image patch. The ensemble of algorithms can be generated using Bayesian deep learning, where drop out is applied at test time. The next image patches to be annotated by the user are then selected according to this ranking E8, and the annotation procedure continues E4. This process is ongoing until the user is satisfied with the quality of the cell categories classifier generated.

FIG. 8 illustrates a flowchart representation of an embodiment of a method of the invention for effective cell annotation using data balancing strategy for active learning. In which digital slides are submitted F1, and the generic cell detector described in FIG. 2 is applied to provide the centers and contours of all the cells in the slides F2. Images patches around the centers of all the cells detected are created F3 and are submitted to annotation by the user F4. Once a sufficient number of image patches is available, according to a predefined threshold, a detection algorithm for specific cell categories is trained using this data F5. The current cell categories classifier is then applied to image patches that were not annotated yet by the user F6.

Unbalanced data is a common situation where the number of instances of one category is significantly smaller than the number of instances of another category. In order to obtain a robust network there should be enough examples of each category. We therefore add data balancing methodology for effective active learning F7. The data balancing methods can be one of the following: we rank the cells inversely proportional to their existence. We duplicate image patches that belong to the least frequent category. We can also add data balancing using weighting. The weight is inversely proportional to the proportion of least frequent category. Another approach is to add data balancing using the following weighting: Weight=E*A−B*(N-E)*Pminority where: *E=Entropy(class proportion)

• • A=Acquisition function as defined in Active Learning.
  • B=parameter
  • N=number of categories
  • Pminority=output of neural network that detects cell categories that gives the probability for the minority category.

Once we have enough examples of each category than we can move to the usual approach of active learning. The image patches are then ranked according to active learning methodology F8 as was described in the text for FIG. 7. The next image patches to be annotated by the user are then selected according to this ranking F9, and the annotation procedure continues F4. This process is ongoing until the user is satisfied with the quality of the cell categories classifier generated.

FIG. 9 illustrates a flowchart representation of an embodiment of a method of the invention for suggested auto annotation of cells. In which digital slides are submitted G1, and the generic cell detector described in FIG. 2 is applied to provide the centers and contours of all the cells in the slides G2. Images patches around the centers of all the cells detected are created G3 and are submitted to annotation by the user G4. Once a sufficient number of image patches is available, according to a predefined threshold, a detection algorithm for specific cell categories is trained using this data G5. The current cell categories classifier is then applied to image patches that were not annotated yet by the user G6. Once the cell categories detector is of sufficient quality, the activation of the auto annotation process can be triggered. This will be according to one of the following: 1) Time elapsed from beginning of training is higher than a threshold. 2) Classification results of the algorithm are close enough (according to some metric and threshold) to that of a human annotator. 3) Accuracy on a pre-defined validation set is good enough according to a pre-defined metric and threshold. 4) There is large enough number of annotations of cells of all categories of interest. Thus, the output of the current cell categories detector is used to annotate the image patches not annotated yet G7. The user can correct the annotations as suggested by the system G8 and these annotations are submitted for further training of the cell categories classifier G5. This process is ongoing until the user is satisfied with the quality of the cell categories classifier generated.

Claims

1. A method for generic cell detection in WSI: providing a digital WSI scanned by a digital scanner from a histological specimen-stained slide; detect the centers and contours of cells of all types in the WSI using image processing and deep learning algorithms.

2. A method to create a detector that can classify between various types of cells of different categories using neural network algorithms. In order to create the neural network algorithm image crops are selected from the WSI that contain cells detected using the method described in claim 1. The crop size is set to a constant value or is adjusted to the size of cells in the image crop. An interactive annotation scheme of the sampled image patches with a GUI (graphical user interface) application can be used by human annotators. Each time a single image crop is presented to the annotator, and using a keyboard press or mouse click, or touch screen tap, the annotator chooses one of the possible categories, each containing a specific type of cell or background.

3. The method of claim 2 where online learning methodology is used, thus the classification neural network is trained in parallel to the annotation process. The deep learning neural network is continuously trained on an evolving dataset. During the training the number of annotated cells in each cell category constantly changes and this can affect the loss function of the neural network. Thus, the weights of each cell category have to be constantly updated as the database constantly changes during online learning. Therefore the loss function is calculated as weighted cross entropy where the weights can be set using methods for class balancing, e.g. median frequency balancing. In addition, during the generation of the algorithm the architecture of the neural network, the number of layers, the number of parameters and the connection between layers may change.

4. A method for selecting the next image patches to be annotated using active learning framework. An active learning algorithm is applied on the cells yet to be annotated. The active learning algorithm ranks each cell. Each rank has a life time of a pre-defined parameter that can be configured. When the life time elapses the rank is then reset to a default value. The active learning algorithm chooses the best images to be annotated in order to make the annotation process most efficient. This ranking can be based on the acquisition function (should be minimized) or using an ensemble of models. If the ensemble of algorithms is in disagreement this means that the algorithm is less confident for that image patch. The ensemble of algorithms can be generated using Bayesian deep learning, where drop out is applied at test time.

5. The method of claim 4 where we add data balancing using resampling. Unbalanced data is a common situation where the number of instances of one category is significantly smaller than the number of instances of another category. In order to obtain a robust network there should be enough examples of each category. We therefore add data balancing methodology for effective active learning. We rank the cells inversely proportional to their existence. Once we have enough examples of each category we can move to the usual approach of active learning. We duplicate image patches that belong to the least frequent category.

6. The method of claim 4 where we add data balancing using weighting. The weight is inversely proportional to the proportion of the least frequent category.

7. The method of claim 4 where we add data balancing using the following weighting: Weight=E*A−B*(N−E)*Pminority where: E=Entropy(class proportion)A=Acquisition function as defined in Active Learning.B=parameterN=number of categoriesPminority=output of neural network of claim 2 that gives the probability for the minority category.

8. A method for suggested auto annotation of the image crops that contain different kind of cells. The activation of the auto annotation process can be triggered by one of the following: 1) Time elapsed from beginning of training is higher than a threshold. 2) Classification results of the algorithm are close enough (according to some metric and threshold) to that of a human annotator. 3) Accuracy on a pre-defined validation set is good enough according to a pre-defined metric and threshold. 4) There is large enough number of annotations of cells of all categories of interest.

9. The method of claim 8 where we add noise to the auto annotation process so that some of the images will be selected at random so that we do not fall in dead ends.

10. Cell Detection Studio: an AI based system that provides pathologists with a semi-automatic tool to create new algorithms aiming to find cell of specific categories in WSI digitally scanned from histological specimen; a computer running a dedicated WSI viewer that contains data upload and save utilities, zoom and pan and the ability to quickly navigate to a region of interest. The WSI viewer also offers computer vision, machine learning and deep learning algorithms. Specifically, the system provides the methods detailed in claims 1-9 that allows detection of the presence of specific types of cells.

11. The method of claim 10, wherein the system is adapted to the transfer, storage and retrieval of the associated images, and for the generation of reports.

12. The method of claim 10 for Quality Assurance (QA) as part of the training process. The user can annotate cells of specific category in a region of interest and apply the algorithm developed on that region of interest. Then the system generates statistical report on the accuracy level achieved. Accuracy measures to be used can be but are not limited to: FA, AUC, confusion matrix, false positives and false negatives.

13. The method of claim 10 with the addition of calculating various attributes related to the specific cell categories for which the detection algorithm was created. These features are based on the location and contours of the above mentioned cells: number, density, area, location and perimeter.