Patent Grant
10943346
When reviewing digital pathology tissue specimens, multiple slides may be created from thin, sequential slices of tissue. These slices may then be prepared with various stains and digitized to generate a Whole Slide Image (WSI). Review of multiple WSIs is challenging because of the lack of homogeneity across the images.
In embodiments, to facilitate review, WSIs are aligned with a multi-resolution registration algorithm, normalized for improved processing, annotated by an expert user, and divided into image patches. The image patches may be used to train a Machine Learning (ML) algorithm to identify features useful for detection and classification of Regions of Interest (ROIs) in images. The trained ML model may be applied to other images to detect and classify ROIs in the other images, which can aid in navigating the WSIs. When the resulting ROIs are presented to the user, the user may easily navigate and provide feedback through a display layer.
In one example embodiment, a system is provided for analyzing WSIs of tissue specimens. The system may include a computer processing system having at least one processor communicatively coupled to memory that is configured to analyze WSI of the tissue specimens.
For WSI in a set of slides with a plurality of stains, a stain type for a WSI may be specified by the system. The stain type may be specified by the system through one of: metadata from another system, manual labeling by a pathologist or other domain expert, or an automatic stain detector. The WSI may be preprocessed to form an intermediate form of the data, which separates the foreground tissue from the background and normalizes the WSI data based on parameters of a ML model, where the parameters include the possible stains as well as resolution or other parameters.
The WSI may be presented for annotation, where the annotation may involve applying non-default classification labels to regions of the WSI data via a user interface by a pathologist or other domain expert. Image patches may be created from the annotated regions, each image patch corresponding to a single class. The parameter appropriate ML model that dynamically generates features useful for classification may be trained. The trained ML model classifier may be applied to unannotated WSI data to produce a set of classifications for the stain and patch parameters.
The preprocessed WSIs from the stain sets may be registered via a multi-resolution registration algorithm. The multi-resolution registration algorithm may include (1) an application of a coarse registration algorithm to produce a general Affine Transform Matrix and (2) iterative registration on successively smaller subsections to produce a sparse hierarchical multi-resolution ATM pyramid, which is then processed to generate a non-sparse Field of Affine Transform Matrices (fATMs) such that corresponding regions between images are aligned to the highest degree possible based on tissue and imaging variations.
The produced set of non-default classifications from multiple stain sets may be aggregated, such that each set of classifications may be generated by applying a stain-specific, trained ML model to the normalized WSI data, with the aggregation achieved through the translation of the non-default classifications using the fATMs generated for the individual WSIs;
The aggregated non-default classifications from each of the multiple WSIs in the stain set may be correlated. The individual classifications may be correlated by enhancing true classifications and removing false classifications, resulting in one or more non-default classified regions for a given stain set and metadata for the one or more non-default classified regions of the given stain set.
In an example embodiment, a system may be provided for presenting a stain set of registered WSIs and annotations. The computer processing system may include at least one processor communicatively coupled to memory. The processor may be configured to present each image of the stain set in a separate viewing panel of a plurality of viewing panels. The processor may be configured to display a set of annotated regions of an image and corresponding metadata in an organized tabular display, along with the plurality of viewing panels. The processor may be configured to enable capability for a user to: (i) click on one of the set of annotated regions in the organized tabular display and (ii) navigate all of the plurality of viewing panels to the same location by making use of previously calculated registration information of the images. The processor may be configured to enable capability for a user to: (i) apply navigation events within one or more of the plurality of viewing panels, including dragging, zooming, and panning, and (ii) subsequently move all of the registered viewing panels to the same location by making use of the previously calculated registration information of the image;
The processor may be configured to enable capability for a user to disable navigation of the plurality of viewing panels to the same location. The processor may be configured to enable capability for a user to provide additional annotated regions, at least one of the additional annotated regions corresponding to labeling of image regions improperly annotated, including missed annotations or misclassified annotations.
The foregoing will be apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings in which like-reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments.
A description of example embodiments follows.
The teachings of all patents, published applications and references cited herein are incorporated by reference in their entirety.
For many medical imaging applications, but especially in the field of pathology, current state of the art requires the viewing of multiple data samples for a particular specimen. The data samples may be viewed in either the analog or digital domain. The samples to be viewed are created by producing multiple glass slides from a single specimen. These slides are thin, sequential slices of tissue from the single specimen. As the slices are typically generated sequentially, the morphology and structures contained within a series of slides are similar but not identical; the pieces of tissue and the structures within each slide are irregular objects, like a cross-section of a tree trunk, and the structures and the overall tissue shape change from one slide to the next. The slices are then prepared using different stains that allow for better viewing of various types of structures within the tissue.
To view the samples in the analog domain, the pathologist views the slides using different light sources and filters (visible light, fluorescent, polarization, and such) under a microscope in order to provide a review.
To view the samples in the digital domain, the pathologist views the slides using either a microscope-mounted camera or using a Whole Slide Imaging device (digitizer) to produce a Whole Slide Image (WSI). In both instances, the captured images are displayed on a computer monitor.
In both the analog and digital domain, the current state of the art requires the pathologist to view each sample (slide or WSI) in a serial manner. When the pathologist sees a region of interest on one slide or image, the pathologist must manually locate the corresponding region on other slides to see the corresponding information. By extension, the use of digital Whole Slide Images (WSIs) for tasks beyond viewing by a human, such as when applying Machine Learning (ML) or utilizing Computer Vision (CV) algorithms, is challenging because of the irregularities and lack of direct correlation between images of different slides.
Embodiments of the present invention herein describe a workflow (method or process) for effectively reviewing specimens in digital pathology. Additionally, the present invention provides additional utility, for both the current WSIs and similar future WSIs, by allowing data to be used for training models, or for inferencing the current WSIs against a trained model, to assist in pre-screening, case review or quality reviews, utilizing more information from the WSI than is currently available or utilized in the analog domain. In the workflow, a set of images for a specimen containing progressively different structures (based on both sample variation and on the stains and light sources) may be preprocessed (color and resolution normalized), evaluated for desired features, aligned (registered) across stains, post-processed (correlated), and displayed to a user. Preprocessing may include color normalization to account for stain variations from different organizations, scanners, and technicians. Evaluation (also referred to as inferencing) may be carried out via one or more machine learning (ML) algorithms. Cross-stain image alignment may be accomplished via a multi-resolution registration algorithm. Post-processing may include boosting of strong detections and removal of weak ones using ML outputs from multiple stain sections, to produce a cleaner, more sensible display to the user. Additional embodiments describe the process by which an expert can create the ground truth for training an ML algorithm. Additionally, the expert may provide feedback on the results from an initial evaluation in order to retrain or augment an ML algorithm to improve accuracy.
Embodiments of the Machine Learning algorithms which are implemented within the embodiment of the current invention include Deep Learning or Convolutional Neural Networks (CNNs), Region-based CNNs (R-CNN), Support Vector Machines (SVMs), AutoEncoders (AEs) or Shallow Neural Networks (SNNs).
System Implementation
In the Stain Tagging stage (110), the system/method associates metadata containing the stain type with each WSI. This process can be done via user input, via externally supplied metadata or using an Automatic Stain Detection algorithm to determine the stain types of the WSI data. All 3 methods result in tagging (i.e., labelling) the WSI data with the identified stain types, producing Tagged Images (119). The system/method inputs the Tagged Images to both the User Interface Processing stage (180) for use by an expert user, and the Preprocessing Stage (120).
The Preprocessing stage (120 in
The Registration stage (160 in
The Patch Generation stage (130 in
The Labeled Training Data (142) is used by the system/method in the CNN Training stage (140 of
The Labeled Validation Data (152) is used by the system/method in the CNN Validation stage (150 of
The system/method performs the Evaluation stage (170 of
The User Interface Processing stage (180 on
Additionally, the User Interface Processing stage (180 on
Stain Tagging
Embodiments of the present invention perform stain detection on WSI data, such as in Stage 110 of
In some embodiments, the stain type is specified by either externally supplied metadata or is assessed by an expert.
In another embodiment, if the stain type is not specified, then Automatic Stain Detection is used to determine the most likely stain type from the overall WSI image. In one embodiment of automatic stain detection, an SVM (Support Vector Machine) ML model is trained on color histogram data from a set of WSIs with known stain types, identified by an expert. The 2-dimensional color histogram used for model training characterizes the hue and saturation information for an entire WSI excluding the background areas. This process may require converting the WSI to both the desired resolution and the correct color space, such as the hue-saturation-value (HSV) color space, red-green-blue (RGB) color space, or a luminance-chrominance (e.g., YUV) color space. The pre-trained SVM then uses the histogram data to classify new WSIs with unknown staining types.
Preprocessing
Embodiments of the present invention perform preprocessing on images, such as in Step 120 of
Normalization can take on many embodiments, but typically it may include color normalization (defining the optimal color characteristics for a given stain) and resolutional normalization (correcting discrepancies in Microns Per Pixel [mpp] between images).
Color normalization involves segmenting the WSI into foreground (tissue) and background, and applying a transform to make the foreground more closely conform to target stain colors and to whiten the background. The separation of foreground and background can be done in several ways. In one embodiment, such an algorithm detects the off-white or off-black background via value-based segmentation, by converting the image to grayscale and replacing large areas that are below, or above, a certain threshold (i.e., areas that are almost white or almost black) with an absolute white background. This step serves to remove any background artifacts. This brightness thresholding does not always work with darker and busy backgrounds. In another embodiment, segmentation is saturation-based. Foreground segmentation is based on the assumption that backgrounds are always less saturated in color than the tissue. Hence, the WSI is converted to hue-saturation-value color space, and then the saturation threshold is computed by finding a maximum of the second derivative of the saturation histogram. Then, the WSI is processed by comparing each pixel's saturation value to the threshold, creating the background/foreground mask.
Once the background is standardized, color deconvolution, or the classification and separation of the different stain components in the image by absorbance values, can be performed. In one embodiment, the classification of the components may be implemented using an SVM, k-nearest neighbors (KNN) algorithm, or some other similar ML classifier. In another embodiment, classification can be performed using matrix transformations on the absorbance values [Ruifrok & Johnston, “Quantification of histochemical staining by color deconvolution”, Anal Quant Cytol Histol, 23: 291-299, 2001, http://ww.aqch.com/toc/auto_abstract.php?id=15581]. After the stain components of a WSI are separated, their color values in the given colorspace (e.g., RGB, HSV, etc.) are adjusted to match those of the deconvoluted target color components. The target colors may be determined by a professional, or by averaging the color components from training WSIs of known stain type.
Further embodiments may include at least one of spatial information and optical density information, in addition to chromatic information, to deconvolute the color components, in order to improve the accuracy of the normalization.
Resolution normalization is the processing of all input WSIs to identify or generate a resolution in microns per pixel (mpp) that is within a range of acceptable values based on what is required by the CNN. Some structures and features utilized by the CNN for its processing and classification are most visible at specific resolutions, so the normalization step ensures that all WSIs provide that data to the best degree possible.
In one embodiment, if a WSI contains multiple resolutions, and one of those resolutions falls within the range acceptable to the CNN, then that resolution is utilized for processing. If the acceptable range is of higher resolution than the maximum capture resolution, the highest capture resolution is scaled and smoothed using a scaling algorithm to produce the input data for the CNN. In a preferred embodiment, the scaling algorithm is bilinear interpolation. If the acceptable range falls between resolutions of the WSI, or is lower than the capture resolution, the next highest resolution in excess of the target resolution is selected, and the data is scaled and smoothed to produce the input data for the CNN.
Annotation
Embodiments of the present invention perform annotation on images, such as in Step 180 of
WSI-based annotations apply an annotation (i.e., label) to the entire image (minus the background, if applicable). Region-based annotations provide for the specification (by whichever process is used) of both one or more defined regions (such as an ellipse or polygon) of the WSI and the appropriate annotation for that region. In many embodiments, the assignment of a classification is a binary representation (a particular region either does or does not demonstrate a particular characteristic). In further embodiments, each region may also be assigned a class from a plurality of choices (such as normal, grade 1, grade 2, etc.).
In one example embodiment, an expert user may use annotations to define a region (e.g., an ellipse, a rectangular region, or a free-hand region) by highlighting the area that entirely contains a given feature for training a classifier. The annotated areas are then assigned a class based on the feature identified. Any subsections of the WSI containing tissue (not the background) which does not contain an expertly annotated classification are assigned a default classification, and can be used as such by the system during training.
In another embodiment, Computer Vision (CV) algorithms may be employed to facilitate the segmentation of the image (such as automated cell or nucleus detection, deconvolution, or color filtering). Once segmented, the expert can then more easily use one of the previously mentioned tools to mark a region as belonging to a specific class, and the expert may manually adjust the annotation to exclude those pieces not belonging to the class that fall within the region, or include those pieces belonging to the class that fall just outside of the region.
In another embodiment, the expert's annotations can be automatically classified into stricter categories such as “middle of the tissue” vs. “tissue borderline,” which would allow for future training fine-tuning. This might be done by analyzing the color histogram of the annotated part of the WSI. For instance, finding significant background color peaks on the histogram would mean that annotation belongs to the “tissue borderline” category.
Creating an ML Model
Embodiments of the present invention create an ML model through training, such as in Step 140 of
In an example embodiment, to further enrich and expand the set of training patches, the training patches can also be modified by rotation, flipping, scaling, adding noise, blurring, etc. to increase the overall size and variety of the dataset used for ML training. The set of annotated patches are then provided as training data to the ML algorithm, to produce the model. In another embodiment, the annotated images generated above could be divided into groups by certain criteria such as “middle of the tissue” vs “tissue borderline,” with each group used to train a separate model.
In one embodiment, the model may be generated by training an ML algorithm, such as a Convolutional Neural Network (CNN) that operates on the raw image data. Other embodiments may include a preprocessing step before employing an ML algorithm, such as a Support Vector Machine (SVM) or a “shallow” Neural Network. In embodiments, preprocessing steps may include Image Processing (edge detection, sharpening, blurring, color manipulation/deconvolutions), Computer Vision, or Feature Detection (SURF, SIFT, BRISK, FREAK, ORB, etc.) algorithms.
In an example embodiment, the CNN may consist of two or more convolutional layers, pooling layers to reduce the dimensionality of the data and prevent overfitting, nonlinearity layers (e.g., Rectified Linear Unit [ReLU] layers) to increase the nonlinearity properties of the network, dropout layers to prevent overfitting, one or more fully-connected layers, and an output classification layer. In an embodiment, the classification layer is a softmax classifier with cross-entropy loss function. In one embodiment, the CNN's optimization algorithm for training is stochastic gradient descent with momentum (SGDM). In an alternate embodiment, the CNN's optimization algorithm is Adaptive Moment Estimation (ADAM).
CNNs differ from basic backpropagation neural networks (also known as multilayer perceptrons [MLPs]) in that CNNs are “deep” neural networks that contain more than one hidden layer (layers between the input and output layers). As such, CNNs dynamically generate features for classification (i.e., “on the fly”) while training. This is distinguished from feature-based classifiers and classifiers that use basic backpropagation neural networks, which require a separate feature extraction step in which features useful for classification are explicitly calculated.
In an embodiment, the CNN may be trained from scratch using the WSI-based image patches generated as described above. In another embodiment, a CNN, which is pretrained on similar or dissimilar image data, may be imported via transfer learning, in which most of the layers of the pretrained CNN are preserved and only the output classification layer is re-trained on the WSI image patch data.
In an embodiment, annotations (i.e., classifications or labels) are mapped to a set of desired output labels. During this process, the different annotations labels (either a pre-established set of human readable strings or numerical values) are mapped to the desired output labels. In one example, each different input label is mapped to an output label. In another example, all region-based annotations are mapped to a ‘positive detection’ and unannotated areas are deemed ‘negative detections’. In a further example, a subset of input labels may be mapped to class A, another set to class B, and unannotated areas to class C.
In an example embodiment, more than one model may be trained, corresponding to particular capture characteristics (stain, lighting, imaging conditions, resolution) or specimen characteristics (e.g., shape, texture, etc.) or mapped output labels. In a further embodiment, preprocessing steps may be used to divide the training data by such characteristics into multiple sets of training data, such that separate models may be trained on different sets of training data, resulting in models that are “tuned” to and can more accurately classify image patches with those particular characteristics.
In one embodiment with multiple models, all of the models may be used simultaneously to inference a given WSI or area, utilizing a post-processing step, such as selecting the maximum confidence score from each model, to determine a final label based on the results from the individual models.
In a further embodiment with multiple models, a preprocessing step, leveraging Machine Learning models, Computer Vision processing or Feature Detectors, may be used to select the appropriate model from the multiple models for inferencing a particular region or for the WSI as a whole.
Once trained, the one or more ML models can be validated by subsequently testing the algorithm against samples of the labeled data to determine its validation error or training accuracy.
Evaluation of New Samples Against the Model (Classification)
Embodiments of the present invention perform evaluation of new samples, such as in Step 170 of
In one embodiment, once the classification model has been computed on the training data for the CNN, it may be applied to “test” data to classify the test data. Similar to the training process, normalized and otherwise preconditioned image patches may be supplied to the model. The pipeline selected for the classifier accepts patches of a predetermined size as input and results in a classification and scoring of each patch based on the classes identified during training (either binary detection [default or non-default classifications] or n-class classifier [default classifications or non-default classifications with sub-classes]).
In another embodiment, the test data may be previously annotated and the results of the classification process may be compared to the test data annotations to determine how accurate the model is. In an embodiment, binary classification (i.e., detection) problems, in which a phenomenon (feature) may be declared present (positive or non-default classification) or absent (negative or default classification), may be evaluated via standard metrics such as sensitivity (correct positive percentage) or specificity (correct negative percentage) or F1 score.
In a further embodiment, incorrectly classified image patches may be collected and then fed back into the system to create a more accurate model via incremental training. The additional, incorrectly classified image patches may be added to the original training set to form a larger, combined training set. In one example embodiment, the CNN is re-trained from scratch on the combined training set. In another example embodiment, most of the layers of the original CNN are preserved, and the output classification layer is incrementally trained on the combined training set, via transfer learning. In an alternate embodiment, transfer learning is applied to the original CNN by retraining the output classification layer only on the “new” training data (the incorrectly classified image patches).
Registration
Embodiments of the present invention perform registration on the preprocessed WSIs, such as in Step 160 in
At minimum, a single coarse ATM describing the alignment between whole images can be calculated. This is particularly useful to correlate highly irregular tissue in compared WSIs. A coarse ATM principally aligns the tissue to account for offsets (horizontal and vertical placement of the tissue on the slide), rotation (clockwise or counterclockwise rotation of the tissue, +/− 180 degrees), and inversion (where one specimen was flipped as it was placed on the slide).
The generation of an ATM can be achieved via identifying at least 3 matching keypoint pairs and then solving a linear system of equations using identified keypoint pairs. Keypoints are distinct positions in an image, usually relying on the visible features in the WSI such as tissue corners or blots. In each keypoint pair, one keypoint belongs to the first WSI and the second keypoint refers to the matching location on the second WSI. For example, the keypoint on a cellular feature in the first stained image (such as the edge of the tissue, or a cluster of red blood cells) should correspond to the same cellular feature in the second image. For the real world WSIs, there are usually more than three matching keypoint pairs identified, hence special mathematical methods are used to find the ATM that best satisfies all keypoint matches.
Normally, these keypoints are identified using a feature detection algorithm. In one embodiment, the ORB (Oriented FAST and rotated BRIEF) algorithm is used to detect keypoints, which are typically corners or edges in the image. The keypoints may be matched using a matching algorithm with an evaluation criterion. In one embodiment, this algorithm is the Brute-Force Matcher and the evaluation distance is the Hamming Distance, which is a measure of the differences between two strings. Next, the keypoint matches are analyzed for consistency, then unfit or poorly correlated matches are removed. The ATM candidates for the remaining matches may be found, further filtered for consistency, and then averaged to obtain a single, refined ATM for the region.
The process of averaging and finding the most correlated matches is done via an averaging algorithm. In one embodiment, the averaging algorithm is the Mean Shift clustering algorithm applied iteratively with dynamically configured parameters to achieve correlation goals. This clustering algorithm is executed several times with varying cluster size parameters, in order to find a cluster of correlated keypoints which is big enough to characterize most of the keypoint matches and at the same time compact enough to eliminate erroneous matching keypoints.
Using the information from the coarse ATM about the general translation and rotation of the WSIs, progressively finer ATMs can be calculated for smaller subsections of the images to improve precision, such that the WSI is divided by a “grid” of areas, each with a separate ATM. Fine ATMs provide better alignment precision than the coarse ATM, but they are only valid within a smaller subsection of the whole image. Finer ATMs are found because the images being registered are of biological origin and the physical tissue involved is cut, stained, and set by technicians; thus, there can be a variety of discrepancies among the images that would be difficult for an entire coarse WSI ATM to capture while maintaining good precision. There may be dislocations, folds, and other processing artifacts present on the images. Additionally, since the tissue sections are 2-dimensional slices of a 3-dimensional tissue, there can be differences in size, orientation, and presence of some of the features. With the small tissue subsections compared between the WSIs, the likelihood of the good match between them increases, hence enhancing the match precision.
In a preferred embodiment, a multi-resolution registration algorithm is used to provide fine ATMs at different stages. The image can be recursively divided into additional subsections of smaller size, generating progressively finer ATMs for each subsection based on the previously calculated ATMs, until no additional precision can be calculated, or the largest amount of subdivision is achieved. Once determined, these local ATMs comprise a progressively more precise, sparse pyramid of ATM levels, where higher precision fine ATMs represent smaller subsections of the images.
For some areas of the WSI, it is possible that they do not contain any detectable features (e.g., white background, missing portions of tissue). For these areas, the keypoints cannot be found, hence the ATM cannot be calculated. In one embodiment, for such feature-less subsections of the WSI, the local ATMs are computed by approximation from the previously computed nearby ATMs. In another embodiment, the ATM for these areas is taken from the lower precision ATM calculated for the bigger area, or from the entire coarse WSI ATM.
The combination of the most precise ATM for any region into a single structure creates a field of ATMs (fATMs).
In one embodiment for creating fATMs, a coarse ATM is initially generated using a low-resolution image that represents the whole image but is subsampled by many factors such as in step 162
In another embodiment for creating fATMs, the coarser ATMs are calculated via a keypoint-based method as a first step to provide basic alignment of the two WSIs, and then a parametric, intensity-based registration algorithm is applied to bolster the coarse registration's precision. This algorithm compares the pixels of the two images instead of finding keypoints, matching up comparable areas based on a defined similarity metric and assessing the differences in the images in an iterative stochastic optimization process. This algorithm requires initial placement of the compared areas to be fairly good, hence the requirement of the keypoint WSI registration as a first step.
The keypoints-based registration and the intensity-based registration both may repeat on progressively smaller areas of WSIs, with higher resolution, as needed to obtain the optimal registration. Measuring the registration quality can be done in several ways. In one embodiment of the evaluation criteria, keypoints-based registration may be evaluated by the standard deviation and number of keypoints detected. For intensity-based registration, the second image is warped to fit the first image in the registration, producing a new result image. This result image can be overlayed on the first image to assess the quality of the registration, where greater non-overlapping area suggests a lower quality registration.
Correlation (Post-Processing)
Embodiments of the present invention perform correlation on classification results, such as in Step 172 of
The output of the Correlation step is a set of ROIs, each of which may have a classification and an associated confidence metric (or score).
In one embodiment of such correlation, classification and scoring results may be obtained separately from several (e.g., four) stains of tissue WSIs. The per-staining detected feature results may be thresholded by a confidence score and combined together using inter-WSI registration in order to align the feature locations. The results may be further filtered out by removing spatial locations with positive detections on fewer than the required number of stain WSIs. The remaining positive classification data and scores may be merged into a score-heatmap, from which local combined score maxima are found, which in turn become the centers of newly found ROIs. Classification data may be further used after additional filtering to aggregate around the found local maxima and create clusters that would define the sizes of the newly found ROIs.
Using the combined classification results from possibly multiple stained WSIs and/or classification models with enhanced registration of WSI pairs utilized to correlate the localities of the positively classified areas results in significant improvement of the detection characteristics compared to the case of using a single image.
Display of Results & Feedback Loop
Embodiments of the present invention display results of the above steps to the user, such as in Step 180 of
In an example embodiment, the result thumbnails, when clicked, may navigate the corresponding viewport to the location indicated by the selected thumbnail. In the case of registered Multi-Sample WSIs, the interface may also navigate each viewport to the same or similar location on the corresponding stained slide based on the best available ATM generated by WSI registration. Clicking on any of the slide viewers, when panned or zoomed, triggers all other slide viewers to navigate to the same position by applying the best available ATM generated by registration. This ensures the same section of tissue across all WSIs is simultaneously viewable by the user.
In the preferred embodiment, if a user disagrees with the classification of a result generated by a model, the user can correct the classification by clicking a button next to the result thumbnail (e.g., false positive, incorrect classification). The corrected classification and result sub-image may then be used as future training data to improve the model. A user may also additionally annotate and classify regions of the slide which belong to a classification of the model that were not identified by the model. The corrected results and added classifications can then be used by the system to improve the model via retraining.
Digital Processing Environment
Client computer(s)/devices 50 can also be linked through communications network 70 (e.g., via interface 107) to other computing devices, including other client devices/processes 50 and server computer(s) 60. Communications network 70 can be part of a remote access network, a global network (e.g., the Internet), cloud computing servers or service, a worldwide collection of computers, Local area or Wide area networks, and gateways that currently use respective protocols (TCP/IP, Bluetooth, etc.) to communicate with one another. Other electronic device/computer network architectures are suitable.
Client computers/devices 50 may include features to input WSI data (e.g., set of images for a specimen containing progressively different structures based on both sample variation and on the stains and light sources). The Client computer/devices 50 may also present a user interface tool consisting of a series of viewports to each display a view of differently stained registered WSI and a sidebar that may provide various tools and present a series of thumbnails representing detected and classified ROIs. Server computers 60 may be a user computer device, which may receive the input WSI data from the client computer/devices 50 and perform stain detection and image tagging on the WSI data. The server computers 60 may then perform on the tagged images preprocessing (color and resolution normalized), CNN training, evaluation for desired features, alignment (registered) across stains, post-processing (correlated), and displayed to a user via the user interface tool, as shown in
Attached to system bus 79 is I/O device interface 82 for connecting various input and output devices (e.g., keyboard, mouse, wheels, buttons, touch screens, displays, printers, speakers, voice controls, etc.) to the computer 50, 60. Network interface 86 allows the computer to connect to various other devices attached to a network (e.g., network 70 of
In an example mobile implementation, a mobile agent implementation of the invention may be provided. A client-server environment can be used to enable mobile configuration of the capturing of the navigation of slide images. It can use, for example, the XMPP protocol to tether WSI data. The server 60 can then issue commands via the mobile phone on request. The mobile user interface framework to access certain components of the WSI processing system may be based on XHP, Javelin and WURFL. In another example mobile implementation for OS X, iOS, and Android operating systems and their respective APIs, Cocoa and Cocoa Touch may be used to implement the client side components 115 using Objective-C or any other high-level programming language that adds Smalltalk-style messaging to the C programming language.
Disk storage 95 provides non-volatile storage for computer software instructions 92 and data 94 used to implement an embodiment of the slide navigation system. The system may include disk storage accessible to the server computer 60. The server computer (e.g., user computing device) or client computer (e.g., sensors) may store information, such as images and models, from the reviewing of images. Central processor unit 84 is also attached to system bus 79 and provides for the execution of computer instructions.
In one embodiment, the processor routines 92 and data 94 are a computer program product (generally referenced 92), including a computer readable medium (e.g., a removable storage medium such as one or more DVD-ROM's, CD-ROM's, diskettes, tapes, etc.) that provides at least a portion of the software instructions for the WSI processing system. Executing instances of respective software components of the WSI processing system, may be implemented as computer program products 92, and can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable, communication and/or wireless connection, via for example, a browser SSL session or through an app (whether executed from a mobile or other computing device). In other embodiments, the invention programs are a computer program propagated signal product 107 embodied on a propagated signal on a propagation medium (e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network such as the Internet, or other network(s)). Such carrier medium or signals provide at least a portion of the software instructions for the routines/program 92 of the slide navigation system.
In alternate embodiments, the propagated signal is an analog carrier wave or digital signal carried on the propagated medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a telecommunications network, or other network. In one embodiment, the propagated signal is a signal that is transmitted over the propagation medium over a period of time, such as the instructions for a software application sent in packets over a network over a period of milliseconds, seconds, minutes, or longer. In another embodiment, the computer readable medium of computer program product 92 is a propagation medium that the computer system 50 may receive and read, such as by receiving the propagation medium and identifying a propagated signal embodied in the propagation medium, as described above for computer program propagated signal product.
Generally speaking, the term “carrier medium” or transient carrier encompasses the foregoing transient signals, propagated signals, propagated medium, storage medium and the like.
In other embodiments, the program product 92 may be implemented as a so-called Software as a Service (SaaS), or other installation or communication supporting end-users.
While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments encompassed by the appended claims.
It should be noted that although the figures described herein illustrate example data/execution paths and components, one skilled in the art would understand that the operation, arrangement, and flow of data to/from those respective components can vary depending on the implementation and type of medical image data being processed. Therefore, any arrangement of data modules/data paths can be used.
While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.
In one example implementation, a blockchain system may be used to facilitate recording the registrations of the preprocessed WSIs from the stain sets of the multi-resolution registration algorithm. Annotations involves applying non-default classification labels to regions of the WSI data may, for example, further be recorded in a data blocks of a blockchain implementation. In this way, the registrations and annotations of the preprocessed WSIs may be recorded in a blockchain distributed ledger, which can facilitate maintaining the data integrity. Further, smart contracts can be used to control access to each WSI, and facilitate access among potentially disparate users.
This application claims the benefit of U.S. Provisional Application No. 62/674,368, filed on May 21, 2018. The entire teachings of the above application(s) are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5561556 | Weissman | Oct 1996 | A |
| 5733721 | Hemstreet, III et al. | Mar 1998 | A |
| 5741648 | Hemstreet, III et al. | Apr 1998 | A |
| 5956418 | Aiger | Sep 1999 | A |
| 6011809 | Tosaka | Jan 2000 | A |
| 6049421 | Raz et al. | Apr 2000 | A |
| 6466352 | Shahar et al. | Oct 2002 | B1 |
| 6606413 | Zeineh | Aug 2003 | B1 |
| 6650357 | Richardson | Nov 2003 | B1 |
| 6798571 | Wetzel et al. | Sep 2004 | B2 |
| 6905300 | Russum | Jun 2005 | B1 |
| 6961080 | Richardson | Nov 2005 | B2 |
| 7109464 | Cartlidge et al. | Sep 2006 | B2 |
| 7132636 | Cartlidge et al. | Nov 2006 | B1 |
| 7151246 | Fein et al. | Dec 2006 | B2 |
| 7155049 | Wetzel et al. | Dec 2006 | B2 |
| 7224839 | Zeineh | May 2007 | B2 |
| 7248716 | Fcin et al. | Jul 2007 | B2 |
| 7338168 | Cartlidge et al. | Mar 2008 | B2 |
| 7346200 | Tsipouras et al. | Mar 2008 | B1 |
| 7417213 | Krief et al. | Aug 2008 | B2 |
| 7463761 | Eichhorn et al. | Dec 2008 | B2 |
| 7518652 | Olson et al. | Apr 2009 | B2 |
| 7522757 | Tsipouras et al. | Apr 2009 | B2 |
| 7605356 | Krief et al. | Oct 2009 | B2 |
| 7638748 | Krief et al. | Dec 2009 | B2 |
| 7640112 | Tsipouras et al. | Dec 2009 | B2 |
| 7646495 | Olsen et al. | Jan 2010 | B2 |
| 7689024 | Eichhorn et al. | Mar 2010 | B2 |
| 7692131 | Fein et al. | Apr 2010 | B2 |
| 7756357 | Yoneyama | Jul 2010 | B2 |
| 7772535 | Krief et al. | Aug 2010 | B2 |
| 7835869 | Tsipouras et al. | Nov 2010 | B2 |
| 7863552 | Cartlidge et al. | Jan 2011 | B2 |
| 7867126 | Nakajima et al. | Jan 2011 | B2 |
| 7876948 | Wetzel et al. | Jan 2011 | B2 |
| 7901887 | Tafas et al. | Mar 2011 | B2 |
| 7945391 | Tsipouras et al. | May 2011 | B2 |
| 8170698 | Gusack | May 2012 | B1 |
| 8271251 | Schwartz et al. | Sep 2012 | B2 |
| 8574590 | Doranz et al. | Nov 2013 | B2 |
| 8774518 | Cosatto et al. | Jul 2014 | B2 |
| 8879813 | Solanki | Nov 2014 | B1 |
| 8891851 | Spaulding | Nov 2014 | B2 |
| 8896620 | Orrock | Nov 2014 | B2 |
| 8897537 | Cosatto et al. | Nov 2014 | B2 |
| 8934699 | Yamane et al. | Jan 2015 | B2 |
| 8934718 | Cosatto et al. | Jan 2015 | B2 |
| 9036255 | Loncy et al. | May 2015 | B2 |
| 9097909 | Halushka | Aug 2015 | B2 |
| 9134524 | Yamamoto et al. | Sep 2015 | B2 |
| 9213027 | Doranz et al. | Dec 2015 | B2 |
| 9217694 | Sieckmann et al. | Dec 2015 | B2 |
| 9224106 | Cosatto et al. | Dec 2015 | B2 |
| 9332190 | Murakami | May 2016 | B2 |
| 9335534 | Schaudraff | May 2016 | B2 |
| 9430829 | Madabhushi | Aug 2016 | B2 |
| 9547898 | Hall et al. | Jan 2017 | B2 |
| 9575301 | Loney et al. | Feb 2017 | B2 |
| 9581800 | Corwin et al. | Feb 2017 | B2 |
| 9609196 | Yamakawa et al. | Mar 2017 | B2 |
| 9712732 | Kancko et al. | Jul 2017 | B2 |
| 9759551 | Schaudraff et al. | Sep 2017 | B2 |
| 9772282 | Tucker-Schwartz et al. | Sep 2017 | B2 |
| 9791659 | Yamakawa et al. | Oct 2017 | B2 |
| 9805166 | Spaulding | Oct 2017 | B1 |
| 9818190 | Chukka et al. | Nov 2017 | B2 |
| 9846938 | Steigauf et al. | Dec 2017 | B2 |
| 9892341 | Reicher et al. | Feb 2018 | B2 |
| 9928278 | Welinder et al. | Mar 2018 | B2 |
| 9934568 | Reicher et al. | Apr 2018 | B2 |
| 9940719 | Kneepkens | Apr 2018 | B2 |
| 9963745 | Mai et al. | May 2018 | B2 |
| 9972087 | Isoda et al. | May 2018 | B2 |
| 9979868 | Fujinami et al. | May 2018 | B2 |
| 10012537 | Garsha et al. | Jul 2018 | B2 |
| 10048462 | Mitsuyasu | Aug 2018 | B2 |
| 10049450 | Madabhushi | Aug 2018 | B2 |
| 10061107 | Loney et al. | Aug 2018 | B2 |
| 10102418 | Bredno | Oct 2018 | B2 |
| 10175154 | Campbell | Jan 2019 | B2 |
| 10444486 | Rainbolt et al. | Oct 2019 | B2 |
| 10489633 | Molin | Nov 2019 | B2 |
| 10527836 | Sakamoto et al. | Jan 2020 | B2 |
| 10572725 | Becker | Feb 2020 | B1 |
| 10573001 | Bredno et al. | Feb 2020 | B2 |
| 10600184 | Golden | Mar 2020 | B2 |
| 10643396 | Eastwood | May 2020 | B2 |
| 10663711 | Wirch et al. | May 2020 | B2 |
| 20080123916 | Adams et al. | May 2008 | A1 |
| 20100194681 | Halushka | Aug 2010 | A1 |
| 20110158491 | Markova | Jun 2011 | A1 |
| 20120002034 | Matsunobu et al. | Jan 2012 | A1 |
| 20120068928 | Bruss et al. | Mar 2012 | A1 |
| 20130077891 | Nimnual | Mar 2013 | A1 |
| 20130339260 | Johnson | Dec 2013 | A1 |
| 20140257857 | Meissner et al. | Sep 2014 | A1 |
| 20150110348 | Solanki et al. | Apr 2015 | A1 |
| 20150279032 | Hall et al. | Oct 2015 | A1 |
| 20150330776 | Hayashi et al. | Nov 2015 | A1 |
| 20160019691 | Imamura | Jan 2016 | A1 |
| 20160019695 | Chukka et al. | Jan 2016 | A1 |
| 20160042511 | Chukka et al. | Feb 2016 | A1 |
| 20160157751 | Mahfouz | Jun 2016 | A1 |
| 20160252713 | Corwin | Sep 2016 | A1 |
| 20160321809 | Chukka et al. | Nov 2016 | A1 |
| 20160350651 | Devarajan | Dec 2016 | A1 |
| 20170169567 | Chefd'hotel | Jun 2017 | A1 |
| 20170185846 | Hwangbo | Jun 2017 | A1 |
| 20170348509 | Burkholz et al. | Dec 2017 | A1 |
| 20170372471 | Eurcn | Dec 2017 | A1 |
| 20180188519 | Wirch et al. | Jul 2018 | A1 |
| 20180247408 | Wang | Aug 2018 | A1 |
| 20180315193 | Paschalakis | Nov 2018 | A1 |
| 20180322632 | Barnes | Nov 2018 | A1 |
| 20180374210 | Barker | Dec 2018 | A1 |
| 20190164054 | Lee | May 2019 | A1 |
| 20190293637 | Day | Sep 2019 | A1 |
| 20190304092 | Akselrod-Ballin | Oct 2019 | A1 |
| 20190362841 | Aysin | Nov 2019 | A1 |
| 20200126207 | Saltz | Apr 2020 | A1 |
| 20200167930 | Wang | May 2020 | A1 |
| 20200211191 | Joshi | Jul 2020 | A1 |
| 20200348503 | Wirch et al. | Nov 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| WO 2017087415 | May 2017 | WO |
| 2018129318 | Jul 2018 | WO |
| 2019226270 | Nov 2019 | WO |
| Entry |
|---|
| International Search Report and Written Opinion for In'l Application No. PCT/US2019/029006, titled: Multi-sample Whole Slide Image Processing via Multi-resolution Registration, dated Sep. 16, 2019. |
| Trahearn, Nicholas et al., “Hyper-Stain Inspector: A Framework for Robust Registration and Localised Co-Expression Analysis of Multiple Whole-Slide Images of Serial Histology Sections,” Scientific Reports, 7: 5641, 13 pages (Published Jul. 2017). |
| Roullier, Vincent et al., “Multi-resolution graph-based analysis of histopathological whole slide images: Application to mitotic cell extraction and visualization,” Computerized Medical Imaging and Graphics, 35: 603-615 (2011). |
| Rojo, M.G., et al., “Critical Comparison of 31 Commercially Available Digital Slide Systems in Pathology,” International Journal of Surgical Pathology, vol. 14, No. 4, pp. 285-305, Oct. 1, 2006. |
| Number | Date | Country | |
|---|---|---|---|
| 20190355113 A1 | Nov 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62674368 | May 2018 | US |
