Patent Application
20240265542
Glaucoma is the leading cause of visual morbidity and blindness worldwide, and it is projected to affect 111.8 million people by 2040. Visual loss from glaucoma is currently irreversible even with treatments, and early to moderate glaucoma is largely asymptomatic, because the visual loss usually starts from the periphery and slowly progresses towards the center. Prompt and accurate detection of glaucoma is extremely important in inhibiting and reducing irreversible visual impairment and blindness. Optical coherence tomography (OCT), a non-contact and non-invasive imaging technology for cross-sectional and three-dimensional (3D) view of the retina and optic nerve head (ONH), is now commonly used to evaluate the structural changes of glaucoma, i.e., glaucomatous optic neuropathy (GON, also referred to as “glaucoma” herein). OCT is widely used to quantify retinal nerve fiber layer (RNFL), neuro-retinal rim, and other inner retinal layers (e.g., ganglion cell layer, inner plexiform layer). OCT is sensitive and specific for detecting GON, especially when combined with other ophthalmoscopic modalities.
However, poor scan quality due to patients' poor co-operation, operators' skills, or device-dependent factors (e.g., inaccurate optic disc margins delineation) can affect the metrics generated from the OCT. Conventionally, for commercial systems (e.g., the Cirrus High-Definition OCT, Carl Zeiss Meditec, Dublin, CA, USA), signal strength (SS) is the main parameter to include or exclude OCT scans for further quantitative analysis. Image quality is indicated by SS ranging from 0 (worst quality) to 10 (best quality), representing the average of signal intensity of OCT volumetric scans, and scans with SS of 6 or above are often defined as sufficient for the further analysis. However, even with acceptable SS, it is still hard to assess other OCT image quality issues, such as off-centration, out of registration, signal loss, motion artifacts, mirror artifacts, or blurriness of OCT volumetric data. Such image quality assessment indeed requires highly trained operators and interpreters with specialized knowledge in OCT which is a big challenge due to the lack of manpower in clinics. Besides, it is impractical for human assessors to grade every OCT volumetric scan which can be a time-consuming and tedious process, particularly in busy clinics.
In addition, myopic optic disc morphology (also referred to as “myopic features” or, “MF” herein), such as peripapillary atrophy (PPA) and optic disc tilting, also influences GON identification based on RNFL thickness measurement alone, which should be taken into account when interpreting the optic disc and its circumpapillary regions for diagnosis. For example, PPA beta zone correlates with glaucoma, while gamma zone is related to axial globe elongation. A higher degree of vertical optic disc tilting is associated with a more temporally positioned RNFL thickness peak. Eyes with longer axial length are associated with significantly higher percentages of false-positive errors based on an OCT built-in normative database. Hence, evaluating glaucoma structural changes using OCT based on RNFL thickness and built-in normative databases alone may not be reliable. MF can also result in thinning of RNFL thickness (i.e., outside of the normal RNFL range) in eyes without glaucoma structural changes. Other diagraphs and metrics, such as topographical ONH measurements, RNFL thickness map, RNFL deviation map, and circumpapillary RNFL thickness with “double-hump pattern” should also be evaluated to differentiate these two pathologies carefully. For example, in purely myopic eyes, the “double-hump pattern” can be present but with temporal shift due to optic disc tilting. The RNFL thickness map also shows normal thickness except that the angle between superior and inferior RNFL bundles is smaller. While in eyes with glaucoma, RNFL “double-hump pattern” is altered and thinner RNFL thickness appears at specific regions. Thus, interpretation of the results requires experienced glaucoma specialists or highly trained assessors who have good knowledge on both glaucoma and OCT limitations.
Embodiments of the subject invention provide three-dimensional (3D) artificial intelligence (AI)-aided classification systems and methods for glaucomatous optic neuropathy (GON) and myopic optic disc morphology (MF) from optical coherence tomography (OCT) scans. Embodiments provide a novel method of building an AI platform integrating an information management system, an AI image analysis system, and a user interface. In certain embodiments the image analysis system includes image quality assessment, GON detection, and MF detection. Embodiments provide systems and methods for deep learning (DL), composed of multiple processing layers, that allows computational models to learn representative features with multiple levels of abstraction. Embodiments provide models useful in pattern recognition and image analysis. Embodiments provide a 3D AI-aided automated image analysis for glaucoma and myopia structural changes classification from volumetric OCT scans, which includes a DL-based pre-diagnosis model developed by SE-ResNeXt in 3D version for image quality control (e.g., providing an outcome of “sufficient” or “insufficient”) and a multi-task DL-based classification model developed by ResNet-37 in 3D version for glaucoma and myopia structural changes (e.g., providing outcomes of “Yes GON” or “No GON” and “Yes MF” or “No MF” with AI scores). Embodiments generate heatmaps using class activation map for visualizing the identified features.
Embodiments of the subject invention provide an AI-platform with the integration of developed 3D DL models (e.g., an image quality control model, a multi-task model for glaucoma and myopic features detection), an information management system (e.g., a graphic processing unit server enabling rapid data storage, search, and retrieval), and a commercially available OCT device (e.g., Cirrus HD-OCT which can directly export raw volumetric data and XML file for data extraction and analysis, or other devices now known or later developed). Embodiments of the provided AI-platform include a user interface (front-end and back-end) for real-time OCT image extraction, input data configuration (e.g., subject study ID, exam date, age, gender, imaging protocol), image uploading, images analysis via a graphics processing unit (GPU) server, and AI reports generation. Embodiments provide AI reports with outputs including but not limited to image quality, glaucoma classification, myopic features classification, AI scores, and referral suggestions rapidly (e.g., within 5 minutes for a typical clinical imaging data set.)
Embodiments of the subject invention provide systems and methods for a novel 3D AI-aided classification system for GON and MF from OCT volumetric scans with additional integrated image quality control. Embodiments exhibit numerous advantages, including but not limited to robust application across disparate datasets and populations, integrated DL techniques, straightforward (e.g., yes/no) outputs facilitate rapid screening, improved performance from 3D volumetric scanning, integrated platform from user interface through AI to output, direct integration to bring AI analysis to commonly available imaging platforms, and further integration with additional deep learning models for the detection of additional disease states.
Embodiments provide a novel method of building an AI platform integrating an information management system, an AI image analysis system, and a user interface. The image analysis system includes image quality assessment, GON detection, and MF detection.
In certain embodiments the development and testing datasets can be collected from multiple eye centers from different countries and regions including different ethnic backgrounds. Embodiments have performed consistently well in all tested datasets. The training-tuning curves also illustrated that the proposed DL model was not overfitted. Thus, embodiments can be applied on other unseen datasets, even among different populations.
Embodiments provide state-of-the-art DL techniques such as irrelevancy reduction and self-attention mechanism for the image quality control task, and multi-task technique for the disease detection task. Irrelevancy reduction omits the parts of irrelevant signals that should not be noticed by the signal receiver, which can improve the AI performance for image quality control. In certain embodiments denoising is provided to reduce the irrelevancies of OCT scans since the noise of OCT scans can impeded the medical analysis either visually or programmatically. For denoising, embodiments provide nonlocal means which can be performed both vertically (along x, z facets) and horizontally (along x, y facets) with different sets of parameters. Vertically, the template window size can be set to 10, whereas the search window size can be set to 5 with a filter strength of 5. Horizontally, the template window size can be set to 5, and search window size can be set to 5 with a filter strength of 5. The self-attention mechanism is provided to help the model recognize the more important areas and extract features automatically in the original OCT volumetric scans. Multi-task learning is provided as a training paradigm to train DL models with data from multiple tasks simultaneously, using shared representations to learn the common features between a collection of related tasks, which provides advantages of integrating information across domains and extracting more general features for different tasks.
Embodiments generate straightforward output of Yes/No GON and Yes/No MF with automated image quality assessment, which can further strengthen OCT as a screening tool in settings without sufficient ophthalmologists experts on site as it provides a clear AI report with outcomes of image quality (“sufficient” or “insufficient”), GON detection (“Yes GON” or “No GON”), MF detection (“Yes MF” or “No MF”) with AI scores, and referral suggestions. End users, such as primary care technicians, optometrists, or family physicians, can interpret the AI report easily.
Embodiments analyze 3D OCT scans and show generally better performance than related art 2D models analyzing cross-sectional 2D B-scans for both GON and MF detection. OCT receives tissue information in depth by measuring the magnitude and echo delay of backscattered light. Cross-sectional images are generated by transversely scanning the incident optical beam and performing axial scans, named B-scan. Volumetric scans can be generated by raster scanning a series of cross-sectional images (i.e., B-scans). For certain type of commercialized OCT device (e.g., Cirrus OCT), there can be 200 or 128 B-scans in each volumetric scan depending on the imaging protocol. Embodiments provide volume-level output instead of B-scan level output which is more straightforward for physicians (e.g., non-ophthalmologists) to interpret the results and requires less manpower or computation power to deal with a required large number of B-scans.
Embodiments of the subject invention provide an AI-platform that includes a user interface (e.g., front-end and back-end) for real-time OCT image extraction, input data configuration (e.g., subject study ID, exam date, age, gender, imaging protocol), image uploading, and images analysis via a graphics processing unit (GPU) server, and AI reports generation. Embodiments provide outputs including image quality, glaucoma classification, and myopic features classification within a few minutes. In certain embodiments the AI-platform is implemented into a commercial OCT device and configured to automatedly detect the exported data for further image analysis and disease detection (
In certain embodiments the AI-platform integrates additional DL models using OCT macula volumetric scans to detect other diseases such as diabetic macula edema in the future (
Certain embodiments of the subject invention provide algorithms, programs, systems, or methods for identifying GON and MF by pre-diagnosis image quality control and ensuring the gradeability of OCT scans by providing an immediate onsite assessment of image quality. This can allow retaking of OCT, if necessary, of subjects within the same visit and also reduce the expertise required in collecting OCT images (
Embodiments of the subject invention provide improved medical care to millions of patients at risk for glaucoma (e.g., elderly patients, 65 years and over) to inhibit or prevent irreversible vision loss, with faster and more reliable screening delivered at a reduced overall cost. These benefits are multiplied when patients are recommended for re-screening every two years.
The inventors have tested the image quality control and glaucoma and myopia structural changes DL modules in retrospective multi-center cohorts. The performance results are shown in Table 1, and Table 2. Embodiments of the subject invention provide an AI-platform to integrate the provided DL modules with an information management system for clinical deployment with an OCT device.
In creating certain embodiments, the inventors utilized a series of data pre-processing methods and on-the-fly data augmentation methods to train the provided DL model with reduced GPU memory costs while avoiding the over-fitting issue. Embodiments provide 3D DL models to analyze the OCT volumetric images. Embodiments provide a multi-task technique to develop a 3D DL model for classifying both GON and MF.
Embodiments have withstood external testing from different centers in different countries featuring different patient populations for the provided DL models, verifying generalizability as shown in Table 1 and Table 2.
Embodiments provide class activation maps to visualize the discriminative features (i.e., heatmaps). The feature maps, i.e., the intermediate outputs of the network layers, before the global average pooling layer as well as the parameters of the fully connected layer can be used to obtain the heatmap. In one exemplary embodiment there are 256 feature maps each with dimension 4×4×32, while the parameters of the fully connected layer are of dimension 1×1×1×256. The sum of the feature maps weighted by the parameters can be taken to generate the class activation map. Each entry in the weights represents the importance of each feature map. Finally, the class activation map can be resized to the same dimension of the original OCT image by interpolation to obtain the heatmap. These heatmaps can provide end-users some insights on where the discriminative areas are for the AI to detect diseases.
Embodiments provide an AI-platform that integrates the provided image quality control DL model into the provided disease classification model, which provides a more accurate disease detection.
Embodiments of the provided AI-platform can be implemented into available commercialized OCT devices and configured to automatedly detect the exported data for further image analysis and disease detection.
Embodiments of the provided AI-platform can also integrate other DL models for other disease detection.
Embodiments of the subject invention address the technical problem of detecting GON and MF from imaging such as OCT image data being expensive, needing excessive human processing and experience in ophthalmology, not being suitable for rapid screening, and requiring expert resources to complete.
This problem is addressed by providing a system for 3D AI-aided classification using digital image processing in which a machine learning method applying a combination of advanced techniques is utilized within an AI-platform to provide easy-to-interpret AI reports with outputs including image quality, glaucoma classification, myopic features classification, AI scores, and referral suggestions within a few minutes.
The transitional term “comprising,” “comprises,” or “comprise” is inclusive or open-ended and does not exclude additional, unrecited elements or method steps. By contrast, the transitional phrase “consisting of” excludes any element, step, or ingredient not specified in the claim. The phrases “consisting of” or “consists essentially of” indicate that the claim encompasses embodiments containing the specified materials or steps and those that do not materially affect the basic and novel characteristic(s) of the claim. Use of the term “comprising” contemplates other embodiments that “consist of” or “consisting essentially of” the recited component(s).
When ranges are used herein, such as for dose ranges, combinations and subcombinations of ranges (e.g., subranges within the disclosed range), specific embodiments therein are intended to be explicitly included. When the term “about” is used herein, in conjunction with a numerical value, it is understood that the value can be in a range of 95% of the value to 105% of the value, i.e., the value can be +/−5% of the stated value. For example, “about 1 kg” means from 0.95 kg to 1.05 kg.
The methods and processes described herein can be embodied as code and/or data. The software code and data described herein can be stored on one or more machine-readable media (e.g., computer-readable media), which may include any device or medium that can store code and/or data for use by a computer system. When a computer system and/or processor reads and executes the code and/or data stored on a computer-readable medium, the computer system and/or processor performs the methods and processes embodied as data structures and code stored within the computer-readable storage medium.
It should be appreciated by those skilled in the art that computer-readable media include removable and non-removable structures/devices that can be used for storage of information, such as computer-readable instructions, data structures, program modules, and other data used by a computing system/environment. A computer-readable medium includes, but is not limited to, volatile memory such as random access memories (RAM, DRAM, SRAM); and non-volatile memory such as flash memory, various read-only-memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic/ferroelectric memories (MRAM, FeRAM), and magnetic and optical storage devices (hard drives, magnetic tape, CDs, DVDs); network devices; or other media now known or later developed that are capable of storing computer-readable information/data. Computer-readable media should not be construed or interpreted to include any propagating signals. A computer-readable medium of embodiments of the subject invention can be, for example, a compact disc (CD), digital video disc (DVD), flash memory device, volatile memory, or a hard disk drive (HDD), such as an external HDD or the HDD of a computing device, though embodiments are not limited thereto. A computing device can be, for example, a laptop computer, desktop computer, server, cell phone, or tablet, though embodiments are not limited thereto.
The invention may be better understood by reference to certain illustrative exemplary embodiments, including but not limited to the following:
Embodiment 1. A system for three-dimensional (3D) artificial intelligence (AI)-aided classification of glaucomatous optic neuropathy (GON) and myopic optic disc morphology (MF) from optical coherence tomography (OCT) scans, the system comprising:
Embodiment 2. The system of Embodiment 1, wherein the pre-diagnosis image quality assessment model comprises a three-dimensional squeeze-and-excitation (SE) model.
Embodiment 3. The system of Embodiment 2, wherein the three-dimensional SE model is based on a ResNeXt architecture.
Embodiment 4. The system of Embodiment 3, wherein the three-dimensional SE model comprises a multiplicity of SE-ResNeXt building blocks.
Embodiment 5. The system of Embodiment 4, wherein the three-dimensional SE model comprises batch normalization, global average pooling, convolution, and averaging.
Embodiment 6. The system of Embodiment 1, wherein the 3D SE model is configured and adapted to produce an output comprising a confirmation of image quality sufficient for reliable operation of the GON and MF multi-task detection model.
Embodiment 7. The system of Embodiment 6, wherein the GON and MF multi-task detection model comprises a shared feature extraction module configured and adapted to deliver a respective input to each of a GON classification module and an MF detection module.
Embodiment 8. The system of Embodiment 7, wherein the shared feature extraction module comprises convolution, batch normalization, a multiplicity of residual units, and global average pooling.
Embodiment 9. The system of Embodiment 7, wherein the GON classification module, and the MF detection module each, respectively, comprise a fully connected softmax layer.
Embodiment 10. The system of Embodiment 7, wherein the GON and MF multi-task detection model utilizes ResNet-37 in a 3D version.
Embodiment 11. The system of Embodiment 7, wherein the pre-diagnosis image quality assessment model comprises an irrelevancy reduction mechanism and a self-attention mechanism; and the GON and MF multi-task detection model is trained using a multi-task learning paradigm.
Embodiment 12. An artificial intelligence (AI) enhanced system for three-dimensional classification of glaucomatous optic neuropathy (GON) and myopic optic disc morphology (MF) from optical coherence tomography (OCT) scans, the system comprising:
Embodiment 13. The system according to Embodiment 1, wherein:
Embodiment 14. The system according to Embodiment 12, the instructions when executed further performing the following additional steps:
Embodiment 15. The system according to Embodiment 13, the instructions when executed further performing the following additional steps:
Embodiment 16. The system according to Embodiment 15, the instructions when executed further performing the following additional steps:
Embodiment 17. The system according to Embodiment 16, the instructions when executed further performing the following additional steps:
Embodiment 18. The system according to Embodiment 17, the instructions when executed further performing the following additional steps:
Embodiment 19. A system for rapid three-dimensional artificial intelligence-aided classification of glaucomatous optic neuropathy (GON) and myopic optic disc morphology (MF) from optical coherence tomography (OCT) scans, the system comprising:
Embodiment 20. The system of Embodiment 19, wherein the shared feature extraction module comprises convolution, batch normalization, a multiplicity of residual units, and global average pooling;
Turning now to the figures,
By way of example, but not limitation, referral suggestions can include: 1) “Refer to glaucoma specialists”: any eye “Yes GON” with or without any eye “Yes MF”; 2) “Non urgent referral”: both eyes “No GON” and any eye “Yes MF”; 3) “Observe only”: both eyes “No GON” and both eyes “No MF”.
The benefits of the intervention by providing a clear AI report with outcomes of image quality (e.g., a binary determination such as, “sufficient” or “insufficient”), GON detection (e.g., a binary determination such as, “Yes GON” or “No GON”), MF detection (e.g., a binary determination such as, “Yes MF” or “No MF”) with AI scores, and referral suggestions will be more user-friendly and efficient for end-users' interpretation in a busy clinic. If there are further needs, end-users can also review the raw images via the information management system.
All patents, patent applications, provisional applications, and publications referred to or cited herein are incorporated by reference in their entirety, including all figures and tables, to the extent they are not inconsistent with the explicit teachings of this specification.
Following are examples that illustrate procedures for practicing the invention. These examples should not be construed as limiting. All percentages are by weight and all solvent mixture proportions are by volume unless otherwise noted.
Data augmentation strategies, including random flipping, random rotating, and random shifting, were used to enhance the training samples and alleviate overfitting. The original OCT volumes were with size of 200×200×1024 in three axes, x-axis, y-axis, and z-axis, respectively. To mimic the real OCT imaging in clinical practice, some data augmentation methods were only applied on one or two axes for the whole volume. For instance, 20% chance random flipping, 15-degree random rotation were applied on only x-axis (200) and y-axis (200). The color channel was set to one since all OCT images were grey scaled.
The DL model was implemented with Keras and Tensorflow, on a workstation equipped with i9-7900× and Nvidia GeForce GTX 1080Ti. Firstly, there were 32 filters with 7×7×7 kernel convolution layer with the stride of 2, along with a 3×3×3 max pooling with the same stride setting. Secondly, the obtained feature maps went through 18 ResNet blocks. A pooling size 2 with stride 2 average pooling was performed every 3 blocks to aggregate the learnt features. Channel-wise batch normalization and ReLU activation were performed after all convolution operations. Finally, a global average pooling followed by a fully connected softmax layer was used to produce the binary output as gradable or ungradable. This ResNet-based model was taken as the benchmark model. The inventors further experimented with the SE-ResNet-block27 and SE-ResNeXt-block28, as the basic building block. In each SE-ResNet or SE-ResNeXt block, the SE reduction ratio was set to 4 and the cardinalities of the transformation layer were set to 8, with 32 filters. Cross-entropy and Adam were used as the loss function and the optimizer. During the training, 3,000 volumetric scans were selected with data balancing. Batch size was set to one due to the limited GPU memory. The initial learning rate was set to 0.0001, and then reduced by multiplying 0.75 in every 2 epochs.
The inventors applied standardization and normalization for data pre-processing. Specifically, standardization was used to transfer data to have zero mean and unit variance, and normalization rescaled the data to the range of 0 to 1. To alleviate the over-fitting issue, during the training process, the inventors used several data augmentation techniques, including random cropping and random flipping at three axes, to enrich training samples for the 3D OCT volumetric data. Consequently, the final input size of the network was 200×1000×200. The inventors implemented the DL model using Keras package and python on a workstation equipped with 3.5 GHz Intel® Core™ i7-5930K CPU and GPUs of Nvidia GeForce GTX Titan X. The inventors set the learning rate as 0.0001 and optimized the weights of the networks with Adam stochastic gradient descent algorithm.
The provided network included three modules, 1) shared feature extraction module, 2) glaucoma classification module, and 3) myopic features detection module, respectively. The constructed network was similar to the inventors' previous study29 with ResNet-37 as the backbone. The inventors used shortcut connections to perform identity mapping and evade the vanishing gradient problem during backpropagation. The inventors removed the fully connected layer from the 3D ResNet-37. This module acted as the shared feature extraction module. In the GON classification module, a fully connected layer with softmax activation accepted the feature from the first module and output the classification probabilities for “Yes GON” and “No GON”. Likewise, there was also a fully connected layer with softmax activation in the MF detection module and output the classification probabilities for “Yes MF” and “No MF”.
All gradable OCT volumetric scans were randomly divided for training (80%), tuning (10%), and internal validation (10%) at the patient level. In each set, the ratio of “Yes GON & Yes MF”, “Yes GON & No MF”, “No GON & Yes MF”, and “No GON & No MF” was similar, and multiple images from the same subjects were in the same set to inhibit leakage or performance overestimation. The inventors trained the multi-task DL model from scratch, and the tuning dataset was used to select and modify the optimum model during training. During the training, tuning, and internal validation, the inventors observed the training-validation curve to evaluate for any over-fitting issue, which could also provide a further reference to the generalizability of the models. Additionally, OCT volumetric scans from 5 centers were used for external testing.
Finally, the inventors generated heatmaps for selected eyes by class activation map (CAM)30 to visualize the classification. Results are seen in Table 1, Table 2, and
It should be understood that the examples and embodiments described herein are for illustrative purposes only and that various modifications or changes in light thereof will be suggested to persons skilled in the art and are to be included within the spirit and purview of this application and the scope of the appended claims. In addition, any elements or limitations of any invention or embodiment thereof disclosed herein can be combined with any and/or all other elements or limitations (individually or in any combination) or any other invention or embodiment thereof disclosed herein, and all such combinations are contemplated with the scope of the invention without limitation thereto.
Ophthalmology 2009; 116(7): 1257-63, 63 el-2.
