Patent Application
20220004838
Typically, medical imaging is the technique and process of creating visual representations of the interior of a body for clinical analysis and medical intervention, as well as visual representation of the function of some organs or tissues (physiology). One of the goals of medical imaging is to reveal internal structures hidden by the skin and bones, as well as to diagnose and treat disease.
Described herein is a system that can be used to perform any of several functions on medical images:
At least one embodiment is designed for use in the context of mammography screening exams. In this embodiment, the abnormalities of interest are primarily those that influence the likelihood of a diagnosis of cancer, as described in the Breast Imaging Reporting and Data System (BI-RADS) published by the American College of Radiology (ACR). See ACR BI-RADS Atlas® 5th Edition, available at https://www.acr.org/Clinical-Resources/Reporting-and-Data-Systems/Bi-Rads. These abnormalities may include suspected malignant lesions as well as optionally suspected benign lesions, such as fibroadenomas, cysts, lipomas, and others.
Beyond breast imaging, many other embodiments of the described system involving different types of abnormalities observed in radiology are possible, including but not limited to:
We describe separately embodiments of each of these systems:
At least one purpose of the Abnormality Flagging User Interface is to provide a list of studies to a radiologist and call their attention to those studies that may be higher priority than others. Those studies may be of higher priority for many reasons, including but not limited to:
In radiological terms, multiple images that make up a single acquisition are defined as a series. Multiple series that correspond to a single scanning session are defined as a study. A study may contain one or more series, and a series may contain one or more images. These precise definitions are not integral to the design of the system described herein; for the purpose of this description, a study is defined as a collection of one or more related images of a single patient from a single scanning session.
For each of the studies in the list, image (pixel) data is loaded at (108). That image data is optionally combined with clinical data (110) or other data and processed at (112). The processing may include inference by a convolutional neural network (CNN), inference by other machine learning algorithms, heuristic algorithms, decision trees, other image processing, any combination of the above or other processing techniques. Optionally, if the processing results in a sortable characteristic (such as percent likelihood of an abnormality being present), the images may be sorted by that characteristic at (116). Optionally, one or more characteristics may be associated with each study at (114), without those characteristics necessarily being used to sort or rank the studies. In the case of screening mammography, various characteristics may be assessed, including but not limited to:
If characteristics of the studies are determined, some studies may optionally be removed from the list at (118), such as those that have a low likelihood of containing an abnormality, or those that have a low likelihood of diagnostic quality. The list of studies, which in some embodiments will be curated or sorted, is then displayed to the user at (120) on a display at (122). The displayed list of studies may include some indication of the characterization of the studies, such as a “high priority” flag adjacent to studies characterized as likely to contain one or more abnormalities. Multiple indications may be used for separate abnormalities or groups of abnormalities, such as, e.g., a separate indication for any of, but not limited to:
The data processing at (112) can be accomplished in various ways, including by one or more convolutional neural network (CNN) models. One or more CNNs may be a detection model, and one or more may be a segmentation model. One or more CNNs may return any of various results, including but not limited to:
In this embodiment, images that compose a study are originally contained in a Study Database (302). Those studies are analyzed at (304). In this embodiment, the results of the analysis are an abnormality probability at (306). This probability itself may be displayed any of several ways, including as a raw probability value, such as a percentage at (316), or as a graph, color scale, or other visual indicator of the percentage at (318). The abnormality probability may be quantized at (308) into discrete risk levels and that risk level may be displayed as a textual label, such as “low,” “medium” and “high,” or using a visual indicator such as a number of bars or dots, or a color, such as green, yellow and red, at (312). Alternatively, the abnormality probability may be thresholded into a Boolean True or False value at (310) which would indicate whether the abnormality is likely to be present; this could be displayed as a flag, as in (210), as a highlight, or via other indicators at (314).
One purpose of the Abnormality Detection User Interface is to provide a visual indication of the location of suspected abnormalities on the original radiological pixel data. These visual indications guide the user's eye to the abnormality to allow the user to confirm or deny any of the presence, characteristics, or diagnosis of the abnormality. Some embodiments of this type of interface may be referred to as Computer Aided Detection, or CAD or CADe.
In this embodiment, the characteristics are shown adjacent to the annotation overlaid on the pixel data; however, they could also be shown in the sidebar at (504), in a modal dialog, or in other formats. The characteristics may be displayed when the image is first opened, or they may be revealed upon some interaction with the annotation or the sidebar list, such as via a tap or click.
The Abnormality Detection Machine Learning Model is a system that ingests image data, possibly in conjunction with other clinical data, and returns an assessment of some subset of abnormality locations, classifications, and probabilities. The embodiments described here operate in the context of mammography screening, but an equivalent system could be used in any medical environment involving an assessment of abnormalities in medical images.
The abnormality location proposals may also include associated probabilities for different classes or diagnoses. For example a location proposal may be designated at a malignant lesion with 75% probability, pre-cancerous ductal carcinoma in situ with 20% probability and an intramammary lymph node with 5% probability. The output optionally includes characteristics for those abnormalities at (910). The location proposals may define proposed locations for any abnormalities regardless of subtype, or there may be separate location proposals for specific subclasses of abnormalities (e.g., invasive cancers, non-malignant tumors, cysts, calcifications, etc.). The location proposals (908) may also include confidence indicators or probabilities that the specific proposed location contains the given abnormality. Abnormality characteristics (910), if assessed, may include, without being limited to:
After being calculated, one or both of the location proposals and characteristics are optionally presented to the user at (912) on a display at (914). In at least some embodiments, only abnormalities detected with high confidence from one or more CNNs are shown. In at least some abnormalities, the likelihood of one or more classes of abnormality or characteristics are displayed. In at least some embodiments, one or both of the location proposals and characteristics are saved to a database for later display or analysis.
One or more of the CNNs at (904) may include a backbone (pre-trained) CNN, a classification CNN or a bounding box regression CNN. The backbone CNN, if included, may be based on a classification, segmentation or other CNN. One or more of the CNNs may be trained with any of various loss functions, including but not limited to focal loss. Focal loss is a modification of standard cross entropy loss such that the loss of predictions whose probabilities are close to the true prediction are downweighted such that their values are reduced when compared to cross entropy loss. One or more CNNs may either operate perform inference on a full input image, or on patches extracted from the input image.
The detection CNNs at (1004) may have any of the same properties as the CNNs at (904).
A medical study is loaded at (1102) and is divided into one or more of its constituent medical images at (1104). Note that although a pipeline consisting of three separate images is shown in (1104) through (1110), any number of images could be analyzed in this pipeline. A trained CNN model at (1106) performs inference on each of the images at (1108). Inference may be performed on each image independently, or inference may be performed on some subsets of images simultaneously (for example, multiple images that constitute a volume, or images representing the same anatomy that have been acquired with Mill different pulse sequences). In at least some embodiments, inference includes one or both of detection or characterization of abnormalities. The output of inference is a set of image-level characteristics at (1110). As with inference, these characteristics may be associated with an individual image, or with a collection of images. These characteristics are then synthesized together at (1116), optionally combined with patient demographic data, such as age, sex, lifestyle choices, family disease history, etc., at (1112) or patient electronic health record (EHR) data, such as disease history, test results, procedures, etc. (1114). The output is a set of study level characteristics at (1118).
In at least one embodiment of this system, a study includes mammography screening images that are taken with different views of the two breasts. For example, each of the left and right breasts may have images acquired in the craniocaudal and mediolateral oblique views, resulting in a total of four images. In this embodiment, a lesion detection CNN is applied independently to each image and generates location proposals for detected lesions, along with confidence levels of the proposals for each of various classes of abnormality, such as malignancies and other lesions. A gradient boosted tree algorithm takes in a table containing the list of proposals, their confidence levels, the view and breast side with which the proposals are associated, as well as demographic and clinical data that is associated with breast cancer risk such as age, family history and BRCA mutation status. That gradient boosted tree algorithm then assigns an overall confidence level that any lesion is present in the study. That confidence level may be a continuous score, or it may be quantized to two or more levels of confidence. Quantization to more than 10 classes of likelihood is unlikely to provide significant value over a continuous confidence level.
The processor-based device 1204 may include one or more processors 1206, a system memory 1208 and a system bus 1210 that couples various system components including the system memory 1208 to the processor(s) 1206. The processor-based device 1204 will at times be referred to in the singular herein, but this is not intended to limit the implementations to a single system, since in certain implementations, there will be more than one system or other networked computing device involved. Non-limiting examples of commercially available systems include, but are not limited to, ARM processors from a variety of manufactures, Core microprocessors from Intel Corporation, U.S.A., PowerPC microprocessor from IBM, Sparc microprocessors from Sun Microsystems, Inc., PA-RISC series microprocessors from Hewlett-Packard Company, 68xxx series microprocessors from Motorola Corporation.
The processor(s) 1206 may be any logic processing unit, such as one or more central processing units (CPUs), microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Unless described otherwise, the construction and operation of the various blocks shown in
The system bus 1210 can employ any known bus structures or architectures, including a memory bus with memory controller, a peripheral bus, and a local bus. The system memory 1208 includes read-only memory (“ROM”) 1012 and random access memory (“RAM”) 1214. A basic input/output system (“BIOS”) 1216, which can form part of the ROM 1212, contains basic routines that help transfer information between elements within processor-based device 1204, such as during start-up. Some implementations may employ separate buses for data, instructions and power.
The processor-based device 1204 may also include one or more solid state memories, for instance Flash memory or solid state drive (SSD), which provides nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the processor-based device 1204. Although not depicted, the processor-based device 1204 can employ other nontransitory computer- or processor-readable media, for example a hard disk drive, an optical disk drive, or memory card media drive.
Program modules can be stored in the system memory 1208, such as an operating system 1230, one or more application programs 1232, other programs or modules 1234, drivers 1236 and program data 1238.
The application programs 1232 may, for example, include panning/scrolling 1232a. Such panning/scrolling logic may include, but is not limited to logic that determines when and/or where a pointer (e.g., finger, stylus, cursor) enters a user interface element that includes a region having a central portion and at least one margin. Such panning/scrolling logic may include, but is not limited to logic that determines a direction and a rate at which at least one element of the user interface element should appear to move, and causes updating of a display to cause the at least one element to appear to move in the determined direction at the determined rate. The panning/scrolling logic 1232a may, for example, be stored as one or more executable instructions. The panning/scrolling logic 1232a may include processor and/or machine executable logic or instructions to generate user interface objects using data that characterizes movement of a pointer, for example data from a touch-sensitive display or from a computer mouse or trackball, or other user interface device.
The system memory 1208 may also include communications programs 1240, for example a server and/or a Web client or browser for permitting the processor-based device 1204 to access and exchange data with other systems such as user computing systems, Web sites on the Internet, corporate intranets, or other networks as described below. The communications programs 1240 in the depicted implementation is markup language based, such as Hypertext Markup Language (HTML), Extensible Markup Language (XML) or Wireless Markup Language (WML), and operates with markup languages that use syntactically delimited characters added to the data of a document to represent the structure of the document. A number of servers and/or Web clients or browsers are commercially available such as those from Mozilla Corporation of California and Microsoft of Washington.
While shown in
A user can enter commands and information via a pointer, for example through input devices such as a touch screen 1248 via a finger 1244a, stylus 1244b, or via a computer mouse or trackball 1244c which controls a cursor. Other input devices can include a microphone, joystick, game pad, tablet, scanner, biometric scanning device, etc. These and other input devices (i.e., “I/O devices”) are connected to the processor(s) 1206 through an interface 1246 such as touch-screen controller and/or a universal serial bus (“USB”) interface that couples user input to the system bus 1210, although other interfaces such as a parallel port, a game port or a wireless interface or a serial port may be used. The touch screen 1248 can be coupled to the system bus 1210 via a video interface 1250, such as a video adapter to receive image data or image information for display via the touch screen 1248. Although not shown, the processor-based device 1204 can include other output devices, such as speakers, vibrator, haptic actuator, etc.
The processor-based device 1204 may operate in a networked environment using one or more of the logical connections to communicate with one or more remote computers, servers and/or devices via one or more communications channels, for example, one or more networks 1214a, 1214b. These logical connections may facilitate any known method of permitting computers to communicate, such as through one or more LANs and/or WANs, such as the Internet, and/or cellular communications networks. Such networking environments are well known in wired and wireless enterprise-wide computer networks, intranets, extranets, the Internet, and other types of communication networks including telecommunications networks, cellular networks, paging networks, and other mobile networks.
When used in a networking environment, the processor-based device 1204 may include one or more wired or wireless communications interfaces 1252a, 1256 (e.g., cellular radios, WI-FI radios, Bluetooth radios) for establishing communications over the network, for instance the Internet 1214a or cellular network 1214b.
In a networked environment, program modules, application programs, or data, or portions thereof, can be stored in a server computing system (not shown). Those skilled in the relevant art will recognize that the network connections shown in
For convenience, the processor(s) 1206, system memory 1208, network and communications interfaces 1252a, 1256 are illustrated as communicably coupled to each other via the system bus 1210, thereby providing connectivity between the above-described components. In alternative implementations of the processor-based device 1204, the above-described components may be communicably coupled in a different manner than illustrated in
The foregoing detailed description has set forth various implementations of the devices and/or processes via the use of block diagrams, schematics, and examples. Insofar as such block diagrams, schematics, and examples contain one or more functions and/or operations, it will be understood by those skilled in the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In one implementation, the present subject matter may be implemented via Application Specific Integrated Circuits (ASICs). However, those skilled in the art will recognize that the implementations disclosed herein, in whole or in part, can be equivalently implemented in standard integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers) as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of ordinary skill in the art in light of this disclosure.
Those of skill in the art will recognize that many of the methods or algorithms set out herein may employ additional acts, may omit some acts, and/or may execute acts in a different order than specified.
In addition, those skilled in the art will appreciate that the mechanisms taught herein are capable of being distributed as a program product in a variety of forms, and that an illustrative implementation applies equally regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of signal bearing media include, but are not limited to, the following: recordable type media such as floppy disks, hard disk drives, CD ROMs, digital tape, and computer memory.
The various implementations described above can be combined to provide further implementations. To the extent that they are not inconsistent with the specific teachings and definitions herein, all of the U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and/or listed in the Application Data Sheet, including but not limited to U.S. Provisional Patent Application No. 61/571,908 filed Jul. 7, 2011; U.S. Pat. No. 9,513,357 issued Dec. 6, 2016; U.S. patent application Ser. No. 15/363,683 filed Nov. 29, 2016; U.S. Provisional Patent Application No. 61/928,702 filed Jan. 17, 2014; U.S. patent application Ser. No. 15/112,130 filed Jul. 15, 2016; U.S. Provisional Patent Application No. 62/260,565 filed Nov. 20, 2015; 62/415203 filed Oct. 31, 2016; U.S. patent application Ser. No. 15/779,445 filed May 25, 2018, U.S. patent application Ser. No. 15/779,447 filed May 25, 2018, U.S. Provisional Patent Application No. 62/415,666 filed Nov. 1, 2016; U.S. patent application Ser. No. 15/779,448, filed May 25, 2018, U.S. Provisional Patent Application No. 62/451,482 filed Jan. 27, 2017; International Patent Application No. PCT/US2018/015222 filed Jan. 25, 2018, U.S. Provisional Patent Application No. 62/501,613 filed May 4, 2017; International Patent Application No. PCT/US2018/030,963 filed May 3, 2018, U.S. Provisional Patent Application No. 62/512,610 filed May 30, 2017; U.S. patent application Ser. No. 15/879,732 filed Jan. 25, 2018; U.S. patent application Ser. No. 15/879,742 filed Jan. 25, 2018; U.S. Provisional Patent Application No. 62/589,825 filed Nov. 22, 2017; U.S. Provisional Patent Application No. 62/589,805 filed Nov. 22, 2017; U.S. Provisional Patent Application No. 62/589,772 filed Nov. 22, 2017; U.S. Provisional Patent Application No. 62/589,872 filed Nov. 22, 2017; U.S. Provisional Patent Application No. 62/589,876 filed Nov. 22, 2017; U.S. Provisional Patent Application No. 62/589,766 filed Nov. 22, 2017; U.S. Provisional Patent Application No. 62/589,833 filed Nov. 22, 2017 and U.S. Provisional Patent Application No. 62/589,838 filed Nov. 22, 2017 are incorporated herein by reference, in their entirety. Aspects of the implementations can be modified, if necessary, to employ systems, circuits and concepts of the various patents, applications and publications to provide yet further implementations.
These and other changes can be made to the implementations in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific implementations disclosed in the specification and the claims, but should be construed to include all possible implementations along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.
This application claims the benefit of priority to U.S. Provisional Application No. 62/770,038, filed Nov. 20, 2018, which application is hereby incorporated by reference in its entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2019/062034 | 11/18/2019 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62770038 | Nov 2018 | US |
