Patent Application
20230214994
Embodiments described herein relate to systems and methods for estimating a difficulty metric of a medical image study. Some systems and methods use various machine learning models of an artificial intelligence (AI) system to estimate a difficulty metric of a medical image study, wherein the medical image study is assigned for review based on the difficulty metric.
A medical image study may include one or more medical images captured of a patient. An image study may also include information regarding the patient, image study information, order information, or a combination thereof. A healthcare provider, such as a radiologist, may receive the medical image study for review and generation of an associated report (e.g., with annotations, notes, finding, diagnoses, etc.).
Difficulty varies amongst medical image studies depending on the content of the information of the medical image study and other related information. Relative Value Units (RVUs) are a current measure for standardizing a difficulty level of various types of medical imaging studies. RVUs may be used to determine reimbursement for healthcare providers for different study types. However, embodiments described herein recognize that RVUs do not account for many factors that significantly affect the complexity and difficulty of a medical image study, such as studies with current and multiple priors, clinical findings, demographic information of a patient (e.g., age, gender, body mass index, etc.), study details (contrast or no contrast), etc. Additionally, embodiments described herein recognize that greater amounts of relevant priors increase the amount of work needed to review the medical image study, particularly when the priors include multiple findings or impressions that may be correlated with artificial intelligence and computer aided diagnosis findings in a current exam. Embodiments described herein also recognize that accurately measuring study difficulty or complexity is important for efficient workload balancing and medical image study distribution among healthcare providers.
Accordingly, embodiments described herein provide methods and systems for estimating a difficulty metric of a medical image study. The methods and systems can use models of an artificial intelligence (AI) system to learn patterns of study difficulty using factors of information of the medical image study, for example, such as medical image study information, information regarding a patient, information regarding a prior image study, etc. In particular, embodiments described herein can use ensemble methods to account for factors of information of the medical image study that RVUs do not consider. Ensemble methods are a machine learning technique that combines various machine learning models to produce a predictive performance that any of the various machine learning models alone cannot produce. In addition to removing the time required for manual workload distribution, using an AI system as described herein to assign medical image studies to healthcare providers provides a difficulty metric, which is more effective than RVUs, to estimate and balance workload of healthcare providers. Furthermore, as compared to simple rules-based assignment system, the machine learning model of the AI system as described herein can automatically adjust over time to changing parameters of medical image studies.
Accordingly, embodiments described herein use models of an artificial intelligence (AI) system to automatically assign medical image studies to a healthcare provider. For example, one embodiment provides a computer-implemented method for assigning a medical image study for review. The method includes receiving a plurality of labeled medical image studies, wherein each of the plurality of labeled medical image studies including a medical image study and a label representing a difficulty of the respective medical image study. The method also includes receiving, for each of the plurality of labeled medical image studies, one or more prior image studies of a patient associated with the respective labeled medical image study. The method further includes creating a set of training data including the plurality of labeled medical image studies and the one or more prior image studies received for each of the plurality of labeled medical image studies and training an artificial intelligence (AI) system using the set of training data. In addition, the method includes estimating, using the AI system as trained, a difficulty metric for an unlabeled medical image study based on the unlabeled medical image study and one or more prior image studies of a patient associated with the unlabeled image study and assigning the unlabeled medical image study for review based on the difficulty metric.
Another embodiment provides a system for assigning a medical image study for review. The system includes an electronic processor. The electronic processor is configured to receive a plurality of labeled medical image studies, wherein each of the plurality of labeled medical image studies including a medical image study and a label representing a difficulty of the respective medical image study. The electronic processor is also configured to receive, for each of the plurality of labeled medical image studies, one or more prior image studies of a patient associated with the respective labeled medical image study, create a set of training data including the plurality of labeled medical image studies and the one or more prior image studies received for each of the plurality of labeled medical image studies, and train an artificial intelligence (AI) system using the set of training data. The electronic processor is further configured to estimate, using the AI system as trained, a difficulty metric for an unlabeled medical image study based on the unlabeled medical image study and one or more prior image studies of a patient associated with the unlabeled image study and assign the unlabeled medical image study for review based on the difficulty metric.
Yet a further embodiment provides a non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, perform a set of functions. The set of functions include receiving a plurality of labeled medical image studies, wherein each of the plurality of labeled medical image studies including a medical image study and a label representing a difficult of the respective medical image study, and receiving, for each of the plurality of labeled medical image studies, one or more prior image studies of a patient associated with the respective labeled medical image study. The set of functions further include creating a set of training data including the plurality of labeled medical image studies and the one or more prior image studies received for each of the plurality of labeled medical image studies, training an artificial intelligence (AI) system using the set of training data, estimating, using the AI system as trained, a difficulty metric for an unlabeled medical image study based on the unlabeled medical image study and one or more prior image studies of a patient associated with the unlabeled image study, and assigning the unlabeled medical image study for review based on the difficulty metric.
Other aspects of the invention will become apparent by consideration of the detailed description and accompanying drawings.
Before any embodiments are explained in detail, it is to be understood that the embodiments are not limited in their application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. Other embodiments are capable of being practiced or of being carried out in various ways.
It should be understood that although certain drawings illustrate hardware and software located within particular devices, these depictions are for illustrative purposes only. In some embodiments, the illustrated components may be combined or divided into separate software, firmware and/or hardware. For example, instead of being located within and performed by a single electronic processor, logic and processing may be distributed among multiple electronic processors. Regardless of how they are combined or divided, hardware and software components may be located on the same computing device or may be distributed among different computing devices connected by one or more networks or other suitable communication links.
Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The terms “mounted,” “connected” and “coupled” are used broadly and encompass both direct and indirect mounting, connecting, and coupling. Further, “connected” and “coupled” are not restricted to physical or mechanical connections or couplings, and may include electrical connections or coupling, whether direct or indirect. Also, electronic communications and notifications may be performed using any known means including direct connections, wireless connections, etc.
A plurality of hardware and software-based devices, as well as a plurality of different structural components may be utilized to implement the embodiments. In addition, embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic-based aspects of the embodiments may be implemented in software (e.g., stored on non-transitory computer-readable medium) executable by one or more processors. As such, it should be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components, may be utilized to implement the embodiments. For example, “mobile device,” “computing device,” and “server” as described in the specification may include one or more electronic processors, one or more memory modules including non-transitory computer-readable medium, one or more input/output interfaces, and various connections (e.g., a system bus) connecting the components.
As described above, embodiments provided herein provide methods and systems for estimating a difficulty metric of a medical image study.
The information repository 110 stores medical data, including, for example, medical image studies. A medical image study may comprise a plurality of images captured of a patient using an imaging modality. For example, the information repository 110 may include a picture archiving and communication system (PACS) that stores various types of medical images. In some embodiments, the information repository 110 may also store other medical data such as patient information, reports for prior exams, pathology reports or results, or the like. For example, in some embodiments, the information repository 110 may include an electronic medical record (EMR) system, hospital information system (HIS), a radiology information system (RIS). In some embodiments, the information repository 110 may also be included as part of the server 105. Also, in some embodiments, the information repository 110 may represent multiple servers or systems, such as for example, a PACS, an EMR system, a RIS, and the like. Accordingly, the server 105 may be configured to communicate with multiple systems or servers to perform the functionality described herein. Alternatively or in addition, the information repository 110 may represent an intermediary device configured to communicate with the server 105 and one or more additional systems or servers (e.g., a PACS, an EMR system, a RIS, etc.). Accordingly, the medical data stored in or accessible through the information repository 110 can include patient information, images, reports of findings, pathology reports or results, EMR information, historical reading times of the medical image studies, relative value units (RVUs), etc.
In some embodiments, the patient information stored in or accessible through the information repository 110 can include information such as demographic information, procedure history, disease history, etc. related to a specific patient.
The images stored in the information repository 110 are generated by an imaging modality (not shown), such as an X-ray, a computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, or the like. In some embodiments, the information repository 110 may also be included as part of an imaging modality. The images stored in the information repository 110 may be grouped into image studies. In some embodiments, images within an image study are generated by the same image modality (not shown) for a patient. In addition to one or more medical images, an image study can include metadata. The metadata may include study description, number of series/slices, an imaging modality type or identifier, and patient information. The metadata may be defined according to one or more standards for communicating medical data, such as, for example, the digital imaging and communications in medicine (DICOM) standard, the health level seven (HL7) standard, or the like.
Reports or findings stored in or accessible through the information repository 110 can include reports or findings automatically generated by one or more systems, such as, for example, one or more computer-aided diagnosis (CAD) systems, artificial intelligence systems, or the like. Alternatively or in addition, the reports and findings can include electronic reports or findings generated by a radiologist or other healthcare professional, such as for, example, an image study report, a pathology report, or the like. For example, a radiologist may use a RIS to create an electronic report for an image study, wherein the report includes findings or impressions, one or more diagnoses, annotations, measurements, or the like. Metadata regarding such reports or findings can also be stored in or accessible through the information repository 110. For example, timing information relating to completion of an image study report can be stored, which may represent how long it took a radiologist to read an image study and create the associated report. Similarly, other information relating to how a report was generated can be stored, such as, for example, what images (or what number of images) were reviewed as part of creating a report, what or what number of prior reports were reviewed as part of creating a report, or the like.
As illustrated in
The electronic processor 130 may be, for example, a microprocessor, an application-specific integrated circuit (ASIC), and the like. The electronic processor 130 is generally configured to execute software instructions to perform a set of functions, including the functions described herein. The memory 135 includes a non-transitory computer-readable medium and stores data, including instructions executable by the electronic processor 130. The communication interface 140 may be, for example, a wired or wireless transceiver or port, for communication over the communication network 115 and, optionally, one or more additional communication networks or connections.
As illustrated in
For example,
Server 105 receives completed medical image studies and associated medical information from the information repository 110 to train the difficulty model 145. The difficulty model 145 may be trained, for example, by a supervised learning method using labeled medical image studies to estimate a difficulty metric of a received unlabeled medical image study. A supervised learning method is a machine learning task that learns a function that maps an input to an output based on a set of input-output pairs. The set of input-output pairs (e.g., a set of training data) may include a medical image study (i.e., input) tagged with one or more labels and a difficulty metric (i.e., output). For example, an expert can provide a label (e.g., a numerical scale, such as from 1-10) for a medical image study that represents a difficulty of an image study. Alternatively or in addition, a label for a medical image study can be automatically assigned based on a predicted reading time associated with historical reading times of comparable medical image studies (e.g., excluding outliers).
The information used to train the difficulty model 145 (also referred to herein as the “training data”) can include a plurality completed or labeled (i.e., reviewed and assigned a difficulty label or score) image studies (images included in each study) and associated additional information. For example, the training data used to train the difficulty model 145 can include a plurality of factors that impact study difficulty and, thus, result in a more accurate model 145 that considers multiple factors that can influence the difficulty of a medical image study (i.e., in addition to the images themselves included in the medical image study needing to be assigned and analyzed). This additional information received and included in the training data can include, for example, study information (e.g., a study description, a number of series or slices, a modality type, a number of prior studies, a total number of images in prior studies, an imaging protocol used, or the like), patient information (e.g., demographic information, disease history, etc.), one or more prior image studies, one or more exam reports (e.g., a report for the image study, reports for prior image studies, findings, impressions, annotations, pathology reports, etc.), CAD or other AI findings in the image study or prior image studies, reading time information for the image study, an RVU assigned to the image study, or a combination thereof. Accordingly, rather than simply training a model to estimate a difficulty metric for an image study based on a set of image studies and associated labels, which may not represent all factors that make one image study more complex or difficult to analyze than another image study, embodiments described herein train the model using additional relevant information, such as, for example, patient demographic information and medical history information. In particular, a number of prior image studies associated with a patient can impact the difficulty in analyzing a new image study for the patient. For example, when a patient has only a single prior image study available, analyzing a new image study for this patient may be less complex than when the patient has multiple prior image studies available. If a model is not trained with information regarding prior image studies (e.g., numbers, types, findings, progressions, etc.), a model cannot take this factor into account. Thus, by incorporating prior image studies in the training data, embodiments described herein provide more accurate difficulty metrics than other systems, which results in a more balanced distribution and workload and more accurate medical reports and findings. For example, by including prior image studies in the training data, the models described herein may use the number of prior image studies, the types of prior image studies, a timing of prior image studies, findings in prior image studies (e.g., how a lesion or area of interest has changed over time between one or more prior studies), or the like to output an improved difficulty metric for an unlabeled image study that results in improved user efficiency as well as computing resource efficiency (e.g., given a more accurate initially-assigned metric and associated radiologist assignment).
For example, to assign a difficulty metric to an unlabeled medical image study associated with a patient (referred to herein as a “current patient” to distinguish from other patients associated with training data and labeled medical image studies) some embodiments described herein receive prior image study information of the current patient, prior exam information of the current patient, and current exam information of the unlabeled medical image study. The prior image study information may include a number of prior image studies associated with the current patient, a number of image series in a prior image study associated with the current patient, and a total number of images in the prior image studies associated with the current patient. The prior exam information may include findings and impressions in prior exam reports associated with the current patient, and the current exam information may include computer-aided diagnosis (CAD) results of the unlabeled medical image study. The difficulty model 145 uses this information in combination with the unlabeled medical images study itself (and optionally additional information as described above) to estimate a difficulty metric for the unlabeled medical image study. In particular, the difficult model 145 can be trained using training data that includes similar data (prior image study information, prior exam information, and CAD results) for labeled medical image studies to correlate prior image study, exam reports, and CAD results to associated difficulty metrics and, thus, recognize the fact that higher number of prior image studies the more work generally needed to review an image study, especially if there are multiple findings or impressions that could be correlated with CAD results (including AI-driven results) in the image study being assigned a difficulty metric.
As shown in
For example, in some embodiments, the difficulty model 145 receives training data A 310 and training data B 320 to train the model A 145-1 and the model B 145-2, respectively, wherein each model, once trained, is configured to output a respective difficulty metric (e.g., a difficult sub-metric) for a medical image study. The combiner 145-3 is configured to generate a respective difficulty metric for the medical image study based on the output of the model A 145-1 and the model B 145-2 (e.g., combining the sub-metrics, averaging the sub-metrics, or the like). As illustrated in
The patient information 330 may include, for example, demographic information such as a gender, age, weight, medical condition, ethnicity, geographic location, or the like, or a combination thereof. Additionally, the patient information 330 may include, for example, disease history, such as abnormal condition of a part, organ, or system of a patient resulting from various causes, such as infection, inflammation, environmental factors, or genetic defects.
The medical records 332 may include, for example, medical records such as prior reports, findings/impressions, annotations, pathology reports, pathology results, computer aided diagnosis (CAD) or other artificial intelligence (AI) findings in current and prior exams. Additionally, the medical records 332 may include, for example, medical image study descriptions that identify the purpose of the study, type of data collected, and/or how the collected data will be used.
In contrast, the training data B 320 includes the patient information 330, the medical records 332, and images 334 that correspond to the plurality of labeled medical image studies of the information repository 110. The images 334 may include, for example, a study description, number of series images/slices, modality, number of priors, imaging protocol, image volume, relative pathological findings from prior reports images, annotations, biopsies, etc. Additionally, the images 334 may include metadata, such as lesion findings and findings of lesion complexity (e.g., number, size, shape, mass, calcification, etc.) of current and prior images of the images 334.
The model A 145-1 may be, for example, a machine learning model for estimating relationships between a dependent variable (e.g., difficulty metric, procedure information, etc.) and one or more independent variables, such as the patient information 330 and the medical records 332. In some embodiments, the model A 145-1 is configured to use regression analysis using the patient and procedure information for the labeled image studies. For example, the difficulty model 145 utilizes the model A 145-1 to identify causal relationships between a dependent variable and a collection of independent variables in a fixed dataset, such as medical study information (e.g., the patient information 330 and the medical records 332) of a medical image study.
The model B 145-2 may be, for example, a sequence machine learning model for estimating an output (e.g., difficulty metric, medical image study complexity, etc.) based on a sequence of data inputs, such as the patient information 330, the medical records 332, and the images 334. The model B 145-2 may be, for example, a recurrent neural network (RNN), temporal convolutional network (TCN), long-short term memory (LSTM), or any other machine learning model capable of analyzing time-series data. For example, the difficulty model 145 utilizes the model B 145-2 to identify causal relationships between a complexity of a medical image study (e.g., output, difficulty metric, etc.) and time series data (e.g., the patient information 330, the medical records 332, and the images 334) corresponding to a patient of the medical image study.
As noted above, the combiner 145-3 is configured to generate a final or overall difficulty metric for a medical image study based on the output of the model A 145-1 and the model B 145-2. For example, the combiner 145-3 may, for example, determine a sum (e.g., difficulty metric) of respective outputs of the model A 145-1 and the model B 145-2. In another example, the combiner 145-3 may, for example, determine an average (e.g., difficulty metric) of respective outputs of the model A 145-1 and the model B 145-2. Other pooling, stacking, and boosting algorithms can be used by the combiner 145-3 in various embodiments.
While
After the difficulty model 145 (i.e., the models 145-1 and 145-2) are trained using the labeled image studies and associated information, the model 145 can be used to assign a difficulty metric to an unlabeled image study needing review. For example,
While
In an example embodiment,
The difficulty metric assigned to the medical image study needing review can be used to assign the medical image study to a healthcare provider (e.g., a radiologist) and, in particular, can be used to provide a more balanced distribution of image studies needing review. For example,
The method 400 includes creating a set of training data including the labeled medical image study information (at block 410). For example, the server 105 may utilize a plurality of received medical image studies uploaded to the information repository 110 to create a labeled set of data that may include information, such as input-output pairs, in memory 135. The input-output pairs may include a set of features of a medical image study (e.g., input) and difficulty metric corresponding to the set features (e.g., output). As noted above, the labels (i.e., the difficulty metrics) may be defined manually by an expert or determined based on reading information.
The method 400 includes training an artificial intelligence system using the set of training data (at block 415). For example, the server 105 inputs a created labeled set of data into the difficulty model 145. In some embodiments, the server 105 reserves a segment of the plurality of received medical image studies uploaded to the information repository 110 to create a test set of data, which qualifies performance of the difficulty model 145. For example, as training the difficulty model 145 with an initial set of data, the server 105 inputs the test set of data into the difficulty model 145 to determine an accuracy of the difficulty model 145. In some embodiments, the server 105 may iteratively input labeled set of data and the test set of data into the difficulty model 145 until performance of the difficulty model 145 reaches a target accuracy. As also noted above, in some embodiments, the difficulty model 145 includes multiple (e.g., two) models, wherein each model can be trained using a particular subset of the training data.
The method 400 also includes, after training the difficulty model 145, receiving an unlabeled medical image study (at block 420). For example, an unlabeled medical image study may be uploaded to the information repository 110. In this example, the server 105 can use information included in the uploaded image study to access or receive associated medical information regarding the unlabeled medical image study, such as from the information repository 110, other data sources, or a combination thereof. Again, as noted above, the associated information can include patient information, procedure information, prior image studies, reports associated with prior image studies, pathology reports, CAD or AI results for the prior image studies, the unlabeled image study, or combinations thereof. It should be understood that the type of information used to estimate a difficulty metric for an unlabeled image study via the difficulty model 145 is similar to the data used to train the difficulty model 145 (e.g., the same type of data with the exception of a label).
As illustrated in
The method 400 further includes assigning the unlabeled medical image study for review based on the estimated difficulty metric (at block 430). For example, the server 105 assigns an unlabeled medical image study to an identifier of a care provider in a worklist table stored in the memory 135. In this example, the server 105 may assign the unlabeled medical image study to a care provider with an available status based on the worklist table. In another example, the server 105 receives a total workload (e.g., a cumulative difficulty metric for a care provider) from a worklist table stored in the memory 135 for each care provider working within the system 100. In this example, the server 105 may assign the unlabeled medical image study to a care provider using the total workload for each care provider and a determined difficulty metric for the unlabeled medical image study. Also, the server 105 may assign the unlabeled medical image study to adhere to a set of parameters, such as a cumulative difficulty metric threshold, an average total workload of care providers of the worklist table, etc. to balance workloads. In some embodiments, the method 400 includes transmitting a received unlabeled medical image study to a workstation of a care provider. For example, the processor 130 may route the unlabeled medical image study to workstation 120 of a care provider using updated information (e.g., assignment information) of a worklist table stored in the memory 135.
Accordingly, embodiments described herein account for the many factors that can contribute to the difficulty of reviewing a medical image study, including whether a current study has multiple prior studies and prior findings or reports and patient information. Using artificial intelligence allows embodiments described herein to learn patterns of study difficulty taking into account these factors, which allows for more accurate difficulty metrics and, consequently, more balanced workload distribution among radiologists.
Various features and advantages of the embodiments are set forth in the following claims.
