Patent Application
20210338185
The present invention relates to medical imaging. In particular, it relates to a medical imaging device, as well as a method of operating an medical imaging device, a computer program element, and a computer readable medium.
In medical imaging, timely communication of in particular a critical finding at examined subjects may be desired. There may even be cases where timely communication of such a finding is mandated by, for example, institutional or statutory requirements. By way of example, there may exist different categories of noticeable or actionable finding which may require recognition and/or communication within minutes, hours or days. Especially for the minute-based category, immediate action or at least immediate communication may be required in order to avoid deterioration or even mortality.
In addition, in medical imaging, a time elapsing between image acquisition and evaluation of the same by a radiologist, physician or the like may affect the overall time of treatment. Typically, radiologists, physicians etc. work through a reading list sequentially, or guided by a manual prioritization process. In particular for an unexpected finding, this process may cause unnecessary delays in capturing a finding.
US 2008/0091473 A1 describes a notification system for a medical imaging device, in which a notification is sent informing about the progress of the image processing or its completion.
WO 2017/136762 discloses a system for processing medical diagnostic images for review, e.g., by a physician or radiologist.
US 2014/358585 discloses an apparatus for implementing a medical critical results communication.
US 2009/028403 discloses a system for analyzing a source medical image of a body organ.
US 2009/196479 discloses a system for prioritizing medical imaging scans.
US 2016/350919 discloses a system for processing electronic imaging data obtained from medical imaging procedures.
There may be, therefore, a need to improve medical imaging in terms of loss of time. The object of the present invention is solved by the subject-matter of the appended independent claims, wherein further embodiments are incorporated in the dependent claims, in the accompanying drawings and the following description.
According to a first aspect, there is provided a medical imaging device, comprising:
The image acquisition unit may be adapted to acquire images of a region of interest of the subject to be imaged, and may in particular adapted to use imaging technologies of X-ray radiography, magnetic resonance imaging, computer tomography, ultrasound, or the like. Accordingly, it may interact with or comprise one or more of a processing unit, a data and/or image storage etc.
The local data processing device may be a suitable computing device, comprising one or more of a processing unit, a data and/or image storage etc. It may be arranged without a remote data connection but implemented “on system”, i.e. in close spatial proximity to the image acquisition unit, e.g. within a radiologist department.
The AI-module may be implemented by program instructions utilizing machine learning techniques, in particular deep learning techniques which may be trained for that purpose, or the like. Further, it may be adapted for image classification, which may allow automatic identification of one or more findings within the image data. The finding can be a medical, clinical, diagnostic and/or a anatomical finding. More specifically, the employed deep learning technique may be a convolutional neural network (CNN) consisting of one or more layers, e.g. convolutional layers, batch-normalization layers, dense layers, or the like, which may be optimized to process image data using techniques such as backpropagation. Further, according to some embodiments, the AI-module may be adapted to classify content of the image and to automatically detect the finding based on the classification. Optionally, the AI-module may be adapted to determine whether a further examination and/or treatment may be appropriate, wherein the determined additional examination and/or treatment is added to the notification data. In particular, the automatically detecting of a finding, in particular a medical finding, can be achieved with the AI-module. Therefore, the AI-module can comprise at least an input and an output layer. The AI-module can receive via the input layer the image data and outputs via the output layer the finding and/or the priority status of the finding. The AI-module is directed to a multi-label classification problem in order to process the image data from the input layer to the finding and priority status on the output layer. The images comprised by the image data, can be described with X={, . . . ,
N),
iεX. The images of the image data can be associated with a ground truth label
i, while the classification function
: X→Y is applied, which reduces a specific loss function 1 using N training sample-label pairs (
,
), I=1 . . . N. The label can be encoded for each image of the image data as a binary vector
ε{0,1}M=Y (with M labels). In addition, “no finding” can be encoded as an explicit additional label and hence have M=15 labels. Furthermore, following an initial calibration and/or investigation of weighting loss functions, e.g. positive/negative balancing, can be performed. In addition, a class-averaged binary cross entropy can be implemented such as:
Furthermore, ResNet-50 and DenseNet-121 architectures can be implemented. The AI-module can be trained and therefore the weight initialization strategies can be performed. This can be achieved with the help of random values and therefore the AI-module can be trained from the scratch. In addition, the AI-module can initiated with predefiend values from other AI-modules. Additionally, the transfer-learning approach can be implemented with the help of off-the-shelf (OTS) and fine-tuning (FT).
Furthermore the publication “Comparison of deep learning approaches for multi-label chest x-ray classification” Jan. 29, 2019 by Baltruschat el al. is incorporated herein by reference.
The determination of the priority status is based on the detected finding, in particular medical finding, of the AI-module. Based on the kind and/or the type of the detected finding, a priority status can be determined. The AI-module can detect the finding, which can be for example a pneumothorax. In addition, the local data processing device and/or the AI-module can determine the priority status by use of a look-up table. The look-up table may have on a input side a type of finding and on the output side a priority indicator, which specifies whether a notification should be issued or not. For example, the pneumothorax finding is inputted on the input side of the look-up table and the priority indicator is outputted, which indicates that a notification should be issued. Such a look-up table can be for example the ACR Appropriateness Criteria of the American College of Radiology.
In this description, the term “priority status” can generally be understood as a distinction as to whether a finding is critical and therefore, e.g. has to be treated within a very short time, or whether the finding need not be prioritized. Additionally, the “priority status” is indicative for an urgency of treatment of the identified finding. For example, the priority status may have a higher value corresponding to a critical finding and should therefore have a high priority, or may have a lower value that corresponding to a less critical finding and should therefore not have a high priority. Examples for a critical finding may comprise at least signs of a pneumothorax, an arterial dissection, or the like. Illustratively stated, a high priority status may mean a flag or special identification as a critical finding. In other words, the AI-module may be adapted for an automatic prioritization based on processing of the acquired image data.
The notification module may be implemented by program instructions, by electronic components or a combination thereof. It may be implemented within the data processing device that also has the AI-module. Alternatively, the notification module may be implemented in a further data processing device, or the like. In some embodiments, the notification module may comprise a communication data interface that enables a data connection via a data network, a telecommunications network, e.g. a cellular or mobile network, a radio network, or the like. In general, the notification module may be adapted to transmit and/or receive text messages, combined text-picture messages etc., which may also be provided as a push message. For example, the notification contains a text-based description of the finding which may be obtained from the AI-module. The notification can be about or can contain medical condition, medical situation and/or be indicative for a medical finding. The notification may be encrypted for data security.
In this description, the term “notification threshold” may be related to a pre-selection of critical finding, some of which may require the notification to be generated, but others may not, wherein the notification threshold makes these distinguishable. The critical finding may be included in a list, may be flagged etc.
An effect of this medical device is that a untimely communication and/or miscommunicated finding, in particular of a critical and/or time-sensitive finding, may be overcome. In more detail, an actionable finding may be detected directly during the image acquisitions process and not only after submission to a workstation or the like. Further, an automatic detection and prioritization of the finding may allow to inform a radiologist, physician etc. about the at least one potential finding in a timely manner, for example, within minutes. At the time of notification, the subject may, in the best case, not have yet left the diagnostic location, so that additional time to re-order the subject can be saved. Additional examinations, which may be advisable for the finding, can then be carried out promptly.
In an embodiment, the notification module is adapted to provide the notification data to a local display device of the imaging device.
The local display device may be, for example, a display of a system operator console which is operated and/or monitored during imaging by a technician.
Thus, at least the technician so that he can inform the radiologist about it. According to an embodiment, the notification module is adapted to provide the notification data to a first remote terminal.
The remote terminal may be a remoted but stationary device, e.g. a personal computer, or may be a portable device, e.g. a mobile phone, a tablet computer, pager etc. The remote terminal may be adapted to receive the notification data transmitted by the notification module via remote data transmission using a suitable communications protocol. Further, it may be adapted to employ different notification or messaging technologies using different communication paths, such as push-up notification, e-mail, short message service (SMS) etc.
Thus, radiologists or physicians may be notified actively about the presence of a critical finding, even if not monitoring a worklist, system operator console or the like. For example, they may be notified even if attending staff meetings, during break etc.
In an embodiment, the notification module is adapted to request an acknowledgment of receipt and/or a reading confirmation for the notification data from the first remote terminal.
The data connection may at least temporarily be bidirectional. The reading confirmation may be implemented in or provided by the messaging technology used for notification.
Thus, this may allow further subsequent actions, in particular if the notification cannot be delivered to the intended recipient. For example, the notification may be escalated to another recipient.
According to an embodiment, the notification module may be adapted, if the reading confirmation is not received within a certain time period, to notify the first remote terminal again via a second communication path that is different to a first communication path through which the reading confirmation is not received within the certain time period. The certain time period may be dependent from e.g. the detected finding, the notification threshold, or the like. Therefore, the certain time period may be minutes, hours etc. In this description, the term “communication path” may refer to a particular communication technology, like e-mail, SMS, or the like.
Thus, it may again be attempted to notify the intended recipient before the notification is escalated and a substitute recipient is called in.
In an embodiment, the notification module is adapted, if the reading confirmation is not received within a certain time period, to provide the notification to a second remote terminal.
Thus, the notification is escalated to another recipient to lose as little time as possible to communicate the detected finding.
According to an embodiment, the notification to at least the first remote terminal is logged in a log data record.
This may primarily serve documentation purposes, but also quality assurance.
According to an embodiment, the AI-module is adapted to determine a spatial location of the finding within the subject to be imaged, wherein the determined spatial location is added to the notification data.
The AI-module can be configured for identifying, classifying, receiving and/or determining one or more anatomical reference points, e.g. short-rips, within the image data. In addition, the AI-module can be configured for calculating a distance respectively a vector between the finding and the anatomical reference point, thereby determining the location of the finding within the subject. In an example, the AI-module can identify a short-rip as a reference point. Furthermore, the finding can be set in relation to the reference point and the spatial location within the subject can be calculated. The spatial location may be included text-based etc.
Thus, the radiologist or physician may be notified with a higher degree of information. This facilitates confirmation of the automatic detection performed by the AI-module.
In an embodiment, the AI-module is adapted to determine at least one image of the image data that represents at least a part and/or a partial view of the finding, and wherein the determined image is at least partly added to the notification data.
The notification may therefore be a combination of a text message and a picture message.
Thus, the radiologist or physician may be notified with a higher degree of information. This further facilitates a timely confirmation of the automatic detection performed by the AI-module.
According to an embodiment, the AI-module may be adapted to determine a likelihood of detection indicating the likelihood with which the detected finding has been correctly identified, and
wherein the determined likelihood of detection is added to the notification data.
The likelihood of detection may be determined by a machine learning algorithm or the like. For an intuitive perception, it may be indicated in percentage or the like. The AI-module can be configured for identifying a finding in the image data. In other words the AI-module can identify an anomaly in the image data and can then calculate the likelihood respectively plausibility, which kind of anomaly it is, resulting in the finding. By way of example, the AI-module may identify, e.g. by classification, an anomaly within the image data and calculates the likelihood of 80% that the anomaly is a pneumothorax and calculates the likelihood of 20% that the anomaly is a arterial dissection. The AI-module is configured for interpreting the likelihood and describes the anomaly as a pneumothorax, resulting in the finding. The notification can comprise the finding in this example pneumothorax, and the likelihood, 80%.
Thus, the radiologist or physician may be notified with a higher degree of information. This further facilitates a timely confirmation of the automatic detection performed by the AI-module.
In an embodiment, the AI-module may be adapted to perform detecting of the finding and/or providing the notification data exclusively through local data processing. By direct data processing and eliminating other data processing instances, the notification may be provided within a particularly short time period.
According to a second aspect, there is provided a medical imaging system, comprising:
The receiving device may, for example, be the system operator console of the medical imaging device. Alternatively or additionally, if more than one receiving devices shall be notified, the receiving device may be a remote terminal, e.g. a remoted, stationary or portable device.
In an embodiment, a further remote clinical system may be connected to an image acquisition unit of the medical imaging device,
wherein the medical imaging system may be adapted to perform a pre-processing of the image by a local data processing means and providing notification data to the receiving data prior to perform a main-processing of the image data by the clinical picture archiving and communication system.
The further clinical system may be a Picture Archiving and Communication System (PACS), EMR, or the like, using communication standards like DICOM or HL-7. Using direct data processing and eliminating other data processing instances, the notification may be provided within a particularly short time period.
According to a third aspect, there is provided method of operating an medical imaging device. The method may in particular be performed using a medical imaging device according to the first aspect. The method comprises:
acquiring image data of an subject to be imaged by an imaging device,
processing the acquired image data by an imaging device-sided artificial-intelligence-module, AI-module, of a local data processing device to automatically detect a finding in the subject,
determining a priority status of the detected finding by the AI-module, and
providing a notification, containing at least the finding, to a notification module, if the determined priority status reaches or exceeds a notification threshold.
According to a fourth aspect, there is provided a computer program element for operating a medical imaging device, which, when being executed by a processing unit, is adapted to perform the method according to the third aspect.
According to a fifth aspect, there is provided a computer-readable medium having stored the computer program element according to the fourth aspect.
These and other aspects of the present invention will become apparent from and elucidated with reference to the embodiments described hereinafter.
Exemplary embodiments of the invention will be described in the following with reference to the following drawings:
The figures are merely schematic representations and serve only to illustrate embodiments of the invention. Identical or equivalent elements are in principle provided with the same reference signs.
The medical imaging device 100 in this embodiment further comprises an X-ray detector 170 configured to detect X-rays which have passed through the subject 140 and in particular configured to detect an absorption-attenuated radiation after having passed through the examination region. In this embodiment, the radiation source 160 and the X-ray detector 170 are mounted to the gantry 120 and are arranged opposite each other, so that the X-ray detector 170 continuously receives X-rays from the radiation source 160. The X-ray detector 170 may comprise a two-dimensional array of detector elements, wherein other embodiments may be contemplated.
The medical imaging 100 further comprises one or more computational means, wherein in this embodiment mainly a local data processing device 180 will be described. In some embodiments, the data processing device 180 is connected to at least the X-ray detector and/or the radiation source 160 to control these and/or to at least obtain data therefrom, in particular from the X-ray detector 170. The data processing device 180 may also be formed by several subsystems, function modules or units, software modules or units, or the like (not further detailed here), and is configured to reconstruct an image of the subject 140 based on the X-rays detected by the X-ray detector 170, and in particular based on a plurality of acquired projection images of the subject 140. In this embodiment, the data processing device 180 comprises at least one processor 181, at least one memory 182 for storing image data and at least one memory 183 for storing one or more program elements.
The data processing device 180 further comprises an artificial intelligence module, AI-module, which for better illustration is denoted by reference sign 184. The AI-module 184 has a data connection to the processor 181 and/or the X-ray detector 170 to obtain image data already processed by the processor 181 or raw image data directly provided by the X-ray detector 170. The AI-module 184 further comprises one or more artificial neural networks which may use one or more layers, e.g. convolutional layers, batch-normalization layers, dense layers, backpropagation, or the like, and which may be in particular provided as a convolutional neural network (CNN) adapted to process image data. Further, according to some embodiments, the AI-module 184 may alternatively or additionally include a deep learning algorithm and/or classification means, such as a suitable classifier. The CNN, classification means etc. may be pre-trained with suitable training data sets. Additionally, such a training takes place during ongoing operation to further improve the detection result. The image data is provided to the AI-module 184 as an input variable. On this basis, the AI-module 184 is adapted to automatically detect a finding on basis of the acquired image data and to determine a priority status of the detected finding. Further, the AI-module 184 is adapted to determine a spatial location of the finding within the subject 140. Also, the AI-module 184 is adapted to determine at least one image of the image data that represents at least a partial view of the finding. The AI-module 184 is further adapted to determine a likelihood of detection indicating the likelihood with which the detected finding has been correctly identified. The likelihood may be indicated in percentage or another suitable unit of measurement.
Further referring to
As shown in
In a step S2, the AI-module 184 processes the acquired image data to automatically detect a finding in the subject 140.
In a step S3, the AI-module 184 determines the priority status of the detected finding.
In a step S4, the AI-module 184 provides a notification, containing at least the naming of the finding, to the notification module 190, if the determined priority status reaches or exceeds a notification threshold.
In an optional step S5, the notification module 190 transmits the notification as a notification message to one or more of the system operator console 192 and the remote terminal 193.
In another exemplary embodiment of the present invention, a computer program or a computer program element is provided that is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments, on an appropriate system.
The computer program element might therefore be stored on a computer unit, which might also be part of an embodiment of the present invention. This computing unit may be adapted to perform or induce a performing of the steps of the method described above. Moreover, it may be adapted to operate the components of the above-described apparatus. The computing unit can be adapted to operate automatically and/or to execute the orders of a user. A computer program may be loaded into a working memory of a data processor. The data processor may thus be equipped to carry out the method of the invention.
This exemplary embodiment of the invention covers both, a computer program that right from the beginning uses the invention and a computer program that by means of an up-date turns an existing program into a program that uses the invention.
Further on, the computer program element might be able to provide all necessary steps to fulfil the procedure of an exemplary embodiment of the method as described above.
According to a further exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is presented wherein the computer readable medium has a computer program element stored on it which computer program element is described by the preceding section.
A computer program may be stored and/or distributed on a suitable medium (in particular, but not necessarily, a non-transitory medium), such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.
However, the computer program may also be presented over a network like the internet and can be downloaded into the working memory of a data processor from such a network. According to a further exemplary embodiment of the present invention, a medium for making a computer program element available for downloading is provided, which computer program element is arranged to perform a method according to one of the previously described embodiments of the invention.
It has to be noted that embodiments of the invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims whereas other embodiments are described with reference to the device type claims. However, a person skilled in the art will gather from the above and the following description that, unless otherwise notified, in addition to any combination of features belonging to one type of subject matter also any combination between features relating to different subject matters is considered to be disclosed with this application. However, all features can be combined providing synergetic effects that are more than the simple summation of the features.
While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing a claimed invention, from a study of the drawings, the disclosure, and the dependent claims.
In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfil the functions of several items re-cited in the claims. The mere fact that certain measures are re-cited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
| Number | Date | Country | Kind |
|---|---|---|---|
| 18204589.8 | Nov 2018 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2019/078413 | 10/18/2019 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62749152 | Oct 2018 | US |
