Patent Application
20240386692
This application claims priority under 35 U.S.C § 119 (a) to Japanese Patent Application No. 2023-080678 filed on 16 May 2023. The above application is hereby expressly incorporated by reference, in its entirety, into the present application.
The present invention relates to a medical image processing device, a method of operating the medical image processing device, a non-transitory computer readable medium, and an endoscope system.
In a medical field, time-series medical images that are acquired by an endoscope system, an ultrasound diagnostic system, or the like and have been subjected to various types of image processing are used in a case where a medical doctor observes or diagnoses medical images. A medical image processing device disclosed in WO2020/162275A (corresponding to US2021/0343011A1) acquires region information indicating a region in a living body in an acquired medical image, and performs recognition processing, such as the detection of a region of interest, on the medical image. Any one of a plurality of trained models corresponding to the region information trained with deep learning or the like is used in the recognition processing. These trained models are models trained with image sets in which each region in the living body is imaged depending on a target region. Further, WO2020/162275A discloses that a display device is caused to display a result of detecting a region of interest in an aspect corresponding to an imaged region.
A processor device (medical image processing device) disclosed in JP2022-137276A also recognizes an imaged region of a living body shown in an acquired medical image, and further performs recognition processing for acquiring information corresponding to the imaged region on the medical image. Trained models, which are trained with deep learning or the like, are used for these types of recognition processing.
As disclosed in WO2020/162275A and JP2022-137276A, the trained models used for the recognition processing and the like corresponding to the region information are models trained with an image set including medical images in which a target region in a living body is imaged. In a case where acquired region information is incorrect or a medical doctor who is a user inputs region information by mistake, recognition processing is performed by a trained model based on an image set including medical images of a region different from the actual region. In a case where the recognition processing is performed by a trained model based on an image set of a region different from the actual region as described above, false detection may frequently occur, or an unstable behavior that detects a region that is actually not a region of interest as the region of interest or detects an actual region of interest as a region that is not the region of interest, and the like may occur.
An object of the present invention is to provide a medical image processing device that can prevent false detection in detection processing corresponding to region information to perform stable detection, a method of operating the medical image processing device, a non-transitory computer readable medium, and an endoscope system.
A medical image processing device according to an aspect of the present invention comprises a processor. The processor is configured to: acquire a medical image in which a subject is imaged; classify a region of the subject of the medical image into a first region or a second region; and detect a region of interest from the medical image using a first trained model obtained from machine learning with a first data set including data of the first region and data of the second region with regard to the medical image of which the region of the subject is classified into the first region, and detect a region of interest from the medical image using a second trained model obtained from machine learning with a second data set including the data of the second region with regard to the medical image of which the region of the subject is classified into the second region. The first data set includes the data of the first region more than the data of the second region.
It is preferable that the first trained model has detection accuracy of the region of interest in the first region higher than the detection accuracy of the region of interest in the second region, and the second trained model has the detection accuracy of the region of interest in the second region higher than detection accuracy of the region of interest in the first region.
It is preferable that the first data set includes an image including the region of interest and an image not including the region of interest that correspond to the data of the first region and an image not including the region of interest that corresponds to the data of the second region.
It is preferable that the first trained model or the second trained model is the trained model trained with data and/or the first data set or the second data set having different combinations of the data.
It is preferable that the processor analyzes the medical image to acquire a classification result indicating the region of the subject in which the medical image is captured, and selects the first trained model or the second trained model corresponding to the classification result.
It is preferable that the processor acquires a classification result indicating the region of the subject in which the medical image is captured on the basis of a user's input, and selects the first trained model or the second trained model corresponding to the classification result.
It is preferable that the processor performs a control of notifying a user of a detection result obtained from the first trained model or the second trained model. It is preferable that the processor performs a display control of causing a medical image display to display the acquired time-series medical images, and causes the medical image display to display the medical images in which the region of interest is detected such that the detection result is superimposed on the medical images, for the notification.
It is preferable that the processor notifies a user of not only the detection result but also a selection state of the first trained model or the second trained model. It is preferable that the machine learning is deep learning.
An endoscope system according to another aspect of the present invention comprises an endoscope that captures the medical image; and the medical image processing device.
A method of operating a medical image processing device according to still another aspect of the present invention comprises: a step of acquiring a medical image in which a subject is imaged; a step of classifying a region of the subject in which the medical image is captured into a first region or a second region; a step of detecting a region of interest from the medical image using a first trained model obtained from machine learning with a first data set including data of the first region and data of the second region with regard to the medical image of which the region of the subject is classified into the first region; and a step of detecting a region of interest from the medical image using a second trained model obtained from machine learning with a second data set including the data of the second region with regard to the medical image of which the region of the subject is classified into the second region. The first data set includes the data of the first region more than the data of the second region.
A non-transitory computer readable medium storing a computer-executable program according to yet another aspect of the present invention causes a computer to execute: processing for acquiring a medical image in which a subject is imaged; processing for classifying a region of the subject in which the medical image is captured into a first region or a second region; and processing for detecting a region of interest from the medical image using a first trained model obtained from machine learning with a first data set including data of the first region and data of the second region with regard to the medical image of which the region of the subject is classified into the first region; and processing for detecting a region of interest from the medical image using a second trained model obtained from machine learning with a second data set including the data of the second region with regard to the medical image of which the region of the subject is classified into the second region. The first data set includes the data of the first region more than the data of the second region.
According to the present invention, it is possible to prevent false detection in detection processing corresponding to region information to perform stable detection.
As shown in
The endoscope system 10 comprises an endoscope 12, a light source device 13, a processor device 14, a display 15, a processor device-side input device 16, the medical image processing device 20, a medical image display 21, and a medical image processing device-side input device 22. The endoscope 12 is optically or electrically connected to the light source device 13 and is electrically connected to the processor device 14. The medical image processing device 20 is electrically connected to the light source device 13 and the processor device 14.
The endoscope 12 includes an insertion part 12a, an operating part 12b, a bendable part 12c, and a distal end part 12d. The insertion part 12a is inserted into the body of an examinee. The operation part 12b is provided at a proximal end portion of the insertion part 12a. The bendable part 12c and the distal end part 12d are provided on a distal end side of the insertion part 12a. The bendable part 12c is operated to be bent in a case where angle knobs 12e of the operation part 12b are operated. As the bendable part 12c is operated to be bent, the distal end part 12d is made to face in a desired direction.
An imaging optical system for forming an image of a subject and an illumination optical system for irradiating the subject with illumination light are provided in the endoscope 12. The operation part 12b is provided with angle knobs 12e, a mode selector switch 12f, a still image acquisition instruction switch 12h, and a zoom operation part 12i. The mode selector switch 12f is used for an operation for switching an observation mode. The still image acquisition instruction switch 12h is used for an instruction for acquiring a still image of the subject. The zoom operation part 12i is used for an operation for enlarging or reducing an object to be observed. The operation part 12b may be provided with a scope-side input device 19, which is used to perform various operates on the processor device 14, in addition to the mode selector switch 12f and the still image acquisition instruction switch 12h.
The light source device 13 generates illumination light. The processor device 14 performs a system control on the endoscope system 10 and further performs image processing or the like on image signals transmitted from the endoscope 12 to generate medical images, and the like. The processor device 14 transmits the medical images to the medical image processing device 20 in addition to the display 15. The display 15 displays an operation image and the like in addition to the medical images transmitted from the processor device 14. The processor device-side input device 16 includes a keyboard, a mouse, a foot switch, a touch pen, and the like, and receives an input operation, such as function settings made by a user.
The medical image processing device 20 is connected to the medical image display 21. The medical image processing device 20 receives the medical images transmitted from the processor device 14, and performs various types of processing on the basis of the received medical images. The medical image processing device 20 transmits results of the various types of processing to the medical image display 21 in addition to the medical images. The medical image display 21 displays the medical images and the like transmitted from the medical image processing device 20.
The endoscope system 10 has a normal mode, a region-of-interest detection mode, and a region classification mode. In the normal mode, a color normal image obtained in a state where an object to be observed is irradiated with white light is displayed on the display 15 as the medical images. In the region classification mode, region classification processing is performed on the medical images and detector selection processing for selecting a region-of-interest detector corresponding to a classification result is performed. Further, in the region classification mode, the types and the like of classified regions are displayed on the medical image display 21 as necessary. In the region-of-interest detection mode, in a case where region-of-interest detection processing is performed on the medical images and a region of interest is detected, the medical image display 21 displays that the region of interest is present.
The normal mode, the region-of-interest detection mode, and the region classification mode described above can be set by the medical image processing device-side input device 22. With regard to the setting of the mode, two or more modes can also be simultaneously executed (for example, all the region classification mode and the region-of-interest detection mode can also be executed) in addition to the sequential switching of the modes. The details of the region classification mode and the region-of-interest detection mode to be executed by the medical image processing device 20 will be described later.
As shown in
The medical image processing device-side input device 22 is an input device having the same configuration as the above-mentioned processor device-side input device 16, that is, including a keyboard, a mouse, a foot switch, and the like. The present invention is not limited thereto, and a touch panel provided in the medical image display 21, the scope-side input device 19 of the endoscope 12, and/or the like may be included in the medical image processing device-side input device 22. Alternatively, the processor device-side input device 16 and the medical image processing device-side input device 22 may be formed of a common input device, and the operation of the processor device 14 and the operation of the medical image processing device 20 may be switched and performed depending on the purpose of use or the situation of use.
A computer forming the medical image processing device 20 receives an input of various instructions from the medical image processing device-side input device 22. The medical image processing device 20 receives the medical images transmitted from the processor device 14, and performs various types of processing, such as region classification processing or region-of-interest detection processing, on the basis of the received medical images. The medical image processing device 20 transmits results of the various types of processing to the medical image display 21 in addition to the medical images. The medical image display 21 may display various operation screens corresponding to the operation of the medical image processing device-side input device 22 in addition to the medical images acquired from the processor device 14. For example, the operation screen has operation functions using a graphical user interface (GUI). The computer forming the medical image processing device 20 can receive an input from a user via the operation screen.
The controller 25 includes a central processing unit (CPU) 31 that is a processor, a random access memory (RAM) 32, a read only memory (ROM) 33, and the like. The CPU 31 loads a program stored in the storage unit 27 or the like into the RAM 32 or the ROM 33 and executes processing corresponding to the program to generally control each part of the computer. The communication unit 26 is a network interface that controls the transmission of various types of information via a network 30. The RAM 32 or the ROM 33 may have a function of the storage unit 27.
The storage unit 27 is an example of a memory and is, for example, a hard disk drive or a solid-state drive that is built in the computer forming the medical image processing device 20 or is connected to the computer via a cable or the network 30, or a disk array in which a plurality of hard disk drives or the like are continuously mounted. A control program, various application programs, various data to be used for these programs, display data of various operation screens attached to these programs, and the like are stored in the storage unit 27.
The storage unit 27 of the present embodiment stores various data, such as a program 34 for a medical image processing device and data 35 for a medical image processing device. The program 34 for a medical image processing device or the data 35 for a medical image processing device are a program or data that are used to realize various functions of the medical image processing device 20. The functions of the medical image processing device 20 are realized by the program 34 for a medical image processing device and the data 35 for a medical image processing device. The data 35 for a medical image processing device include, for example, a temporary storage section 35a that is a region in which data are temporarily stored and a data storage section 35b in which data and the like of the program are stored.
The computer forming the medical image processing device 20 may be a general-purpose server device, a personal computer (PC), or the like in addition to a device designed exclusively. Further, it is sufficient that the functions of the medical image processing device 20 can be exhibited, the computer may also be shared with devices having other functions, or the functions of the medical image processing device 20 can also be incorporated into a processor device for an endoscope, a medical information management device, and/or the like. In the medical image processing device 20, programs relating to medical image processing are stored in the storage unit 27 that is a memory for a program.
As shown in
The medical image acquisition unit 41 sequentially acquires medical images from the processor device 14 to acquire a medical image in which a subject is imaged. In the present embodiment, the medical images acquired from the processor device 14 by the medical image acquisition unit 41 are a plurality of time-series medical images. In the present embodiment, an endoscopic image obtained by the endoscope 12 is acquired as the medical image. However, the medical image is not limited to an endoscopic image, and other images, such as an ultrasonic image, a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, and the like, may be received.
The region classification unit 42 analyzes the medical images acquired by the medical image acquisition unit 41 to perform region classification processing for classifying a region of a subject in which the medical images are captured. The region-of-interest detection unit 43 performs detector selection processing for selecting a region-of-interest detector and region-of-interest detection processing for detecting a region of interest from the medical image with the selected region-of-interest detector, on the basis of the region classification processing performed by the region classification unit 42. Specific contents of the region classification processing, the detector selection processing, and the region-of-interest detection processing will be described later.
It is preferable that the region-of-interest detection processing or the region classification processing is processing performed by a trained model obtained from, for example, deep learning using a neural network (NN), a convolutional neural network (CNN), a recurrent neural network (RNN), or the like. Alternatively, learning is not limited to the deep learning, and the region-of-interest detection processing or the region classification processing may be processing performed by a trained model obtained from other machine learning, for example, learning using Adaboost or random forest.
A region of interest detected with the region-of-interest detection processing by the region-of-interest detection unit 43 is, for example, a region including a lesion area typified by a cancer, a treatment trace, a surgical scar, a bleeding site, a benign tumor area, and an inflammation area (including a portion with changes, such as bleeding or atrophy, in addition to so-called inflammation). That is, a region including a lesion, a region having a possibility of a lesion, or a region which is required to be observed in detail regardless of a possibility of a lesion, such as a dark region (the back of folds, a region where observation light is difficult to reach due to the deep lumen), or the like may be a region of interest. In the region-of-interest detection processing, a region including at least one of a lesion area, a benign tumor area, or an inflammation area is detected as the region of interest.
Further, in the region classification processing performed by the region classification unit 42, a region of a subject in which medical images 50 are captured is classified into a first region or a second region. In the region classification unit 42, a trained model, which has been trained with medical images as described above and generated, is used to recognize a plurality of preset regions of the subject including the first region and the second region. Then, in the region classification processing, the trained model is used to determine whether or not the first region or the second region is included in the medical images. In the region classification processing, the plurality of regions of the subject that are objects to examined are set depending on the purpose of an examination, or the like.
As shown in
The present invention is not limited thereto, and an examination made by the endoscope system 10 may be, for example, the examination of another portion of the subject, such as the examination of a lower gastrointestinal tract. In the case of the examination of a lower gastrointestinal tract, for example, a large intestine and a small intestine are set as a plurality of preset regions of the subject. Further, in the region classification processing, smaller regions may be set as the preset regions of the subject. For example, in a case where the inside of the stomach is examined, a dome portion, an upper gastric body, a middle gastric body, a lower gastric body, a cardiac region, the stomach corner, and the like may be set as the preset regions of the subject. Further, in a case where the inside of the large intestine is examined, a rectum, a sigmoid colon, a descending colon, a transverse colon, and the like may be set as the preset regions of the subject.
As shown in
The region-of-interest detection unit 43 includes a lesion detector 43A for an esophagus, a lesion detector 43B for a stomach, and a lesion detector 43C for a duodenum that are a plurality of region-of-interest detectors. The lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, and the lesion detector 43C for a duodenum correspond to the esophagus 101, the stomach 102, and the duodenum 103 that are a plurality of preset regions of the subject, respectively.
The region-of-interest detection unit 43 acquires the classification result obtained from the region classification unit 42, and selects any one of the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, or the lesion detector 43C for a duodenum corresponding to the classification result. That is, in a case where the classification result of the classification of the medical images 50 performed by the region classification unit 42 is the esophagus 101, the region-of-interest detection unit 43 selects the lesion detector 43A for an esophagus. Likewise, in a case where the classification result is the stomach 102, the region-of-interest detection unit 43 selects the lesion detector 43B for a stomach. Likewise, in a case where the classification result is the duodenum 103, the region-of-interest detection unit 43 selects the lesion detector 43C for a duodenum. In this way, in a case where the region of the subject is classified into the first region, the region-of-interest detector to be selected by the region-of-interest detection unit 43 corresponds to a “first trained model”. Likewise, in a case where the region of the subject is classified into the second region, the region-of-interest detector to be selected by the region-of-interest detection unit 43 corresponds to a “second trained model”.
The region-of-interest detection unit 43 performs the region-of-interest detection processing for detecting a region of interest from the medical images 50 using any one of the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, or the lesion detector 43C for a duodenum selected in the detector selection processing. Information on the detected region of interest and the medical images 50 in which the region of interest is detected are input to the display controller 44. The display controller 44 performs a display control of causing the medical image display 21 to display the medical images 50 in a time series. Further, the display controller 44 notifies a user of a detection result in a case where the region of interest is detected.
The display controller 44 performs a display control of causing the medical image display 21 to display the medical images 50 that are acquired in a time series and subjected to the region classification processing and the region-of-interest detection processing, and performs a control of notifying a user of the detection result of the region of interest. It is preferable that the display controller 44 notifies a user of the detection result during an examination. Specifically, in a case where the time-series medical images 50 are displayed on the medical image display 21, the notification controller 45 causes the medical image display 21 to display the medical images 50 in which the region of interest is detected such that the detection result of the region of interest is superimposed on the medical images 50.
For example, in a case where a region 55 of interest is detected from the medical image 50 by the region-of-interest detection unit 43 as shown in
Further, the notification of the detection result using the display controller 44 is not limited thereto, and a portion including the region 55 of interest in the medical image 50 displayed on the main screen 51 may be highlighted in a color different (a color having different shading, brightness, chroma saturation, or the like) from an actual color or a figure may be superimposed and displayed on a portion including the region 55 of interest or on the periphery of the region 55 of interest.
The notification controller 45 performs a control of notifying a user of the detection result of the region of interest with a method different from a method of displaying the medical image on the medical image display 21. For example, the notification controller 45 controls the speaker 29, and plays a notification sound from the speaker 29 in a case where the region of interest is detected from the medical image 50 by the region-of-interest detection unit 43. Further, for example, a notification sound is played from the speaker 29 according to highlighting for the notification of the detection result in the above-mentioned display controller 44.
The learning unit 46 generates the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, and the lesion detector 43C for a duodenum described above using deep learning. The lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, and the lesion detector 43C for a duodenum are trained models that are obtained from deep learning performed by the learning unit 46. A configuration in which the region-of-interest detectors included in the region-of-interest detection unit 43 are generated by deep learning is not limited thereto, and trained models, which are generated in a case where the same deep learning as the deep learning performed by the learning unit 46 is performed by a device provided outside the medical image processing device 20, may be acquired and may be used as region-of-interest detectors. For example, an external cloud server, which is connected via the network 30, or the like may be caused to execute the same functions as the learning unit 46, and trained models, which have been subjected to deep learning by a learning unit on this cloud server, may be acquired by the medical image processing device 20 and may be used as a plurality of region-of-interest detectors included in the region-of-interest detection unit 43.
As shown in
The contents of the “first region” and the “second region” change depending on which one of the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, and the lesion detector 43C for a duodenum corresponds to the first trained model. Specifically, in a case where the first trained model is the lesion detector 43A for an esophagus, the “first region” is a region to which the lesion detector 43A for an esophagus corresponds, that is, the esophagus and the “second region” is a region of the plurality of preset regions of the subject except for the first region, that is, the stomach or the duodenum.
As shown in
Further, the second region may be the duodenum in the above-mentioned specific example. In this case, the data D2 of the second region are the data of a medical image in which the duodenum is imaged, and the second data set DS2 includes a plurality of data D2 of the second region, that is, the data of the medical image in which the duodenum is imaged.
Furthermore, in a case where the first trained model is the lesion detector 43B for a stomach, the “first region” is a region to which the lesion detector 43B for a stomach corresponds, that is, the stomach and the “second region” is a region of the plurality of preset regions of the subject except for the first region, that is, the esophagus or the duodenum. Further, in a case where the first trained model is the lesion detector 43C for a duodenum, the “first region” is the duodenum and the “second region” is a region of the plurality of preset regions of the subject except for the first region, that is, the stomach or the esophagus. Even in the cases described above, as in the above-mentioned specific example, the learning unit 46 performs deep learning using the first data set DS1 including the data D1 of the first region and the data D2 of the second region and the second data set DS2 including the data D2 of the second region, so that a first trained model and a second trained model are obtained.
As described above, the first data set DS1 or the second data set DS2 is input to the learning unit 46 and deep learning is performed to obtain the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, and the lesion detector 43C for a duodenum that are the first trained models or the second trained models.
The first data set DS1 includes the data D1 of the first region more than the data D2 of the second region. For example, in a case where the first region is the esophagus, the first data set DS1 includes the data of the esophagus, which are the data D1 of the first region, more than the data of the stomach or the duodenum that are the data D2 of the second region.
It is preferable that the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, and the lesion detector 43C for a duodenum are first trained models or second trained models trained with the data D1, the data D2, and/or a first data set DS1 or a second data set DS2 having different combinations of the data D1 and D2. Accordingly, a first trained model or a second trained model optimal for each of the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, and the lesion detector 43C for a duodenum is generated.
Next, the flow of processing of the medical image processing device 20 in a case where an examination is made using the endoscope system 10 will be described with reference to a flowchart shown in
In the medical image processing device 20, first, the medical image acquisition unit 41 acquires a plurality of medical images from the processor device 14 in a time series (Step ST110). Next, the region classification processing for the medical images 50 acquired by the region classification unit 42 is performed (Step ST120). In the present embodiment, the region classification unit 42 performs the region classification processing for classifying a region of a subject in which the medical images 50 are captured into a first region or a second region. Specifically, the region classification unit 42 performs the region classification processing for recognizing which region of the esophagus 101, the stomach 102, or the duodenum 103 is the region of the subject and classifying the region of the subject.
Next, the region-of-interest detection unit 43 performs the detector selection processing for selecting any one of the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, or the lesion detector 43C for a duodenum, that is, a first trained model or a second trained model using a classification result obtained from the region classification unit 42 (Step ST130). Subsequently, the region-of-interest detection unit 43 performs the region-of-interest detection processing for detecting a region of interest from the medical images 50 by any one of the lesion detector 43A for an esophagus, the lesion detector 43B for a stomach, or the lesion detector 43C for a duodenum, that is, the first trained model or the second trained model which is selected in the detector selection processing (Step ST140).
The display controller 44 and the notification controller 45 notify the user of a detection result of the region of interest on the basis of the detection result of the region of interest obtained from the region-of-interest detection unit 43 and the medical images 50 (Step ST150). In a case where the user wants to continue the examination (Y in Step ST160), the process returns to the acquisition of the medical images 50 using the medical image acquisition unit 41 (Step ST110) and the above-mentioned processing is repeated. In a case where the user wants to end the examination (N in Step ST160), the user turns off the power of the endoscope system 10.
In a case where an examination is made using the endoscope system 10 as described above, a region of a subject may be classified into the second region in the region classification processing performed by the region classification unit 42 even though the first region is actually imaged and the medical images 50 are obtained. For example, with regard to the medical images 50 captured near a boundary of the region of the subject, a false classification result may be generated in the region classification processing performed by the region classification unit 42. Alternatively, a false classification result may be generated in the region classification processing performed by the region classification unit 42 even due to a change in the form of a lumen caused by the peristalsis of a gastrointestinal tract that is a subject, water coverage in a case where liquid or the like is sprayed on the subject, an influence in a case where the imaging optical system of the endoscope 12 is out of focus, or the like. In the example shown in
As shown in
In a case where the medical images 50 are captured near the boundaries 104 and 105 as described above, the region classification unit 42 is likely to generate a classification result in which the region of the subject is the esophagus 101 even though, for example, the stomach 102 is actually imaged and the medical images 50 are acquired. In this case, the region-of-interest detection unit 43 selects the lesion detector 43A for an esophagus on the basis of the false classification result.
However, since the first data set DS1 including the data D1 of the first region and the data D2 of the second region is input to the learning unit 46 in the present embodiment, the data D2 of the second region, that is, the data of the stomach are also included in the first data set DS1 that has been used to train the lesion detector 43A for an esophagus. Accordingly, since detection is performed in the medical images 50, which are acquired in a case where the stomach is actually imaged, by the first trained model trained with the first data set DS1 including the data of the stomach, false detection is less caused by the region-of-interest detection unit 43. Further, since the region-of-interest detection unit 43 less detects a region that is actually not a region of interest as the region of interest and less detects an actual region of interest as a region that is not the region of interest, the region-of-interest detection unit 43 can detect a region of interest with a stable behavior.
On the other hand, in a case where the stomach 102 is imaged and the medical images 50 are acquired, the region classification unit 42 almost always generates a correct classification result of the stomach 102. In this case, the lesion detector 43B for a stomach of the region-of-interest detection unit 43 is selected on the basis of the correct classification result and the detection of a region 55 of interest is performed in the medical image 50. In a case where the first region is the esophagus and the second region is the stomach as in the specific example described above, the data of the medical image of the stomach as the second region are included in the second data set DS2 that has been used to train the lesion detector 43B for a stomach. Accordingly, false detection is less caused by the region-of-interest detection unit 43.
Further, in the present embodiment, the first data set DS1 used for deep learning includes the data D1 of the first region more than the data D2 of the second region. For example, in a case where the first region is the esophagus, the data of the esophagus that are the data D1 of the first region are more than the data of the stomach or the duodenum that are the data D2 of the second region. The first trained model generated in this way is the lesion detector 43A for an esophagus. Accordingly, since the detection of a region of interest is performed by the lesion detector 43A for an esophagus trained with the first data set DS1 including a large amount of data of the esophagus in a case where a classification result is the esophagus 101, the region-of-interest detection unit 43 can detect a region of interest with higher accuracy. Therefore, the region-of-interest detection unit 43 can detect a region of interest with a stable behavior.
In the first embodiment, the region classification unit 42 analyzes medical images to acquire a classification result indicating a region of a subject in which the medical images 50 are captured. However, the present invention is not limited thereto, and the region classification unit 42 may acquire a classification result indicating a region of a subject in which medical images 50 are captured on the basis of a user's input as shown in
In a case where region information S is input, as shown in, for example,
In the flow of processing of the medical image processing device according to the present embodiment, a medical doctor who is a user operates the medical image processing device-side input device 22 to input the region information at the time of the region classification processing (Step ST120; see
In a case where an examination is made using an endoscope system including the medical image processing device according to the present embodiment as described above, a region of a subject may be classified into a second region due to false region information input by the user even though a first region is actually imaged and medical images 50 are obtained. In particular, since it is difficult to determine the region of the subject with regard to the medical image 50 captured near a boundary of the region of the subject, a false classification result may be generated. Alternatively, since the user forgets to perform an operation during the examination, it may not be possible to switch a classification result to an appropriate classification result.
However, since the first data set DS1 including the data D1 of the first region and the data D2 of the second region is input to the learning unit 46 as in the first embodiment, the data D2 of the second region are also included in the first data set DS1 that has been used to train the first trained model. Accordingly, in a case where the region of the subject is classified into the second region due to false region information input by the user, for example, in a case where the region of a subject is classified into the esophagus on the basis of a user's input with regard to the medical image 50 that is acquired in a case where the stomach is actually imaged, detection is performed by a trained model trained with the first data set DS1 including the data of the stomach. Therefore, false detection is less caused by the region-of-interest detection unit 43. Further, the region-of-interest detection unit 43 can detect a region of interest with a stable behavior as in the first embodiment.
In each embodiment, whether or not to include a region of interest in the data of a data set used for deep learning is not limited. In a modification example shown in
In each embodiment and the first modification example, in order to adjust the detection accuracy obtained from the first trained model and the second trained model, the numbers and contents of data D1 and D2 are set to be different in the first data set DS1 and the second data set DS2 used to train the first trained model and the second trained model. However, the present invention is not limited thereto, and the proportion (weighting) of each of the data D1 and D2 may be set to be different at the time of training the models by the learning unit 46 to adjust the detection accuracy of the models. In this case, it is preferable that the proportion of each of the data D1 and D2 is set to be different such that the first trained model has the detection accuracy of the region of interest in the first region higher than detection accuracy of the region of interest in the second region and the second trained model has the detection accuracy of the region of interest in the second region higher than the detection accuracy of the region of interest in the first region.
It is preferable that high detection accuracy in this specification is high sensitivity and/or high specificity. High sensitivity means that a region to be determined as the region of interest is highly likely to be correctly determined as the region of interest, and high specificity means that a region is normal (without the region of interest) and is less likely to be determined as a region including the region of interest.
In each embodiment, in a case where the region 55 of interest is detected from the medical images 50 by the region-of-interest detection unit 43, the display controller 44 causes the medical image display 21 to display the medical images 50 in which the region of interest is detected such that the detection result is superimposed on the medical images 50. However, the present invention is not limited thereto, and the display controller 44 may cause the medical image display 21 to display a medical image 50 such that a message MS1 indicating the detection of the region 55 of interest is superimposed on the medical image 50 as shown in
A message MS1 of “Stomach” indicating that the lesion detector 43B for a stomach is selected as the first trained model or the second trained model selected by the region-of-interest detection unit 43 is displayed in the modification example shown in
Further, the display controller 44 may cause the medical image display 21 to display not only a display aspect in which the detection result of the region 55 of interest is displayed as in each embodiment but also a message that indicates the selection state of the first trained model or the second trained model as in the modification example shown in
The hardware structures of the processing units, which execute various types of processing in each embodiment, such as the medical image acquisition unit, the region classification unit, the region-of-interest detection unit, the display controller, the notification controller, and the learning unit, are various processors to be described below. Various processors include: a central processing unit (CPU) that is a general-purpose processor functioning as various processing units by executing software (programs); a graphical processing unit (GPU); a programmable logic device (PLD) that is a processor of which the circuit configuration can be changed after manufacture, such as a field programmable gate array (FPGA); a dedicated electrical circuit that is a processor having circuit configuration designed exclusively to execute various types of processing; and the like.
One processing unit may be formed of one of these various processors, or may be formed of a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, a combination of a CPU and a GPU, or the like). Further, a plurality of processing units may be formed of one processor. As an example where a plurality of processing units are formed of one processor, first, there is an aspect where one processor is formed of a combination of one or more CPUs and software as typified by a computer, such as a client or a server, and functions as a plurality of processing units. Second, there is an aspect where a processor fulfilling the functions of the entire system, which includes a plurality of processing units, using one integrated circuit (IC) chip as typified by system on chip (SoC) or the like is used. In this way, various processing units are formed using one or more of the above-mentioned various processors as hardware structures.
In addition, the hardware structures of these various processors are more specifically electrical circuitry where circuit elements, such as semiconductor elements, are combined. Further, the hardware structure of the storage unit is a storage device, such as a hard disc drive (HDD) or a solid state drive (SSD).
A medical image processing device comprising a processor,
The medical image processing device according to claim 1,
The medical image processing device according to claim 1 or 2,
The medical image processing device according to any one of claims 1 to 3,
The medical image processing device according to any one of claims 1 to 4,
The medical image processing device according to any one of claims 1 to 5,
The medical image processing device according to any one of claims 1 to 6,
The medical image processing device according to claim 7,
The medical image processing device according to claim 7,
The medical image processing device according to any one of claims 1 to 9,
An endoscope system comprising:
A method of operating a medical image processing device, comprising:
A program for a medical image processing device causing a computer to execute:
| Number | Date | Country | Kind |
|---|---|---|---|
| 2023-080678 | May 2023 | JP | national |
