Patent Application
20220392063
The present invention relates to a program useful for indicating a Hunner lesion. More specifically, the present invention relates to a program for indicating a target Hunner lesion, a learned model, and a method for generating the same in the field of bladder abnormalities, in particular, interstitial cystitis.
Interstitial cystitis is a chronic disease that causes symptoms such as frequent urination, urinary urgency, and cystalgia or discomfort with bladder fullness. A seriously ill patient may urinate 60 times a day, which causes considerable influence and results in difficulty in daily life. Women are more likely to develop interstitial cystitis men. Among about 1.3 million patients of interstitial cystitis in the U.S., at least one million patients are female. Although epidemiological surveys have been frequently conducted, the cause of the disease has not been discovered yet. Furthermore, the definitions, criteria, and expressions of “interstitial cystitis” vary among countries and regions. Thus, a description of “interstitial cystitis” in the present application also includes the concepts of cystalgia and overactive bladder pains.
The criteria of case collection study (Interstitial Cystitis Data Base; ICDB)18) in the U.S. reflect criteria routinely used for interstitial cystitis in the U.S. In the criteria, cystoscopic findings are not preconditions. Moreover, NIDDK (National Institute of Diabates and Digestive and Kidney Diseases) provides criteria that have been frequently cited. However, the criteria are used for strictly selecting cases during study. The criteria require cystoscopic findings and thus are more strictly defined. Some reports show that only less than half of patients diagnosed as interstitial cystitis by ICDB criteria meet the criteria of NIDDK.
As a characteristic of interstitial cystitis, the types of interstitial cystitis are broadly divided into a Hunner type having Hunner lesions and a non-Hunner type having no Hunner lesions. Hunner lesions are unique rubefacient mucous membranes having no normal capillary structures. Pathologically, epitheliums frequently fall off (erosion), and neovascularization and clustered inflammatory cells are found in submucosa. The Hunner type endoscopically and pathologically has definite abnormal findings. The Hunner type is a unique rubefacient mucous membrane having no normal capillary structures. As described above, any international criteria have not been established, and thus a Hunner lesion may be referred to as a Hunner's ulcer or may be simply referred to as an ulcer depending upon the region.
The Hunner type is more likely to cause serious conditions and thus requires correct diagnoses and treatments at an earlier stage. However, as described above, the presence or absence of a Hunner lesion is not defined as a requirement of the diagnosis of interstitial cystitis in some regions. Since the definition of global standards and criteria have not been established yet, only a few doctors can carry out diagnosis of interstitial cystitis and Hunner lesions with correct knowledge. Hence, even if U.S. Pat. No. 8,080,185 about the diagnosis of interstitial cystitis has been disclosed by Tomohiro Ueda, one of the inventors of the present invention, potential patients are missed, and inappropriate diagnoses and treatments including a wrong diagnosis have been performed.
The cause of the difficulty of the problem is that cystoscope observations are frequently avoided in some regions where cystoscope findings are not always required for the diagnosis of interstitial cystitis. In this case, it may be difficult to confirm Hunner lesions with accuracy, so that potential patients may be missed as described above. Even if cystoscope observations are conducted, doctors with insufficient knowledge may miss potential patients. Moreover, cystoscope observations are mostly conducted on patients who have felt bladder abnormalities. Thus, just a few opportunities of observations on normal bladders are provided and only a few cystoscope images of normal bladders are accumulated. Furthermore, some doctors do not have knowledge about how to distinguish normal bladders from abnormal bladders free of Hunner lesions. Such circumstances cause a vicious cycle of difficulty in the definition of global standards and criteria.
However, regardless of the knowledge and skill of doctors, if a versatile technique capable of accurately and quickly identifying a Hunner lesion of a patient is provided and if a technique capable of properly recognizing a normal bladder is provided, remarkable progress is made in all of an accurate diagnosis of interstitial cystitis, the rescue of interstitial cystitis patients, and international consensus building.
In order to solve the problem, the present invention includes a method for generating a learning model, the method including: acquiring, as teaching data, endoscope image data on a Hunner lesion in a bladder; and generating the learning model by using the teaching data such that a bladder endoscope image serves as an input and the position indication of a Hunner lesion in the bladder endoscope image serves as an output.
Furthermore, the present invention includes a program that causes a computer to perform processing for acquiring endoscope image data on a Hunner lesion in a bladder, inputting a target bladder endoscope image to a learning model in which a bladder endoscope image serves as an input and position-indication data on a Hunner lesion in an endoscope image serves as an output, and outputting the position indication of the Hunner lesion.
Furthermore, the present invention includes a learned model that uses a bladder endoscope image acquired by using a bladder endoscope system, the learned model including an input layer that receives the bladder endoscope image, an output layer in which position indication data on a Hunner lesion in the endoscope image serves as an output, and an intermediate layer having a parameter learned by using teaching data in which endoscope image data on the Hunner lesion in a bladder serves as an input and position indication data on the Hunner lesion in the bladder image serves as an output, the learned model causing a computer to function to input a target bladder endoscope image to the input layer, perform an operation in the intermediate layer, and output the position indication data on the Hunner lesion in the image.
Moreover, the learning-model generating method, the program, the learned model preferably include, as teaching data, at least one of endoscope image data on air in a bladder, endoscope image data on a normal bladder, and endoscope image data on abnormal bladders free of Hunner lesions.
Furthermore, the image data preferably includes both of an image of narrow band imaging and an image of white light imaging.
Moreover, process of determination whether a bladder is normal or not and outputting is preferably enabled, and a program or a learned model is preferably installed in the controller of a bladder endoscope.
The program and the learned model of the present invention can be implemented by using known configurations, for example, a controller or a server including a CPU, a GPU, a ROM, a RAM, and a communication interface. Moreover, the program and the learned model can be configured as cloud systems. Furthermore, the present invention preferably includes means that causes display means such as a display to indicate and display the position of an estimated Hunner lesion by visually recognizable means such as framing or coloring. Moreover, the present invention may be used as a system to be used independently of an endoscope or may be used in the controller of a bladder endoscope system in real time during an observation in a bladder.
Furthermore, a deep learning model is used in the present invention. Typically, deep learning using a neural network is used, and a convolutional neural network is preferably used. When an image is inputted, a convolutional neural network acts as estimating means for estimating the position of a Hunner lesion. The estimating means is not limited thereto unless the effect of the present invention is disabled.
A deep learning model includes a large number of internal parameters. The internal parameters are adjusted to obtain, contrary to input data, an output result closest to teaching data. This adjustment is generally called “learning.” In order to generate a high-performance model, the amount and quality of teaching data (a set of input data and teaching data) used for learning is important in addition to a model structure and a learning method.
Teaching data in the present invention is an endoscope image of a Hunner lesion in a bladder. A model capable of indicating the Hunner lesion is generated by indicating and learning the Hunner lesion. As a result of study, it was found that air (bubbles) in a bladder may be identified as a Hunner lesion or a Hunner lesion may be indicated in a normal bladder (it is assumed that a shadow of wrinkles or a height difference in a normal bladder may affect the indication). Thus, in order to prevent these factors from being identified as Hunner lesions, air (bubbles) was learned as air (bubbles). Since a mucosal condition of a normal bladder is different from a bladder condition of interstitial cystitis regardless of the presence or absence of a Hunner lesion, images of abnormal bladders free of Hunner lesions were learned in addition to images of normal bladders.
In the present invention, normal bladders are learned to effectively eliminate a determination rate of false positives about normal bladders. Moreover, images of abnormal bladders free of Hunner lesions are learned to effectively avoid the determination of false positives about abnormal bladders free of Hunner lesions. Furthermore, if a model is generated to output a normal bladder as a normal bladder, a configuration for determining and indicating, as an abnormal bladder, an abnormal bladder free of Hunner lesions can be achieved.
In order to obtain algorithms with high accuracy, it is necessary to learn a large number of Hunner lesion patterns. However, only a small number of bladder endoscope images of interstitial cystitis patients has been globally accumulated. Tomohiro Ueda, one of the inventors, has accumulated the largest number of bladder endoscope images of interstitial cystitis patients (and normal bladder images and images of abnormal bladders free of Hunner lesions) in the world. Thus, a wide variety of Hunner lesion images has been accumulated, allowing learning of images of sufficient types and quality for generating a model. Moreover, images of both of narrow band imaging (NBI) and white light imaging (WLI) are learned in the present invention, so that the model is usable for images of NBI and WLI. In narrow band imaging, narrowband light of two wavelengths of blue (wavelength from 390 nm to 445) and green (wavelength of 530 nm to 550 nm) is emitted, and a microscopic vessel image is outputted with an enhanced contrast. Blue light indicates the presence or absence of a neovascular on a mucous membrane while green light indicates the presence or absence of a vessel deep under a mucous membrane. White light imaging is an observation using illumination light of primary colors of blue, green, and red from the tip of an endoscope. In actual diagnoses by doctors, lesions are more likely to be visually confirmed in narrow band imaging. However, it is assumed that the utilization rate of white light imaging is higher than that of narrow band imaging in the world including developing countries. In white light imaging, a lesion and a background look red, causing difficulty in diagnosis. The present invention is also applicable to white light imaging and thus is so versatile as to be useful for finding patients of interstitial cystitis and building an international consensus.
Dataset
Based on moving images captured by a bladder endoscope, correct information is added (annotation) to the moving images in the following steps:
Data to which the correct information has been added was divided into NBI and WLI to produce a database of still images. In an extracted image, a black area outside an endoscope image was erased by trimming. This processing obtains a pixel size of about 1000×900 pixels. Moreover, an annotation at the position of a Hunner lesion was made by software called Label me.
The number of moving images
Still-Image Database Configuration
Bubbles include images of normal bladders and images having Hunner lesions.
The image examples of bubbles in
Model
An experiment was conducted by using a detection model and a segmentation model. The detection model is a model for estimating a rectangular area including a Hunner lesion area. The position and size of the rectangular area and the lesion confidence factor of the rectangle. The segmentation model is a model for outputting a lesion confidence factor for each pixel. A Hunner lesion area is estimated as well as the shape of the area.
In the experiment, models of high performance for a dataset of general images (COCO, CITYSCOPES) were used as follows:
The three models were learned with data of NBI and WLI, and six models were created in total.
Experiment Setting and Result
In the experiment, the dataset was randomly divided into learning data of 85% and test data of 15% five times, and learning and evaluation were performed five times. The data was divided for each case. Tables 3 and 4 indicate the number of images and the number of cases in NBI and WLI datasets.
The number of images and the number of cases in NBI dataset
The number of images and the number of cases in WLI dataset
Three models (Cascade R-CNN, Cascade Mask R-CNN, OCNET) were learned for each image type and each divided dataset, and the performance of each model was evaluated based on the sensitivity and the positive prediction value of each Hunner lesion area. The sensitivity and the positive prediction value (PPV) are expressed as follows: Sensitivity=#TP/(#TP+#FN) PPV=#TP/(#TP+#FP) where #TP is the number of true positives, #FN is the number of false negatives, and #FP is the number of false negatives.
A method for determining a true positive (TP), a false negative (FN), and a false positive for each Hunner lesion area will be described below. First, the degree of overlapping of a predicted area and a correct area is calculated by an IoU (Intersection over Union). The IoU is the ratio of the number of duplicate pixels in the predicted area and the correct area relative to the number of pixels in the union area of the predicted area and the correct area. If the two areas do not overlap each other at all, the IoU is 0, whereas if the two areas completely agree with each other, the IoU is 1.
In the present example, TP, FN, and FP were defined as follows:
As described above, Cascade R-CNN, a detection model, is evaluated with rectangular regions. Cascade Mask R-CNN and OCNET, segmentation models, are evaluated in consideration of area shapes. The correct areas and the predicted areas are provided. As shown in
Tables 5 to 8 below show the evaluation results of each model.
Table 5: the performance of the detection model (Cascade R-CNN) evaluated in an NBI image
Table 6: the performance of the segmentation model evaluated in an NBI image
Table 7: the performance of the detection model (Cascade R-CNN) evaluated in a WLI image
Table 8: the performance of the segmentation model evaluated in a WLI image
The performance of the detection model (Cascade R-CNN) evaluated in an NBI image
The performance of the segmentation model evaluated in an NBI image
The performance of the detection model (Cascade R-CNN) evaluated in a WLI image
The performance of the segmentation model evaluated in a WLI image
All predicted areas with a confidence factor other than 0% are displayed. Dotted lines (green in the original images) indicate correct areas, and solid lines (red in the original images) indicate predicted areas, which sufficiently indicates a Hunner lesion.
Rectangular outputs are displayed with confidence factors in addition to area specifications. An upper left image, an upper central image, and an upper right image sequentially indicate Hunner lesions with confidence factors of 0.78 (left), 0.9 (center), and 0.97 (right). A lower left image, a lower central image, and a lower right image sequentially indicate a Hunner lesion with a confidence factor of 0.33 and bubbles with a confidence factor of 0.79 (left), bubbles with a confidence factor of 0.97 (center), and a Hunner lesion with a confidence factor of 0.98 and a confidence factor of 0.4. In the present example, bubbles are indicated as bubbles in output display, so that the indication is successfully distinct from that of a Hunner lesion. In the original image, the correct area is indicated in green.
Rectangular outputs are displayed with confidence factors in addition to area specifications. From the left to the right, a Hunner lesion is indicated with confidence factors of 0.96 and 0.99 (left), 0.97 (center), and 0.997 (right). This successfully provides a sufficient indication of a Hunner lesion. In the original image, the correct area is indicated in green.
As described above, a learned model applicable to images of narrow band imaging and white light imaging can be formed. Moreover, bubbles can be indicated. According to the present invention, an opportunity can be provided to accurately and quickly identify a Hunner lesion in images of narrow band imaging and white light imaging regardless of the knowledge and skill of a doctor. Furthermore, determination of a false positive is avoided even if a height difference is made or a red phase is caused by a shadow or the like in a normal bladder. Consequently, the present invention contributes to the progress of an accurate diagnosis of interstitial cystitis, the rescue of interstitial cystitis patients, and international consensus building.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-195447 | Nov 2020 | JP | national |
The present application is a continuation application of International Patent Application No. PCT/JP2021/043293 filed on Nov. 25, 2021, which claims priority to Japanese Patent Application No. JP2020-195447, filed on Nov. 25, 2020, the entire disclosures of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2021/043293 | Nov 2021 | US |
| Child | 17818853 | US |
