Patent Grant
12125196
The present technology relates to a computer program, a processor for an endoscope, and an information processing method.
In the case of performing an endoscopic examination, one or a plurality of examination target portions are determined in advance for each patient who is a subject, and the examination target portions are imaged in a body cavity of the patient according to a prescribed procedure to perform image diagnosis. Specifically, a person in charge of examination of the endoscope inserts the endoscope insertion portion into the body cavity of the patient, and causes the endoscope insertion portion to reach a predetermined examination target portion while confirming the position of the distal end of the insertion portion of the endoscope in the body cavity. Then, while an image is confirmed on the monitor screen each time it arrives, a still image is photographed by pressing a button for still image photographing while finely adjusting a photographing position and a direction.
The number of captured images may be several to 20 or more in one medical examination, and it is desired to examine a large number of people in a short time at a medical site. Therefore, the person in charge of examination is required to perform an accurate and rapid imaging operation. Under such a situation, particularly an error such as failure in imaging of a part of the examination portion is likely to occur, and if failure in imaging occurs at a necessary examination portion, accurate diagnosis becomes difficult. In addition, in order to obtain a missed image, a burden on the patient is increased in order to re-take the image with the endoscope by re-examination.
In a medical facility, it is effective to define an examination protocol defining an examination procedure and the like in order to prevent a difference in a burden on a patient and an organ observation method from being generated due to a difference in experience and skill of a person in charge of examination. Patent Literature 1 discloses an endoscope operation support system that can easily confirm whether an examination has been performed according to an examination protocol in a medical facility that defines the examination protocol.
Patent Literature 1: JP 2017-108792 A
However, in the endoscope business support system of Patent Literature 1, the confirmation as to whether the examination has been performed is performed after the examination based on the examination record. Therefore, there is a concern about omission of the observation content. In order to perform an appropriate examination content regardless of the experience and skill of a person in charge of examination, it is desirable to derive information regarding the examination contents in real time.
An object of the present disclosure is to provide a computer program or the like that realizes efficient derivation of information regarding an examination content.
A computer program according to an aspect of the present disclosure causes a computer to execute acquiring examination point information regarding an examination point included in an examination target portion using an endoscope, acquiring an endoscopic image captured by the endoscope, determining whether the endoscope has reached the examination point on the basis of image analysis of the acquired endoscopic image, and outputting a notification when the endoscope has reached the examination point.
According to the present disclosure, it is possible to provide a computer program or the like that realizes efficient derivation of information regarding an examination content.
The present invention will be specifically described with reference to the drawings illustrating embodiments of the invention.
The display device 50 is installed on the upper stage of a storage shelf 16 with casters. The endoscope processor 20 is housed in the middle stage of the storage shelf 16. The storage shelf 16 is arranged in the vicinity of an endoscopic examination bed (not illustrated). The storage shelf 16 includes a pull-out shelf on which a keyboard 15 connected to the endoscope processor 20 is mounted.
The endoscope processor 20 has a substantially rectangular parallelepiped shape and is provided with a touch panel 25 on one surface. A reading unit 28 is arranged at the bottom of the touch panel 25. The reading unit 28 is a connection interface for reading and writing a portable recording medium such as a USB connector, an SD (Secure Digital) card slot, or a CD-ROM (Compact Disc Read Only Memory) drive.
The endoscope 40 includes an insertion portion 44, an operation unit 43, a light guide flexible tube 49, and a scope connector 48. The operation unit 43 is provided with a control button 431. The insertion portion 44 is long, and has one end connected to the operation unit 43 via a bend preventing portion 45. The insertion portion 44 includes a soft portion 441, a bending portion 442, and a distal tip 443 in the order from the operation unit 43 side. The bending portion 442 is bent according to an operation of a bending knob 433. Physical detection devices such as a three-axis acceleration sensor, a gyro sensor, a geomagnetic sensor, or a magnetic coil sensor may be mounted on the insertion portion 44, and when the endoscope 40 is inserted into the body of the subject, detection results from these physical detection devices may be acquired.
The light guide flexible tube 49 is long, and has a first end connected to the operation unit 43 and a second end connected to the scope connector 48. The light guide flexible tube 49 is flexible. The scope connector 48 has a substantially rectangular parallelepiped shape. The scope connector 48 is provided with an air/water supply port 36 (see
In the diagnostic support system 100, the endoscope processor 20 determines the examination content of the endoscope 40 in real time based on an endoscopic image 59 captured by the endoscope 40, and notifies the examiner of the determination result. The examiner operates the endoscope 40 while referring to the notification.
The control unit 21 is an arithmetic control device that executes a program 231 of this embodiment. For the control unit 21, one or a plurality of central processing units (CPUs), micro-processing units (MPUs), graphics processing units (GPUs), or the like is used. The control unit 21 is connected to each hardware unit constituting the endoscope processor 20 via the bus.
The main memory device 22 is a memory device such as a static random access memory (SRAM), a dynamic random access memory (DRAM), and a flash memory. The main memory device 22 temporarily holds information necessary during the processing performed by the control unit 21 and a program being executed by the control unit 21.
The auxiliary memory device 23 is a memory device such as an SRAM, a flash memory, or a hard disk. The auxiliary memory device 23 holds the program 231 executed by the control unit 21, a learned model 237, and various data necessary for executing the program 231. The program 231 and the learned model 237 may be downloaded from an external device via a network such as a so-called Internet via the communication unit 24 and stored in the auxiliary memory device 23 by the control unit 21. The program 231 and the learned model 237 may be read by the control unit 21 from a portable storage medium 23a via the reading unit 28 and stored in the auxiliary memory device 23. The program 231 and the learned model 237 may be read from a semiconductor memory 23b by the control unit 21.
The auxiliary memory device 23 stores an examiner DB (Database) 232 and an examination point DB 233. The examiner DB 232 and the examination point DB 233 may be stored in an external mass storage device connected to the endoscope processor 20.
The communication unit 24 is an interface for data communication between the endoscope processor 20 and the network. The touch panel 25 includes a display unit 251 such as a liquid crystal display panel, and an input unit 252 layered on the display unit 251.
The display device I/F 26 is an interface for connecting the endoscope processor 20 and the display device 50. The input device I/F 27 is an interface for connecting the endoscope processor 20 and an input device such as the keyboard 15.
The light source 33 is, for example, a high-luminance white light source such as a white LED. The light source 33 is connected to the bus via a driver (not illustrated). The on/off of the light source 33 and the change of brightness are controlled by the control unit 21. The illumination light emitted from the light source 33 is incident on the optical connector 312. The optical connector 312 engages with the scope connector 48 to supply illumination light to the endoscope 40.
The pump 34 generates pressure for the air supply/water supply function of the endoscope 40. The pump 34 is connected to the bus via a driver (not illustrated). The on/off and pressure change of the pump 34 are controlled by the control unit 21. The pump 34 is connected to the air/water supply port 36 provided in the scope connector 48 via a water supply tank 35.
The function of the endoscope 40 connected to the endoscope processor 20 will be outlined. A fiber bundle, a cable bundle, an air supply tube, a water supply tube, and the like are inserted inside the scope connector 48, the light guide flexible tube 49, the operation unit 43, and the insertion portion 44. The illumination light emitted from the light source 33 is radiated from an illumination window provided at the distal tip 443 via the optical connector 312 and the fiber bundle.
The range illuminated by the illumination light is captured by an image sensor provided at the distal tip 443. The captured image is transmitted from the image sensor to the endoscope processor 20 via the cable bundle and the electric connector 311.
The control unit 21 performs image processing on the captured image to generate the endoscopic image 59 that makes it easy for the examiner performing examination to visually find a lesion. The control unit 21 outputs the endoscopic image 59 to the display device 50. The examiner operates the endoscope 40 while confirming the endoscopic image 59, and when reaching a predetermined examination point, presses a button for still image shooting to shoot a still image.
The examination point information 234 stores check point information, an examination point position, an examination point image, an endoscope type, and notification information in association with an examination point ID for identifying the examination point information. The check point information includes information of a check point related to the examination point, and for example, information of an adjacent check point of the examination point is stored. When the examination point is a check point, a check point number displayed on the display device 50 may be included. The examination point position may include, for example, information related to the position of the examination point, such as position information in the body of the examination point, distance information between the examination points, and the like. The examination point image may include a plurality of pieces of image information indicating the examination point. The endoscope type may include a type of the endoscope 40 connected to the endoscope processor 20. In the examination using the endoscope, the endoscope 40 of a type suitable for the examination target portion is selected and used. As the endoscope type, for example, an upper portion (digestive organ), an upper portion (respiratory organ), a lower portion (digestive organ), and the like are stored. The notification information may include information regarding whether a notification is output at the examination point. In the examination point for which the output of notification is stored, a notification for notifying the examiner of the examination content is output. The notification information may store the presence or absence of notification in association with the skill level.
Further, the examination point information 234 may store the target skill level in association with the examination point ID. The target skill level includes information on the skill level of the examiner to examine the examination point associated with the target skill level. The target skill level will be described in detail in other embodiments.
In the endoscopic examination, a guideline (manual) for allowing the examiner of various skill levels to photograph images all over the inside of the gastrointestinal tract at an appropriate level is defined (see, for example, “Manual of Gastroendoscopic Examination for Countermeasure Screening, 2015 Edition (pages 56 to 63)”). The examination point position may be set based on these guidelines. For example, in the case of performing a gastroscopic examination, the examination point position to be photographed includes the photographing place described in the manual of gastroendoscopic examination.
The candidate information 235 stores the candidate information of the examination point in association with the endoscope type which is information for identifying the type of the endoscope. The candidate information includes information related to a candidate of an examination point to be a determination point of an examination content in a case where a predetermined type of endoscope is used. The candidate information may include the order of passage of the examination points at the time of examination. In the example of
A neural network (learned model 237) learned using training data is assumed to be used as a program module that is a part of artificial intelligence software. The learned model 237 is used by the endoscope processor 20 including the control unit 21 (CPU or the like) and the auxiliary memory device 23 as described above, and is executed by the endoscope processor 20 having the arithmetic processing capability in this manner, thereby constituting a neural network system. That is, the control unit 21 of the endoscope processor 20 operates to perform calculation of extracting the feature amount of the endoscopic image input to the input layer according to the command from the learned model 237 stored in the auxiliary memory device 23, and output the examination point ID corresponding to the endoscopic image from the output layer.
The input layer has a plurality of neurons that receive an input of the pixel value of each pixel included in the endoscopic image 59, and passes the input pixel value to the intermediate layer. The intermediate layer includes a plurality of neurons that extract the image feature amount of the endoscopic image 59, and passes the image feature amount extracted using various parameters to the output layer. For example, when the learned model 237 is a CNN, the intermediate layer has a configuration in which a convolution layer that convolutes a pixel value of each pixel input from the input layer and a pooling layer that maps a pixel value convoluted by the convolution layer are alternately connected. The intermediate layer finally extracts the image feature amount while compressing the pixel information of the image region of the object.
The output layer includes a plurality of neurons that output the examination point ID corresponding to the endoscopic image. Each neuron can be divided in association with each examination point ID. The output value from the output layer can be interpreted as a probability that the examination point ID classified into each classification is included in the endoscopic image. For example, among the neurons with the examination point IDs of 1030, 1130, 2012, . . . , the information on the examination point ID of the neuron with the highest probability or the neuron with the probability equal to or greater than a threshold can be set as the output value of the learned model 237. The output layer outputs an output value based on the image feature amount output from the intermediate layer.
The control unit 21 learns various parameters in the intermediate layer by using training data in which a plurality of endoscopic images obtained by photographing the examination point by the endoscope and the examination point ID in each endoscopic image are associated with each other. For example, as illustrated in
The control unit 21 inputs the endoscopic image included in the training data to the input layer, performs arithmetic processing in the intermediate layer, and acquires the examination point ID from the output layer. The control unit 21 compares the examination point ID output from the output layer with the examination point ID labeled with respect to the image in the training data, that is, the correct value, and optimizes various parameters used for the arithmetic processing in the intermediate layer such that the output value from the output layer approaches the correct value. The parameter is, for example, a weight, a bias, or the like between neurons. The method of optimizing various parameters is not particularly limited, but for example, the control unit 21 optimizes various parameters using an error back propagation method. The control unit 21 performs the above processing on each endoscopic image included in the training data, and generates the learned model 237.
Although the description has been given assuming that the learned model 237 is the CNN in this embodiment, the learned model 237 is not limited to the CNN. When the time-series data is acquired, a neural network other than the CNN, for example, a recurrent neural network (RNN) or a long short term memory (LSTM) network may be used. Further, it may be a learned model constructed by another learning algorithm such as a reinforcement learning model, a support vector machine, or a regression tree that does not use a neural network.
The endoscopic image acquisition unit 211 acquires the endoscopic image 59. The control unit 21 acquires a captured image taken by an image sensor provided at the distal tip 443 of the endoscope 40. The captured image is obtained as a moving image, and is constituted by still images of a plurality of frames such as 60 frames per second, for example. The control unit 21 performs various types of image processing such as gamma correction, white balance correction, and shading correction on the captured image, generates the endoscopic image 59 in a state in which the user can easily view the endoscopic image, and stores the endoscopic image in the auxiliary memory device 23. The endoscopic image acquisition unit 211 acquires the generated endoscopic image 59. Note that the endoscopic image 59 input to the endoscopic image acquisition unit 211 may be a captured image at a stage where various types of image processing are not performed, or may be an image in the middle stage of generating the endoscopic image 59 from the captured image. The acquired endoscopic image 59 is input to the image analysis unit 214.
The endoscope type determination unit 212 acquires endoscope type information based on the information acquired from the endoscope 40. The endoscope type information is information related to the type of the endoscope 40 connected to the endoscope processor 20. The endoscope type determination unit 212 determines the type of the endoscope 40 from a connector shape of the endoscope 40 connected to the endoscope processor 20, signal information obtained from the endoscope 40, or the like, and acquires endoscope type information. The acquired endoscope type information is input to the examination point information acquisition unit 213.
The examination point information acquisition unit 213 refers to the examination point DB 233 and acquires candidate information including the examination point corresponding to the endoscope type information. In addition, the examination point information acquisition unit 213 receives the examination point ID of the check point selected by the examiner, and derives the examination point information based on the candidate information and the check point selected by the examiner.
The examination point information acquisition unit 213 receives an input of the keyboard 15, the mouse, or the like, and acquires the examination point ID of the selected check point. The examination point information acquisition unit 213 refers to the examination point DB 233 and specifies the examination point related to the check point based on the selected check point. The examination point includes a selected check point, a passing point related to each check point, and the like. The examination point information acquisition unit 213 derives the examination point information associated with these examination points from the examination point DB 233. The examination point information acquisition unit 213 temporarily stores the derived examination point information in the auxiliary memory device 23. The examination point information is input to the image analysis unit 214. Note that these input operations and output operations may be performed using the display unit 251 and the input unit 252 of the touch panel 25.
The image analysis unit 214 includes an examination content determination unit (determination unit) 215. The examination content determination unit 215 performs image analysis of the endoscopic image 59 and determines examination point information corresponding to the endoscope 40. The examination content determination unit 215 provides the endoscopic image 59 to the learned model 237 as input information. The learned model 237 outputs the examination point ID included in the endoscopic image as output information according to the endoscopic image 59. The examination content determination unit 215 determines whether the endoscope 40 has reached the examination point on the basis of whether the examination point ID output from the learned model 237 matches the examination point ID included in the examination point information derived by the examination point information acquisition unit 213, that is, the examination point ID included in the examination target portion.
The examination content determination unit 215 determines whether an appropriate examination content has been performed using the endoscope 40. When the endoscope 40 has reached a check point, the examination content determination unit 215 determines whether an appropriate examination content has been performed at the check point. The execution of an appropriate examination content means that, for example, at a check point, the control button 431 is pressed and a capture operation of a predetermined image is performed. The examination content determination unit 215 determines whether a capture operation has been performed at a check point on the basis of operation information of the endoscope 40, capture history information of an image, or the like. Note that the examination content is not limited to the image capture operation, and may be, for example, air supply or water supply operation, removal of a foreign object, or the like. In addition, the examination content determination unit 215 determines the presence or absence of omission of passage through the examination point based on the history of reaching the examination point. The examination content determination unit 215 acquires information of a plurality of examination points to be detected by the endoscope 40 in advance as the examination point information. The examination point information includes information such as a passage order and a position of the examination point. In a case where the examination point is not detected from the endoscopic image 59 in a predetermined order in the middle of the examination, the examination content determination unit 215 determines that a predetermined examination point to pass through is omitted. The examination content determination unit 215 creates notification information according to the determination result. The notification information of the determination result is input to the output unit 218.
The examination content determination unit 215 may acquire the position information in the body of the distal tip 443 of the endoscope 40 at the time of determining the examination content. It is possible to obtain a more accurate determination result by determining whether the endoscope 40 has reached the examination point on the basis of the position information of the distal tip 443 and the image analysis result. Although there are various methods of detecting the position information of the distal tip 443, for example, the examination content determination unit 215 may detect the position information on the basis of the insertion amount of the endoscope 40. The insertion amount of the insertion portion 44 into the body is detected by comparison with the reference point with reference to the previous insertion. The examination content determination unit 215 derives the position information of the distal tip 443 from the insertion amount of the endoscope 40 based on attributes such as the height and weight of the subject acquired in advance. Note that, in this case, a plurality of coils is arranged inside the insertion portion 44, an electric signal generated by the coils is output to the outside via the lead wire by receiving an electromagnetic wave from the outside, and the shape of the endoscope 40 may be grasped based on the amplitude, phase, and the like of the output electric signal. By detecting the shape of the endoscope 40 in the body and correcting the position information on the basis of the insertion amount of the endoscope 40 including rotation, curvature, and the like, accurate position information of the endoscope 40 in the body can be obtained. For example, based on the position information and the image analysis result, the examination content determination unit 215 determines that the examination content at the examination point is omitted when the examination point is not detected when the predetermined position is passed.
In addition, the examination content determination unit 215 may arrange a sensor that detects position information on the distal tip 443 and derive the position information of the distal tip 443 from the detection information of the sensor. For example, the examination content determination unit 215 acquires signals indicating the measurement information of the acceleration and the rotation angular velocity output from a three-axis acceleration sensor and a gyro sensor disposed in the distal tip 443, and derives the position information of the distal tip 443 from the acquired detection result of the sensor.
Note that the method of image analysis is not limited to using the learned model 237, and other known methods may be used. For example, the examination content determination unit 215 may determine whether the examination point is included in the endoscopic image 59 on the basis of the feature amount of the endoscopic image 59 and the feature amount of the examination point stored in advance using a method such as pattern matching.
The output unit 218 creates display screen information based on various types of information such as candidate information and notification information of a determination result, and outputs the created display screen information to the display device 50.
In this embodiment, the sharing of each functional unit in the series of processing is an example, and the present invention is not limited thereto. The function of each functional unit including the learned model 237 may be realized by a control unit (not illustrated) of an external server communicatively connected to the endoscope processor 20. Further, the control unit 21 of the endoscope processor 20 and the control unit of the external server may function as each functional unit in a series of processing in cooperation by performing inter-process communication, for example.
The control unit 21 refers to the examination point DB 233 and specifies a plurality of candidate check points included in the candidate information based on the endoscope type information. The control unit 21 creates display screen information including candidate check points and displays the display screen information on the display device 50 as illustrated in
The control unit 21 acquires endoscope type information from the connected endoscope 40 (Step S11). The control unit 21 refers to the examination point DB 233 and acquires candidate information including the examination point corresponding to the type of the endoscope (Step S13). The candidate information includes information on a plurality of candidate check points. In order to receive the selection input of the examiner, the control unit 21 creates the input screen information including the candidate information and displays the input screen information on the display device 50 (Step S15).
The control unit 21 receives selection of the examination point (check point) by the examiner by an input operation of the keyboard 15, a mouse, or the like (Step S17), and acquires the examination point ID of the selected check point (Step S19). The control unit 21 refers to the examination point DB 233 and acquires examination point information corresponding to the examination content based on the selected check point (Step S21). The examination point information includes information on the examination point between the selected check point and the check point derived from the degree of relevance with the selected check point. The control unit 21 stores the acquired examination point information in the auxiliary memory device 23 (Step S23).
The endoscopic examination is started, and the endoscope 40 photographs an examination target portion. The control unit 21 acquires the endoscopic image 59 (Step S31). The control unit 21 gives the endoscopic image 59 to the learned model 237 (Step S33), and specifies the examination point ID to be output (Step S35). Note that the number of learned models 237 stored in the endoscope processor 20 is not limited to one, and a plurality of learned models according to the type of the endoscope may be prepared. In this case, the learned model according to the endoscope type information is selected. In addition, endoscope type information may be input as one of the elements of the input information.
The control unit 21 determines whether the distal tip 443 of the endoscope 40 has reached the examination point (Step S37). The control unit 21 refers to the auxiliary memory device 23 and determines whether the examination point ID of the examination point information selected and registered in advance matches the examination point ID corresponding to the endoscopic image 59 specified from the output value of the learned model 237.
When it is determined that the examination point IDs do not match or the examination point is not included in the endoscopic image 59 and thus the endoscope 40 has not reached the examination point selectively registered in advance (Step S37: NO), the control unit 21 then determines whether the examination point is omitted (Step S39). The control unit 21 determines whether the examination point is omitted based on the passing order, the position, and the like of the examination point included in the examination point information. When the endoscope 40 is moving to the next examination point, it is determined that the examination point is not omitted (Step S39: NO). In this case, the control unit 21 advances the processing to Step S51. On the other hand, when the endoscopic image 59 including the examination point ID is not acquired in a predetermined order, or when the endoscopic image 59 including the examination point ID is not acquired at a predetermined position, the control unit 21 determines that the examination point that has already passed is omitted (Step S39: YES), and the processing proceeds to Step S49.
When the examination point ID of the examination point information selected and registered in advance matches the examination point ID specified from the output value of the learned model 237, the control unit 21 determines that the endoscope 40 has reached the examination point (Step S37: YES). The control unit 21 determines whether the examination point is a check point on the basis of the examination point ID (Step S41).
In a case where the examination point included in the endoscopic image 59 is an examination point other than the check point (Step S41: NO), the control unit 21 determines whether to output a notification on the basis of the examination point information (Step S43). The control unit 21 refers to the information on the presence or absence of the output of the notification stored in the examination point information 234 of the examination point DB 233, and determines whether the examination point is an output target of the notification. When the examination point is not the output target of the notification, the control unit 21 determines not to output the notification (Step S43: NO), and advances the processing to Step S51. On the other hand, when the examination point is the output target of the notification, the control unit 21 determines that the output of the notification is necessary (Step S43: YES), and advances the processing to Step S49.
In a case where the examination point included in the endoscopic image 59 is a check point (Step S41: YES), the control unit 21 determines whether an appropriate examination content has been performed (Step S45). The control unit 21 determines whether imaging at the examination point has been performed, for example, by acquiring the presence or absence of the capture operation of the endoscope 40. When it is determined that the capture operation has been performed and an appropriate examination content has been performed (Step S45: YES), the control unit 21 stores an operation history of the endoscope 40 in the auxiliary memory device 23 (Step S47). On the other hand, when it is determined that the capture operation has not been performed and an appropriate examination content has not been performed (Step S45: NO), the control unit 21 advances the processing to Step S49.
In a case where various determination results regarding the examination at the examination point, such as the execution of an appropriate examination content by the endoscope 40, omission of the examination content, reaching around the examination point, or omission at the examination point, are acquired, the control unit 21 outputs a notification of the determination result. The control unit 21 acquires the determination result, and creates display screen information including notification information according to the determination result with reference to the notification DB. The control unit 21 outputs the display screen information including the created notification information to the display device 50 (Step S49). The notification information displayed on the display device 50 includes information regarding the examination content such as confirmation completion, omission of examination, and reaching around the examination point.
The control unit 21 determines whether photographing has ended (Step S51). When it is determined that the photographing has not ended (Step S51: NO), the control unit 21 returns the processing to Step S31 and acquires a new endoscopic image. When it is determined that the photographing has ended (Step S51: YES), the control unit 21 ends a series of processing.
According to this embodiment, the examination content is detected in real time on the basis of the endoscopic image 59, and a notification corresponding to the detection result is output. Since the information regarding the implementation of the examination content is efficiently derived, the examiner can confirm the examination content based on the notification information.
The second embodiment is different from the first embodiment in that the endoscope processor 20 acquires motion information of an examiner and outputs a notification according to the skill level of the examiner.
In the second embodiment, the examination point information acquisition unit 213 acquires the examination point information according to the skill level of the examiner. The setting of the examination point may have different contents depending on the skill level of the examiner. For example, in a case where an examiner who is inexperienced in an endoscopic examination using a predetermined type of endoscope performs an examination, it is desirable to perform the examination by passing through all the basically set examination points without omission. On the other hand, in a case where a skilled examiner performs an examination, instead of performing the examination content at all the examination points which are basically set, the examination may be performed at the examination points in which several examination points are reduced. In addition, the examination point set according to the skill level may be changed. In the examiner DB 232, the target skill level of the examiner to include the examination point as the determination point is stored in association with the examination point information.
The examination point information acquisition unit 213 acquires the endoscope type information and the skill level of the examiner who performs the endoscopic examination. The examiner inputs an examiner ID and the like in advance on the input screen. The examination point information acquisition unit 213 acquires the examiner ID, and specifies the skill level associated with the examiner with reference to the examiner DB 232. The examination point information acquisition unit 213 refers to the examination point DB 233, extracts candidate information according to the endoscope type information and the target skill level, and displays a candidate check point included in the candidate information on the display device 50 via the output unit 218. The examiner selects or adds a check point from among the candidate check points displayed on the display device 50 according to the examination purpose, the skill level of the examiner, or the like, and sets the check point. The examination point information acquisition unit 213 receives an input of the keyboard 15, the mouse, or the like, and acquires the examination point ID of the selected check point.
The examination point information acquisition unit 213 specifies the examination point with reference to the examination point DB 233 based on the acquired examination point ID of the check point and the target skill level. In addition to the set check point, the examination point includes a portion around the check point added according to the skill level, a passing point between the check points, and the like. The examination point DB 233 stores the examination point in association with the target skill level. The examination point information acquisition unit 213 derives the examination point information associated with these examination points from the examination point DB 233. The examination point information acquisition unit 213 outputs the derived examination point information to the image analysis unit 214.
The image analysis unit 214 of the second embodiment includes the examination content determination unit 215 and the motion detection unit 216. The motion detection unit 216 performs image analysis of the endoscopic image 59 and detects the motion of the endoscope 40 at the time of examination. As a method of image analysis, for example, a known method such as pattern matching may be used.
The action analysis unit 217 performs an action analysis of the examiner with reference to the action learning DB 236 on the basis of the acquired motion information. The action analysis unit 217 evaluates the skill level of the examiner by the action analysis, and outputs an analysis result according to the skill level. For example, the action analysis unit 217 compares the acquired motion information of the examiner with the motion information associated with various skill levels included in the action learning DB 236. The action analysis unit 217 specifies a motion information ID having motion information with a high matching rate, and acquires a skill level associated with the motion information ID. The action analysis unit 217 specifies a skill level from the operation history of the endoscope 40, the time required for the operation, and the like on the basis of the motion information. For example, an examiner skilled in endoscopic examination can move the endoscope 40 from an examination point to an examination point in a short time without waste in the movement path of the endoscope 40.
The action analysis unit 217 compares the skill level obtained in real time from the analysis result with the skill level of the examiner acquired in advance, and determines whether to change the examination point. The action analysis unit 217 derives analysis result information including a determination result. The analysis result information is input to the examination point information acquisition unit 213. When it is necessary to change the examination point based on the acquired analysis result information, the examination point information acquisition unit 213 acquires the examination point information corresponding to the newly derived skill level.
Note that the action analysis unit 217 may determine the skill level of the examiner using another machine learning model. When the endoscopic image 59 is input, the action analysis unit 217 creates in advance a learned model that outputs information indicating the skill level and stores the learned model in the auxiliary memory device 23. The learned model is generated using data that collects endoscopic images at the time of examination by examiners of various skill levels. In a case where moving image data including a plurality of endoscopic images is input, the learned model outputs the skill level of the examiner estimated from the motion content obtained from the image information. The action analysis unit 217 acquires the information indicating the skill level from the endoscopic image 59 using the learned model. The information input to the learned model may include, for example, information such as an operation signal in addition to the endoscopic image 59.
The control unit 21 acquires examiner information of an examiner who performs an examination (Step S61). The control unit 21 receives an input operation from the keyboard 15, a mouse, or the like, and acquires an examiner ID. The control unit 21 refers to the examiner DB 232 to acquire examiner information including the skill level of the examiner.
The control unit 21 acquires endoscope type information (Step S11). The control unit 21 refers to the examination point DB 233 and acquires candidate information including a plurality of examination points according to the type of the endoscope and the skill level of the examiner (Step S13). The control unit 21 causes the display device 50 to display the input screen information including the candidate information (Step S15). Note that the control unit 21 may cause the display device 50 to display the candidate information based only on the endoscope type information regardless of the skill level.
The control unit 21 receives the selection of the examination point by the examiner (Step S17), and acquires the examination point ID of the selected check point (Step S19). The control unit 21 refers to the examination point DB 233 and acquires examination point information corresponding to the examination content based on the selected check point and skill level (Step S21). The examination point information includes examination point information such as a check point and a portion around the check point according to the skill level.
The endoscopic examination is started, and the endoscope 40 photographs an examination target portion. The control unit 21 performs the processing from Step S31 to Step S49 as in the first embodiment, and outputs a notification corresponding to the examination content. The examination point to be the output target of the notification varies depending on the skill level. For example, in a case where an untrained examiner performs an examination, not only a check point but also a portion around the check point is included in a notification output target. The notification is output in advance at the stage of reaching the vicinity of the check point, and the attention of the examiner is attracted so that the omission of the examination can be prevented. In a case where the skill level of the examiner is high, the notification before the check point is not output, and the notification is output only when the examination at the check point is completed.
Next, the control unit 21 detects motion information based on the endoscopic image 59 (Step S63). The control unit 21 detects motion information on the basis of image analysis of the endoscopic image 59, an operation signal, and the like. The control unit 21 performs action analysis with reference to the action learning DB 236 on the basis of the detected motion information. The control unit 21 specifies the skill level of the examiner reflecting the motion information in real time by the action analysis. The control unit 21 derives an analysis result including the specified skill level (Step S65).
The control unit 21 acquires information of whether to change the skill level of the examiner on the basis of the analysis result (Step S67). If the skill level included in the examiner information acquired in advance is different from the skill level included in the analysis result, or if the difference in skill level is equal to or greater than a threshold (Step S67: NO), the control unit 21 determines to change the skill level. The control unit 21 returns the processing to Step S21, and acquires examination point information based on the newly derived skill level. For example, in a case where the skill level is changed to a low numerical value, the examination point information is changed so that a notification is output to a portion around the check point in addition to the check point. On the other hand, when the skill level included in the examiner information acquired in advance is the same as the skill level included in the analysis result, or when the difference in skill level is less than the threshold (Step S67: YES), the control unit 21 determines not to change the skill level and advances the processing to the next Step S51.
The control unit 21 determines whether photographing has ended (Step S51). When it is determined that the photographing has not ended (Step S51: NO), the control unit 21 returns the processing to Step S31 and acquires a new endoscopic image. When it is determined that the photographing has ended (Step S51: YES), the control unit 21 ends a series of processing.
According to this embodiment, the notification regarding the examination content is output on the basis of the result of the action analysis performed in real time. Since the information regarding the implementation of the examination content is efficiently derived, it is possible to provide information according to the examination content of the examiner.
The embodiments disclosed as described above should be considered to be exemplary in all respects without being limited. The technical features described in the embodiments can be combined with each other, and the scope of the present invention is intended to include all modifications within the scope of the claims and the scope equivalent to the claims.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2019/030488 | 8/2/2019 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2021/024301 | 2/11/2021 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7566301 | Negishi | Jul 2009 | B2 |
| 20120078045 | Sasaki | Mar 2012 | A1 |
| 20150320514 | Ahn | Nov 2015 | A1 |
| 20160206283 | Ota | Jul 2016 | A1 |
| 20170071504 | Wang | Mar 2017 | A1 |
| 20170105609 | Nakayama | Apr 2017 | A1 |
| 20170126786 | Satkunarajah | May 2017 | A1 |
| 20180242818 | Kubo | Aug 2018 | A1 |
| 20180253839 | Zur | Sep 2018 | A1 |
| 20190213185 | Arroyo | Jul 2019 | A1 |
| 20190231444 | Tojo et al. | Aug 2019 | A1 |
| 20200090548 | Kimura | Mar 2020 | A1 |
| 20200352411 | Tojo | Nov 2020 | A1 |
| 20240016366 | Jeong | Jan 2024 | A1 |
| Number | Date | Country |
|---|---|---|
| 2007-014452 | Jan 2007 | JP |
| 2017-055954 | Mar 2017 | JP |
| 2017-108792 | Jun 2017 | JP |
| 2018069992 | Apr 2018 | WO |
| 2018211674 | Nov 2018 | WO |
| Entry |
|---|
| Official Communication issued in International Bureau of WIPO Patent Application No. PCT/JP2019/030488, dated Oct. 1, 2019. |
| Number | Date | Country | |
|---|---|---|---|
| 20220084194 A1 | Mar 2022 | US |
