Patent Application
20240277231
The present disclosure relates to a processing device, a processing program, and a processing method for processing an image of a subject imaged by a camera.
Conventionally, it has been known that a doctor observes a change in a state of an oral cavity of a user to diagnose, for example, a viral cold. Non-Patent Literature 1 (Miyamoto and WATANABE, “Consideration of Meanings and Values of Examination Findings of Pharynx (Influenza Follicles)”, Journal of the Japan Medical Journal 72 (1): 11 to 18 (2013)) reports that there is a pattern peculiar to influenza in lymphatic follicles appearing at the deepest part of the pharynx located in an oral cavity. Lymphatic follicles having this unique pattern are called influenza follicles, which are a characteristic sign of influenza, and are said to appear about 2 hours after onset. However, such a pharyngeal portion has been diagnosed by direct visual inspection by a doctor, and diagnosis using an image has not been made.
In view of the above technique, an object of the present disclosure is to provide a processing device, a processing program, and a processing method for determining a possibility of morbidity from a predetermined disease by using a subject determination image obtained by imaging an oral cavity of a user according to various embodiments.
According to one aspect of the present disclosure, there is provided “a processing device, comprising: at least one processor configured to execute a computer readable instructions so as to: acquire one or a plurality of determination images of a subject imaged by a camera from an imaging device including the camera for imaging an image of the subject including at least a part of an oral cavity of a user, acquire biometric detection information of the subject different from the one or the plurality of determination images from the imaging device, determine a possibility of morbidity for a predetermined disease on the basis of a learned determination model stored in a memory for determining the possibility of morbidity for the predetermined disease, the one or more acquired determination images, and the biometric detection information, and output information indicating the determined possibility of morbidity”.
According to one aspect of the present disclosure, there is provided “computer program product embodying computer readable instructions stored on a non-transitory computer-readable storage medium for causing a computer to execute a process by a processor, the computer configured to perform the steps of: acquiring one or a plurality of determination images of a subject imaged by a camera from an imaging device including the camera for imaging an image of the subject including at least a part of an oral cavity of a user; acquiring biometric detection information of the subject different from the one or the plurality of determination images from the imaging device; determining a possibility of morbidity for a predetermined disease on the basis of a learned determination model stored in a memory for determining the possibility of morbidity for the predetermined disease, the one or more acquired determination images, and the biometric detection information; and outputting information indicating the determined possibility of morbidity”.
According to one aspect of the present disclosure, there is provided “a processing method executed by at least one processor, the method for causing the processor in a computer to execute a process, the method comprising the computer readable instructions on the processor the steps of: acquiring one or a plurality of determination images of a subject imaged by a camera from an imaging device including the camera for imaging an image of the subject including at least a part of an oral cavity of a user; acquiring biometric detection information of the subject different from the one or the plurality of determination images from the imaging device; determining a possibility of morbidity for a predetermined disease on the basis of a learned determination model stored in a memory for determining the possibility of morbidity for the predetermined disease, the one or more acquired determination images, and the biometric detection information; and outputting information indicating the determined possibility of morbidity”.
According to the present disclosure, it is possible to provide a processing device, a processing program, and a processing method suitable for processing an image obtained by imaging the inside of an oral cavity for use in diagnosis of the inside of the oral cavity.
Note that the above effects are merely exemplary for convenience of description, and are not restrictive. In addition to or instead of the above effect, any effect described in the present disclosure or an effect obvious to those skilled in the art can be exhibited.
Various embodiments of the present disclosure will be described with reference to the accompanying drawings. Common components in the drawings are denoted by the same reference numerals.
A processing system 1 according to the present disclosure is mainly used for imaging the inside of a user's oral cavity to obtain a subject image. In particular, the processing system 1 is used to image the back of the throat of the oral cavity, specifically, the pharynx. Therefore, in the following description, a case where the processing system 1 according to the present disclosure is used for imaging the pharynx will be mainly described. However, the pharynx is an example of an imaging site, and as a matter of course, the processing system 1 according to the present disclosure can be suitably used for other sites such as a tonsil in the oral cavity.
The processing system 1 according to the present disclosure is used to determine a possibility of morbidity of a predetermined disease from a subject image obtained by imaging a subject including at least a pharyngeal region of an oral cavity of a user and other biometric detection information, and to diagnose or assist diagnosis of the predetermined disease. An example of the disease determined by the processing system 1 is influenza. Usually, the possibility of morbidity of the influenza is diagnosed by examining the user's pharynx or tonsil region or determining the presence or absence of findings such as follicles in the pharyngeal region. However, it is possible to perform diagnosis or assistance by determining the possibility of morbidity of influenza affection using the processing system 1 and outputting the result. The determination of the possibility of morbidity of influenza is an example. The processing system 1 can be suitably used in any case as long as it is determination of a disease in which a difference appears in findings in the oral cavity due to morbidity. Note that the difference in findings is not limited to those found by a doctor or the like and whose existence is medically known. For example, a difference that can be recognized by a person other than a doctor or a difference that can be detected by artificial intelligence or an image recognition technology can be suitably applied to the processing system 1. Examples of such diseases include, in addition to influenza, streptococcus infection, adenovirus infection, EB virus infection, mycoplasma infection, infections such as hand-foot and mouth disease, herpangina, and candidiasis, diseases exhibiting vascular disorders or mucosal disorders such as arteriosclerosis, diabetes, and hypertension, and tumors such as tongue cancer and pharyngeal cancer.
In the present disclosure, terms such as “determination” and “diagnosis” for a disease are used, but these do not necessarily mean a definite determination or diagnosis by a doctor. For example, the processing system 1 of the present disclosure may be used by a user or an operator other than a doctor, and the determination or diagnosis may be performed by the processing device 100 included in the processing system 1.
Furthermore, in the present disclosure, the user to be imaged by the imaging device 200 can include any human such as a patient, a subject, a diagnostic user, and a healthy person. Furthermore, in the present disclosure, the operator who holds the imaging device 200 and performs the imaging operation is not limited to a medical worker such as a doctor, a nurse, or a laboratory technician, and may include any person such as the user. The processing system 1 according to the present disclosure is typically assumed to be used in a medical institution. However, the present invention is not limited to this case, and the place of use may be any place such as the user's home, school, or workplace.
In the present disclosure, as described above, the subject may include at least a part of the oral cavity of the user. In addition, the disease to be determined may be any disease as long as a difference appears in findings in the oral cavity. However, in the following description, a case will be described in which the subject includes the pharynx or around the pharynx and the possibility of morbidity for influenza is determined as a disease.
Furthermore, in the present disclosure, the subject image or the determination image may be one or a plurality of moving images or one or a plurality of still images. As an example of the operation, when the power button is pressed, a through image is imaged by the camera, and the imaged through image is displayed on a display 203. Thereafter, when the operator presses an imaging button, one or a plurality of still images are imaged by the camera, and the imaged image is displayed on the display 203. Alternatively, when the user presses the imaging button, the imaging of a moving image starts, and an image imaged by the camera during that time is displayed on the display 203. Then, when the imaging button is pressed again, the imaging of the moving image is ended. In this manner, in a series of operations, various images such as a through image, a still image, and a moving image are imaged by the camera and displayed on the display. Furthermore, there is a case where the subject image or the determination image is a processed image subjected to various image processing for the display or the processing according to the present disclosure. However, the subject image and the determination image do not mean only specific images of these images, but may include all of these images imaged by the camera.
Furthermore, in the present disclosure, the biometric detection information is information different from the determination image imaged by the camera of the imaging device and used for the learned determination model among the information regarding a biological body of the user. Examples of the biometric detection information include detection data detected by an information detection sensor such as a temperature sensor, an exhalation sensor, a heart rate sensor, an infrared sensor, a near infrared sensor, an ultraviolet sensor, an acoustic sensor, and a combination thereof provided in the imaging device, information such as a user's body temperature, exhalation, respiratory sound, heart rate, blood vessel condition, and oxygen saturation obtained on the basis of each detected detection data and a subject image imaged by the camera, and a combination thereof. That is, the biometric detection information may be detection data itself detected by the information detection sensor, or may be information processed from the detected data. Furthermore, in a case where the information processed from the detection data is used as the biometric detection information, the processing may be performed in the imaging device, or the processing may be performed in the processing device.
Further, a desired information detection sensor is installed at the distal end of the imaging device 200 as necessary, and the biometric detection information of the user is detected by the information detection sensor. That is, when the imaging device 200 is inserted to the vicinity of the soft palate 713, the information detection sensor is also inserted together. Therefore, it is possible to obtain biometric detection information of the same region as the region in the oral cavity 710 imaged by the camera of the imaging device 200, which is the attention region, or a peripheral region thereof. That is, it is possible to obtain biological information of the attention region or its peripheral region. Furthermore, since the movement of the tongue 714 is restricted as described above, it is possible to eliminate an adverse effect by the tongue 714 or the like in the detection of the biometric detection information in the information detection sensor.
The imaged subject image (typically, an image including the pharynx 715) and biometric detection information are transmitted from the imaging device 200 to the processing device 100 communicably connected via a wired or wireless network. When the processor of the processing device 100 that has received the subject image processes the program stored in the memory, a determination image to be used for determination is selected from the subject image, and a possibility of morbidity for a predetermined disease is determined using the determination image and the biometric detection information. Then, the result is output to a display or the like.
Further, when the operator 600 grips the grip 202 in a direction in which the subject image is displayed in the normal direction on display 203, the imaging button 220 is disposed on the upper surface side of the grip. Therefore, when the operator 600 grips the imaging button 220, the operator 600 can easily press the imaging button with the index finger or the like.
The distal end of the imaging device 200 is inserted into the oral cavity of the user to image the oral cavity, particularly the pharynx. Furthermore, the imaging device 200 detects the biometric detection information of the subject on the basis of an information detection sensor provided at the distal end. The specific processing will be described later. The imaged subject image and biometric detection information are transmitted to the processing device 100 via a wired or wireless network.
Note that the processing system 1 can further include a placing table 400 as necessary. The imaging device 200 can be stably placed on the placing table 400. In addition, the placing table 400 is connected to a power source via a wired cable, so that power can be supplied from the power supply terminal of the placing table 400 to the imaging device 200 through the power supply port of the imaging device 200.
First, in the processing device 100, the processor 111 functions as a control unit that controls other components of the processing system 1 on the basis of a program stored in the memory 112. The processor 111 controls driving of the camera 211 and driving of the light source 212 on the basis of the program stored in the memory 112, stores the subject image and the biometric detection information received from the imaging device 200 in the memory 112, and processes the stored subject image and biometric detection information. Specifically, the processor 111 performs, based on the program stored in the memory 112, processing of acquiring the subject image of the subject from the camera 211, processing of acquiring the biometric detection information of the subject different from the determination image from the imaging device, processing of acquiring a candidate of the determination image by inputting the acquired subject image to a determination image selection model, processing of acquiring the determination image from the acquired candidate of the determination image based on a similarity between the images, processing of acquiring at least one of interview information and attribute information of a user, processing of determining a possibility of morbidity for influenza based on a learned determination model stored in the memory 112, one or a plurality of acquired determination images, and at least one of the interview information and attribute information of the user including the biometric detection information, processing of outputting information indicating the possibility of morbidity determined for diagnosing the morbidity for a predetermined disease or assisting the diagnosis, and the like. The processor 111 is mainly configured by one or a plurality of CPUs, but a GPU, an FPGA, or the like may be appropriately combined.
The memory 112 includes a RAM, a ROM, a nonvolatile memory, an HDD, and the like, and functions as a storage unit. The memory 112 stores instruction commands for various controls of the processing system 1 according to the present embodiment as programs. Specifically, the memory 112 stores a program to be executed by the processor 111 to performs processing of acquiring the subject image of the subject from the camera 211, processing of acquiring the biometric detection information of the subject different from the determination image from the imaging device, processing of acquiring a candidate of the determination image by inputting the acquired subject image to a determination image selection model, processing of acquiring the determination image from the acquired candidate of the determination image based on a similarity between the images, processing of acquiring at least one of the interview information and attribute information of the user, processing of determining a possibility of morbidity for influenza based on the learned determination model stored in the memory 112, one or a plurality of acquired determination images, and at least one of the interview information and attribute information of the user including the biometric detection information, processing of outputting information indicating the possibility of morbidity determined for diagnosing the morbidity for a predetermined disease or assisting the diagnosis, and the like. In addition to the program, the memory 112 stores the image management table for managing the subject image imaged by the camera 211 of the imaging device 200, the image, and the like, the user table for storing the attribute information, interview information, determination results, and the like of the user, and the like. In addition, the memory 112 stores each learned model such as a learned determination image selection model used for selecting a determination image from a subject image and the learned determination model for determining the possibility of morbidity from a disease from the determination image.
The input interface 113 functions as an input unit that receives an instruction input from the operator to the processing device 100 and the imaging device 200. Examples of the input interface 113 include physical key buttons such as an “imaging button” for instructing start/end of recording or detection of the biometric detection information by the imaging device 200, a “confirmation button” for performing various selections, a “return/cancel button” for returning to the previous screen or canceling a confirmation operation input, a cross key button for moving a pointer or the like output to the output interface 114, an on/off key for turning on/off the power of the processing device 100, and a character input key button for inputting various characters. Note that, as the input interface 113, it is also possible to use a touch panel provided to be superimposed on a display functioning as the output interface 114 and having an input coordinate system corresponding to the display coordinate system of the display. In this case, an icon corresponding to the physical key is displayed on the display, and the operator performs an instruction input via the touch panel to select each icon. A method of detecting the instruction input of the user by the touch panel may be any method such as a capacitance type or a resistive film type. The input interface 113 does not always need to be physically provided in the processing device 100, and may be connected as necessary via a wired or wireless network.
The output interface 114 functions as an output unit for outputting the subject image imaged by the imaging device 200 and the detected biometric detection information and outputting the result determined by the processor 111. Examples of the output interface 114 include a display including a liquid crystal panel, an organic EL display, a plasma display, or the like. However, the processing device 100 itself is not necessarily provided with a display. For example, an interface for connecting to a display or the like connectable to the processing device 100 via a wired or wireless network can also function as the output interface 114 that outputs display data to the display or the like.
The communication interface 115 functions as a communication unit for transmitting and receiving various commands related to start of imaging or the like, image data imaged by the imaging device 200, and detected biometric detection information to and from the imaging device 200 connected via a wired or wireless network. Examples of the communication interface 115 include various devices such as a connector for wired communication such as USB and SCSI, a transmission/reception device for wireless communication such as wireless LAN, Bluetooth (registered trademark), and infrared rays, and various connection terminals for a printed mounting board and a flexible mounting board.
Next, in the imaging device 200, the camera 211 functions as the imaging unit that generates the subject image by detecting reflected light reflected on the oral cavity that is the subject. The camera 211 includes, as an example, a CMOS image sensor, a lens system for realizing a desired function, and a drive system in order to detect the light. The image sensor is not limited to the CMOS image sensor, and other sensors such as a CCD image sensor can be used. In addition, it is also possible to additionally use an imaging element according to the band of the reflected light to be detected. For example, the camera 211 in which a CMOS image sensor, which is an imaging element that detects a wavelength in a near-infrared light band from a wavelength in a visible light band, and a sensor, which is an infrared light imaging element that detects a wavelength in an infrared light band, may be arranged adjacent to each other on an image sensor substrate can have an autofocus function, which is not particularly illustrated, and for example, it is preferable that the focus be set on the front of the lens so as to match a specific site. Furthermore, the camera 211 can have a zoom function, and is preferably set to capture an image at an appropriate magnification according to the size of the pharynx or the influenza follicle.
Here, it is known that there is a pattern peculiar to influenza in the lymphatic follicles appearing at the deepest portion of the pharynx located in the oral cavity. Lymphatic follicles having this unique pattern are called influenza follicles, which are a characteristic sign of influenza, and are said to appear about 2 hours after onset. As described above, the processing system 1 of the present embodiment is used to determine the possibility of morbidity for influenza of the user, for example, by imaging the pharynx of the oral cavity and detecting the follicles. Therefore, when the imaging device 200 is inserted into the oral cavity, the distance between the camera 211 and the subject becomes relatively short. Therefore, the camera 211 preferably has an angle of view (2θ) at which a value calculated by [(distance from the distal end portion of the camera 211 to a pharyngeal rear wall)*tan θ] is 20 mm or more in the vertical direction and 40 mm or more in the horizontal direction. By using a camera having such an angle of view, even when the camera 211 and the subject are close to each other, it is possible to image a wider range. That is, as the camera 211, a normal camera can be used, but a camera called a wide-angle camera or an ultra-wide-angle camera can also be used.
Furthermore, in the present embodiment, a main subject imaged by the camera 211 is an influenza follicle formed in the pharynx or a pharyngeal portion. Since the pharynx is generally formed deep in a depth direction, when a depth of field is shallow, a focus is shifted between an anterior portion of the pharynx and a posterior portion of the pharynx, and it becomes difficult to obtain a subject image suitable for use in determination in the processing device 100. Thus, the camera 211 has a depth of field of at least 20 mm or greater, preferably 30 mm or greater. By using the camera having the depth of field, it is possible to obtain the subject image having the focus at any site from the anterior portion of the pharynx to the posterior portion of the pharynx.
Furthermore, the camera 211 can also function as an information detection sensor for detecting the biometric detection information. For example, it is possible to obtain heart rate information by detecting vibration due to a pulse on a surface of an oral tissue from the subject image imaged by the camera 211. In addition, it is possible to obtain blood vessel state information by detecting a blood vessel pattern on the oral surface from the subject image imaged by the camera 211. That is, it is possible to cause the camera 211 to function as an information detection sensor and acquire the imaged subject image or various types of information detected from the subject image as the biometric detection information.
The light source 212 is driven by an instruction from the processing device 100 or the imaging device 200, and functions as a light source unit for irradiating the oral cavity with light. The light source 212 includes one or more light sources. In the present embodiment, the light source 212 includes one or a plurality of LEDs, and light having a predetermined frequency band is emitted from each LED in an oral direction. As the light source 212, light having a desired band from among an ultraviolet light band, a visible light band, and an infrared light band, or a combination thereof is used. In the case of irradiating a plurality of bands, a plurality of LEDs set in advance to emit light in each band are switched and displayed.
In particular, near infrared rays have high permeability of a biological body and are suitable for observing a blood vessel state or the like in the biological body, but it is possible to obtain blood vessel state information of an irradiation region by irradiating near infrared rays and detecting reflected light thereof. In addition, the ultraviolet ray has low tissue transmittance and is suitable for detecting a fine structural change of the biological body surface, but it is possible to obtain structural information of the biological body surface by irradiating the ultraviolet ray and detecting reflected light thereof. Furthermore, the ultraviolet light is used as excitation light of a fluorescent substance that binds to a specific cell (for example, a tumor cell), and it is possible to obtain tumor information of an irradiation region by emitting light in an ultraviolet light band and detecting excitation light from the fluorescent substance.
The information detection sensor 232 functions as a detection unit that is inserted into the oral cavity of the user and acquires the biometric detection information that is information related to the biological body of the user. Examples of the information detection sensor 232 include a temperature sensor, an exhalation sensor, a heart rate sensor, a pulse oximeter sensor, an infrared sensor, a near infrared sensor, an ultraviolet sensor, an acoustic sensor, or a combination thereof. Note that, in some cases, the camera 211 may function as the information detection sensor 232. Examples of the biometric detection information acquired by the information detection sensor 232 include detection data detected by each sensor (including a case where the camera 211 functions as the information detection sensor 232), and the user's body temperature, exhalation, heartbeat, and vascular information obtained on the basis of the detected detection data, and combinations thereof.
When a temperature sensor is used as the information detection sensor 232, a thermistor sensor, a thermocouple sensor, a resistor sensor, a digital temperature sensor, an infrared sensor, or a combination thereof can be used. Among them, a digital temperature sensor or an infrared sensor suitable for non-contact measurement is more preferable. In diagnosis or the like by doctors, the body temperature of a normal user is input as the interview information on the basis of reporting by the user or measurement in a hospital. However, in this case, the measurement position and the measurement method may vary depending on the person who performs the measurement. For example, there is a thermometer capable of measuring a temperature of a skin surface such as a forehead and a wrist in a non-contact manner, but in winter when infectious diseases such as influenza are likely to spread, the thermometer is particularly susceptible to the influence of the outside temperature, and the measurement error increases. However, the oral cavity is hardly affected by the outside air temperature, and more accurate information can be input as the interview information by using the temperature sensor disposed in the imaging device 200 inserted into the oral cavity. That is, temperature information detected by the temperature sensor is used as the biometric detection information.
In particular, when an infrared sensor is used, infrared image data is obtained on the basis of infrared energy obtained by the sensor or temperature information obtained by conversion from the infrared energy. Furthermore, segmentation image data segmented for each site by segmentation such as semantic segmentation is acquired for a separately imaged subject image. Then, by superimposing the infrared image data and the segmentation image data, it is possible to accurately detect the temperature of the measurement region or the attention region. Note that the segmentation can also be performed on the infrared image data, and the temperatures of the measurement region and the attention region can be accurately detected by using the infrared image data after the segmentation. As described above, by using the infrared sensor as the information detection sensor 232, a particularly accurate temperature of the attention region can be used as the biometric detection information.
When the ultraviolet sensor is used as the information detection sensor 232, for example, a fine structure of a biological body surface can be detected. Ultraviolet light is generally less tissue transmittance and may be mostly reflected at the surface of biological tissue. Therefore, it is suitable for detecting a fine structural change of the biological body surface as compared with a wavelength in another band such as visible light. For example, in infectious diseases such as influenza, characteristic structural changes such as follicles occur in the pharynx as described above. Therefore, by detecting such a structural change, it is possible to determine the presence or absence of a disease. In addition, by superimposing the ultraviolet image data obtained by the ultraviolet sensor and the segmentation image data of the subject image, it is possible to detect a more accurate structural change of the measurement region and the attention region. That is, the ultraviolet detection data detected by the ultraviolet sensor or the structural information of the biological body surface obtained therefrom is used as the biometric detection information.
When an exhalation sensor is used as the information detection sensor 232, it is possible to use a sensor device capable of detecting a specific gas component such as ammonia or acetone contained in exhalation. Here, there is a relationship between the amount of a specific gas component, such as ammonia or acetone, which causes halitosis, contained in exhalation and human diseases. For example, a sweet smell is associated with infection, acetone is associated with diabetes, a gangrene smell of ammonia or protein is associated with specific tumor or liver disease, isoprene is associated with hypoglycemia or sleep disorder, methylmercaptan is associated with oral bacteria or liver disease, carbon monoxide is associated with stress, ethanol is associated with drinking, trimethylamine is associated with kidney disease, nitric oxide is associated with asthma, and the like. Therefore, it is possible to estimate the disease by detecting a gas component that causes these from the exhalation. That is, gas component data measured by the exhalation sensor and disease information estimated therefrom are used as biometric detection information.
In a case where a heart rate sensor is used as the information detection sensor 232, a sensor that can be measured by being in contact with a biological body is also known, but an image sensor of the camera 211 that can be detected in a non-contact manner, a microwave or millimeter wave detection sensor, or a combination thereof can be used. For example, it is possible to detect a temporal change in the subject surface position due to pulsation on the basis of the subject image detected by the image sensor and estimate the heart rate of the user. In addition, it is possible to obtain the luminance of the G (green) component in each frame constituting the obtained subject image 8 moving image), generate the luminance waveform of the G component in the moving image, and estimate the heart rate from the peak value. This method is a method utilizing that hemoglobin in blood absorbs green light. That is, the heart rate measured by the heart rate sensor is used as the biometric detection information.
In a case where a near infrared sensor is used as the information detection sensor 232, for example, it is possible to detect a blood vessel state passing through a surface or a deep portion of a subject. Here, diabetes and hypertension inhibit blood flow or give structural damage to blood vessels themselves. Furthermore, vascular disorders caused by these cause blood flow disorders, making people more susceptible to infections. Therefore, detecting the state of the blood vessel is useful for diagnosis of morbidity such as diabetes, hypertension, and infection. That is, the image data of the near-infrared image measured by the near infrared sensor and the blood vessel state information estimated therefrom are used as the biometric detection information.
In a case where a pulse oximeter sensor is used as the information detection sensor 232, for example, an oxygen saturation of blood can be estimated by irradiating the biological body surface with a wavelength in a near-infrared light band and a wavelength in an infrared light band and estimating the amounts of oxygenated hemoglobin and deoxygenated hemoglobin from the reflection. That is, the oxygen saturation of blood measured by the pulse oximeter sensor is used as biometric detection information.
In a case where an acoustic sensor is used as the information detection sensor 232, typically, a microphone can be used. Here, respiratory sound can be an important indicator for diagnosing the possibility of morbidity, particularly in diseases of the respiratory system and the like. In addition, in a specific disease such as bronchiectasis, acoustic data in the oral cavity is important information as oral auscultation data, such as a sound like blisters. Therefore, the acoustic sensor is disposed at the distal end of the imaging device 200 to obtain detection data of a respiratory sound of the user and a sound in the oral cavity. That is, detection data of the respiratory sound of the user or the sound in the oral cavity detected by the acoustic sensor is used as the biometric detection information.
The processor 213 functions as a control unit that controls other components of the imaging device 200 on the basis of the program stored in the memory 214. The processor 213 controls driving of the camera 211 and the information detection sensor 232 and driving of the light source 212 on the basis of the program stored in the memory 214, and controls storage of the subject image imaged by the camera 211 in the memory 214 and storage of the biometric detection information detected by the information detection sensor 232 in the memory 214. In addition, the processor 213 controls output of the subject image, the biometric detection information, and the user information stored in the memory 214 to the display 203 and transmission thereof to the processing device 100. The processor 213 mainly includes one or a plurality of CPUs, but may be appropriately combined with other processors.
The memory 214 includes a RAM, a ROM, a nonvolatile memory, an HDD, and the like, and functions as a storage unit. The memory 214 stores, as a program, instruction commands for various control of the imaging device 200. In addition to the program, the memory 214 stores the subject image imaged by the camera 211, the biometric detection information detected by the information detection sensor 232, various kinds of information of the user, and the like.
The display panel 215 is provided on the display 203 and functions as a display unit for displaying the subject image imaged by the imaging device 200 and the biometric detection information detected by the information detection sensor 232. The display panel 215 is constituted by a liquid crystal panel, but is not limited to the liquid crystal panel, and may be constituted by an organic EL display, a plasma display, or the like.
The input interface 210 functions as an input unit that receives a user's instruction input to the processing device 100 and the imaging device 200. Examples of the input interface 210 include physical key buttons such as an “imaging button” for instructing start/end of recording by the imaging device 200 and detection of the biometric detection information, a “power button” for turning on/off the power of the imaging device 200, a “confirmation button” for performing various selections, a “return/cancel button” for returning to the previous screen and canceling a confirmation operation input, and a cross key button for moving an icon or the like displayed on the display panel 215. Note that these various buttons and keys may be physically prepared, or may be selectable using a touch panel or the like displayed as an icon on the display panel 215 and arranged as the input interface 210 in a superimposed manner on the display panel 215. A method of detecting the instruction input of the user by the touch panel may be any method such as a capacitance type or a resistive film type.
The communication interface 216 functions as a communication unit for transmitting and receiving information to and from the imaging device 200 and/or another device. Examples of the communication interface 216 include various devices such as a connector for wired communication such as USB and SCSI, a transmission/reception device for wireless communication such as wireless LAN, Bluetooth (registered trademark), and infrared rays, and various connection terminals for a printed mounting board and a flexible mounting board.
The main body 201 is formed in a columnar shape having a hollow cylindrical shape whose cross section is a perfect circle. A wall portion 224 may be made of any material as long as the material can guide light into the wall portion, and as an example, the wall portion can be obtained using a thermoplastic resin. As the thermoplastic resin, polyolefin-based resins such as chain polyolefin-based resins (polypropylene-based resins and the like) and cyclic polyolefin-based resins (norbornene-based resins and the like), cellulose ester-based resins such as triacetyl cellulose and diacetyl cellulose, polyester-based resins, polycarbonate-based resins, (meth) acrylic resins, polystyrene-based resins, or mixtures and copolymers thereof are used. That is, the wall portion 224 of the main body 201 functions as a light guide body for guiding the light emitted from the light source in the oral cavity or toward the diffuser plate.
Since the main body 201 is formed in a hollow shape, an accommodation space 223 is formed on the inner surface thereof by the wall portion 224. The camera 211 is accommodated in the accommodation space 223. Note that the main body 201 only needs to be formed in a columnar shape having the accommodation space 223. Therefore, the accommodation space 223 does not need to have a cylindrical shape having a perfect circular cross section, and may have an elliptical or polygonal cross section. Further, the inside of the main body 201 does not necessarily need to be formed hollow.
The distal end of the grip 202 is connected to the proximal end 225 of the main body 201. The user grips the grip 202 and performs operations such as insertion and removal of the imaging device 200. The grip 202 is constituted by a columnar body having a predetermined length in a direction substantially parallel to the direction H in which the grip is inserted into the oral cavity, that is, along the longitudinal direction of the main body 201, and the grip is disposed on the same straight line as the main body 201 in the direction H. In the present embodiment, the cross section in the vertical direction is formed to be substantially oval, but the cross section is not necessarily oval, and may be a perfect circle, an ellipse, or a polygon.
The grip 202 includes a connecting portion 230 formed at a position closest to the proximal end 225 of the main body 201, and is connected to the main body 201 via the connecting portion 230. An engagement protrusion 217 (217-1 to 217-4) for positioning the auxiliary tool 300 and a positioning protrusion 218 are provided on the outer periphery of the connecting portion 230. The engagement protrusion 217 engages with the engagement protrusion 318 (318-1 to 318-4) provided in the auxiliary tool 300. Further, the positioning protrusion 218 is inserted into an insertion hole 321 provided in the auxiliary tool 300 to position the imaging device 200 and the auxiliary tool 300 with each other. In the present embodiment, in the engagement protrusion 217 of the main body 201, a total of four engagement protrusions (engagement protrusions 217-1 to 217-4) are arranged at equal intervals on the surface of the grip 202 in the vicinity of the proximal end 225 of the main body 201. One positioning protrusion 218 is disposed between the engagement protrusions 217 on the surface of the grip 202 and near the proximal end 225 of the main body 201. However, the present invention is not limited thereto, and only one of the engagement protrusion 217 and the positioning protrusion 218 may be disposed. The number of the engagement protrusions 217 and the positioning protrusions 218 may be any number as long as the number is one or more.
The grip 202 includes the imaging button 220 at a position close to the proximal end 225 of the main body 201 on the upper surface of the grip, that is, near the distal end of the grip 202 in the insertion direction H in the oral cavity. As a result, when the operator 600 grips the imaging button 220, the operator 600 can easily press the imaging button with the index finger or the like. The power button 221 is disposed on the upper surface of the grip 202 at a position close to the display 203, that is, at a position opposite to the imaging button 220 of the grip 202. As a result, it is possible to prevent the power button 221 from being erroneously pressed when the operator 600 grips and captures an image.
The display 203 has a substantially rectangular parallelepiped shape as a whole, and is disposed on the same straight line as the main body 201 in the direction H. In addition, the display 203 includes the display panel 215 on a surface in the direction (that is, the direction toward the user is) opposite to the direction H in which the grip is inserted into the oral cavity. Therefore, the display 203 is formed such that the surface including the display panel is substantially perpendicular to the longitudinal direction of the main body 201 and the grip 202 formed to be substantially parallel to the direction H in which the main body and the grip are inserted into the oral cavity. The grip 202 is connected to the grip 202 on the side opposite to the oral cavity on the surface facing the surface including the display panel. Note that the shape of the display is not limited to a substantially rectangular parallelepiped shape, and may be any shape such as a cylindrical shape.
A diffuser plate 219 is disposed at the distal end 222 of the main body 201, and diffuses light emitted from the light source 212 and passing through the main body 201 toward the inside of the oral cavity. The diffuser plate 219 has a shape corresponding to a cross-sectional shape of a portion of the main body 201 configured to be able to guide light. In the present embodiment, the main body 201 is formed in a hollow cylindrical shape. Therefore, the cross section of the diffuser plate is also formed in a hollow shape corresponding to the shape of diffuser plate 219.
The camera 211 is used to generate the subject image by detecting reflected light diffused from the diffuser plate 219, emitted into the oral cavity, and reflected on the subject. The camera 211 is disposed on the same straight line as the main body 201 in the direction H in the inner surface of the wall portion 224 of the main body 201, that is, in the accommodation space 223 formed inside the main body 201. Although only one camera 211 is described in the present embodiment, the imaging device 200 may include a plurality of cameras. By generating the subject image using a plurality of cameras, the subject image includes information regarding a three-dimensional shape. Furthermore, in the present embodiment, the camera 211 is arranged in the accommodation space 223 of the main body 201, but may be arranged at the distal end 222 of the main body 201 or the main body 201 (may be inside the main body 201 or on the outer periphery of the main body 201).
Here, the camera 211 can also function as the information detection sensor 232. For example, reflected light of visible light emitted from the light source 212 can be detected to obtain the subject image as the biometric detection information. Furthermore, the camera 211 can detect reflected light of near infrared rays and obtain the near-infrared image as the biometric detection information. Furthermore, the camera 211 can detect infrared rays emitted from the biological body and obtain the infrared image as the biometric detection information. Furthermore, the camera 211 can obtain the ultraviolet image as the biometric detection information by detecting reflected light of ultraviolet rays.
The information detection sensor 232 is a sensor for acquiring information on a biological body that is a user. As such a sensor, the camera 211 may function, or may be arranged separately from the camera 211. In the example of
Note that the light sources 212-1 to 212-4 may be configured to be independently controlled. For example, by illuminating some of the light sources 212-1 to 212-4, it is possible to include a shade of the subject (influenza follicle or the like) having a three-dimensional shape in the subject image. As a result, the subject image includes information on the three-dimensional shape of the subject, the subject can be more clearly determined, and the possibility of morbidity for the influenza can be more accurately determined by the determination algorithm. In addition, light may be emitted from the light sources 212-1 to 212-4 separately for each band such as a visible light band, a near-infrared light band, an infrared light band, and an ultraviolet light band.
Furthermore, in the present embodiment, the light sources 212-1 to 212-4 are arranged on the proximal end 225 side of the main body 201, but may be arranged at the distal end 222 of the main body 201 or the main body 201 (may be inside the main body 201 or on the outer periphery of the main body 201).
In the present embodiment, the diffuser plate 219 is used to prevent light emitted from the light source 212 from illuminating only a part of the oral cavity and to generate uniform light. Therefore, as an example, a fine lens array is formed on the surface of the diffuser plate 219, and a lens-shaped diffuser plate having an arbitrary diffusion angle is used. Alternatively, a diffuser plate that can diffuse light by another method, such as a diffuser plate that realizes a light diffusion function by fine irregularities randomly arranged on the surface, may be used. Further, the diffuser plate 219 may be configured integrally with the main body 201. For example, the present invention can be realized by a method of forming fine irregularities on the distal end part of the main body 201.
Further, in the present embodiment, the diffuser plate 219 is disposed on the distal end 222 side of the main body 201. However, the present invention is not limited thereto, and any position may be employed as long as the position is between the light source 212 and the oral cavity to be irradiated with the light source. For example, the diffuser plate may be disposed at the distal end 222 of the main body 201 or the main body 201 (which may be inside the main body 201 or on the outer periphery of the main body 201).
According to
According to
Each piece of information stored in the biometric detection information table can be used as the interview information of the user of each piece of user ID information.
According to
Note that the attribute information and the interview information do not need to be input by the user or the operator each time, and may be received from, for example, an electronic medical record device or another terminal device connected via a wired or wireless network. Alternatively, it may be acquired by analyzing the subject image imaged by the imaging device 200. Furthermore, although not particularly illustrated in
According to
Under the control of the processor 111, the processing device 100 generates a two-dimensional code in which the user ID information generated in advance is recorded, and stores the two-dimensional code in the memory 112 (S14). Then, the processing device 100 outputs the generated two-dimensional code via the output interface 114 (S15).
Next, the imaging device 200 activates the camera 211 and the like when an input to the input interface 210 (for example, a power button) by the operator is received (S21). Then, the two-dimensional code output via the output interface 114 is imaged by the activated camera 211, so that the imaging device 200 reads the user ID information recorded in the two-dimensional code (S22).
Next, the operator covers the distal end of the imaging device 200 with the auxiliary tool 300, and inserts the imaging device 200 into the oral cavity of the user to a predetermined position. When receiving the input to the input interface 210 (for example, the imaging button) by the operator, the imaging device 200 starts imaging the subject image of the subject including at least a part of the oral cavity (S23). Furthermore, the imaging device 200 processes the imaged subject image, and with the fact that the camera 211 captures the subject (in particular, the pharynx) in the angle of view as a trigger, detection of the biometric detection information by the information detection sensor 232 starts (S24). When the imaging of the subject image and the detection of the biometric detection information end, the imaging device 200 stores the imaged subject image and the back-to-back detection information in the memory 214 in association with the user ID information read from the two-dimensional code, and outputs the imaged subject image to the display panel 215 of the display (S25). Then, when receiving an input of termination of imaging by the operator via the input interface 210, the imaging device 200 transmits the subject image and the biometric detection information (T21) stored via the communication interface 216 to the processing device 100 in association with the user ID information.
Next, upon receiving the subject image and the biometric detection information via the communication interface 115, the processing device 100 stores the subject image and the biometric detection information in the memory 112, and registers the subject image and the biometric detection information in the image management table on the basis of the user ID information. The processing device 100 selects a determination image to be used for determination of the possibility of morbidity for influenza from the stored subject image (S31). In addition, although not particularly illustrated, the received biometric detection information is processed into biometric detection information that can be used for determination processing and stored. When the determination image is selected, the processing device 100 uses the selected determination image and the stored biometric detection information to execute processing of determining the possibility of morbidity for influenza (S32). When the determination result is obtained, the processing device 100 stores the determination result obtained in association with the user ID information in the user table and outputs the determination result via the output interface 114 (S33). Thus, the processing sequence is ended.
According to
Returning to
Meanwhile, when the processor 111 determines that the mode is selected by accepting the input to either the imaging mode icon 11 or the determination result confirmation mode icon 12 illustrated in
Meanwhile, when determining that the imaging mode icon 11 illustrated in
Next, the processor 111 displays, on the output interface 114, a screen for receiving the input of the interview information of the user (not illustrated). The screen includes items such as the body temperature, the heart rate, the medication status, and the presence or absence of subjective symptoms of the user, which need to be input as the interview information, and an input box for inputting an answer to each item. Then, the processor 111 acquires information input to each input box via the input interface 113 as the interview information, and stores the information in the user table in association with the user ID information (S115).
Note that the case where the attribute information and the interview information are input by the processing device 100 has been described. However, the present invention is not limited thereto, and the information may be acquired by receiving information input to an electronic medical record device or another terminal device connected via a wired or wireless network.
Next, the processor 111 refers to the user table, reads the user ID information corresponding to the user to which these pieces of information are input, and generates a two-dimensional code in which the user ID information is recorded (S116). The processor 111 stores the generated two-dimensional code in the user table in association with the user ID information, and outputs the two-dimensional code via the output interface 114 (S117). Thus, the processing flow ends.
Here,
According to
Meanwhile, when determining that the input by the operator has been accepted, the processor 213 outputs a standby screen to the display panel 215 (S212). The standby screen (not illustrated) includes a through image imaged via the camera 211. Then, when the operator moves the imaging device 200 so that the two-dimensional code output to the output interface 114 of the processing device 100 is included in the angle of view of the camera 211, the processor 213 images the two-dimensional code by the camera 211 (S213). When the two-dimensional code is imaged, the processor 213 reads the user ID information recorded in the two-dimensional code, and stores the read user ID information in the memory 214 (S214). Then, the processor 213 outputs the standby screen to the display panel 215 again (S215).
Next, the operator covers the distal end of the imaging device 200 with the auxiliary tool 300, and inserts the imaging device 200 into the oral cavity of the user to a predetermined position. Then, when the imaging start operation by the operator is received via the input interface 210 (for example, the imaging button), the processor 213 controls the camera 211 to 213 start imaging the subject image of the subject (S216). The imaging of the subject image is performed by pressing an imaging button to collectively image a certain number of images (for example, 30 sheets) at regular intervals. When the imaging of the subject image ends, the processor 213 stores the imaged subject image in association with the user ID information read in the memory 214.
Furthermore, the imaging device 200 processes the subject image imaged as the through image, for example, and determines whether or not the camera 211 has imaged the subject (particularly, pharynx) in an angle of view. As an example, the imaged subject image is input to the learned determination image selection model, and the processor 213 determines whether or not the candidate image is obtained. Then, the information detection sensor 232 is turned on at a timing when it is determined that the candidate image is obtained, and detection of the biometric detection information starts (S217). As a result, it is possible to start detection at the timing when the imaging device 200 captures the subject, that is, in a state where the information detection sensor 232 is in an appropriate direction and position. When driving of each sensor arranged as the information detection sensor 232 starts, the processor 213 detects desired biometric detection information. When the detection of the biometric detection information ends, the processor 213 stores the detected biometric detection information in the memory 214 in association with the user ID information. Note that the use of the learned determination image selection model is an example. For example, segmentation may be performed on the subject image by semantic segmentation or the like to determine whether the pharynx or the like is included in the center of the image. Furthermore, the determination using the learned determination image selection model and the determination by segmentation may be performed by the processor 213 of the imaging device 200, or may be performed by the processor 112 of the processing device 100 that has received the subject image, and the determination result may be transmitted to the imaging device 200. In addition, the detection itself by the information detection sensor 232 may be started at the timing when the imaging button is pressed or may be constantly detected, and only the biometric detection information synchronized with the timing when the candidate image is obtained may be cut out.
Then, the processor 213 outputs the subject image and the biometric detection information stored in the display panel 215 (S218).
Here, the operator takes out the imaging device 200 from the oral cavity together with the auxiliary tool 300, checks the subject image and the biometric detection information output to the display panel 215, and can input a re-imaging instruction when a desired image or desired biometric detection information cannot be obtained. Therefore, the processor 213 determines whether the input of the re-imaging instruction by the operator has been received via the input interface 210 (S219). When receiving the input of the re-imaging instruction, the processor 213 displays the standby screen in S215 again to enable imaging of the subject image and detection of the biometric detection information.
Meanwhile, when the input of the re-imaging instruction is not received and the imaging device 200 is returned to the placing table 400 by the operator and the instruction to finish the imaging is received from the processing device 100, the processor 213 transmits the subject image, the biometric detection information, and the user ID information associated with the subject image stored in the memory 214 to the processing device 100 via the communication interface 216 (S220). Thus, the processing flow ends.
According to
When receiving the selection of the user to be determined via the input interface 113, the processor 111 reads the attribute information associated with the user ID information of the user from the user table in the memory 112 (S313). Similarly, the processor 111 reads the interview information associated with the user ID information on the user to be determined from the user table in the memory 112 (S314). As the interview information, the information stored in the biometric detection information table can be read as the interview information. That is, temperature (body temperature), exhalation, respiratory sound, heart rate, oxygen saturation, a near-infrared image (or a blood vessel state estimated from the image), an ultraviolet image (or structural information of a biological body surface obtained from the image), and the like can be acquired as the interview information from the biometric detection information table without requiring input by the user or the like.
Next, the processor 111 reads the subject image associated with the user ID information of the user selected from the memory 112, and executes the selection processing of the determination image used to determine the possibility of morbidity for influenza (S315: details of this selection processing will be described later). Then, the processor 111 executes processing of determining the possibility of morbidity for influenza based on the selected determination image and the biometric detection information used as the interview information (S316: details of this determination processing will be described later). When the determination result is obtained by the determination processing, the processor 111 stores the determination result in the user table in association with the user ID information, and outputs the determination result via the output interface 114 (S317). Thus, the processing flow ends.
According to
Here,
According to
When the learning subject image and the label information associated with the learning subject image are obtained, the processor executes a step of performing machine learning of the selection pattern of the determination image using the learning subject image and the label information (S514). As an example, the machine learning is performed by assigning a set of the learning subject image and the label information to a neural network configured by combining neurons, and repeating learning while adjusting parameters of each neuron so that an output of the neural network becomes the same as the label information. Then, a step of acquiring a learned determination image selection model (for example, a neural network and a parameter) is executed (S515). The acquired learned determination image selection model may be stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.
Returning to
Next, the processor 111 executes processing of selecting a determination image based on the similarity from the selected candidate images (S413). Specifically, the processor 111 compares the obtained candidate images to calculate the similarity between the candidate images. Then, the processor 111 selects, as a determination image, a candidate image determined to have low similarity to another candidate image. Such similarity between the candidate images is calculated by a method (Bag-of-Keypoints method) using local feature amounts in the candidate images, a method using earth mover's distance (EMD), a method using support vector machine (SVM), a method using Hamming distance, a method using cosine similarity, or the like.
In this manner, by calculating similarity between the obtained candidate images and selecting a candidate image determined to have low similarity with respect to other candidate images, the subject images having different visual fields are selected as determination images. As a result, it is possible to perform the determination processing on the basis of more various types of information as compared with a case where the subject image obtained in the same visual field is used as the determination image, and thus, it is possible to further improve the determination accuracy. Specifically, even when a partial region of the pharynx is hidden by the uvula in a certain determination image, since a partial region of the pharynx hidden in another determination image having a different visual field is visible, it is possible to prevent overlooking of important features such as influenza follicles.
Next, the processor 111 registers the candidate image selected based on the similarity as a determination image in the image management table (S414).
Here, each of the subject image, the candidate image, and the determination image may be one image or a plurality of images. However, as an example, it is preferable to select a candidate image group from a subject image group of about 5 to 30, for example, and finally obtain a determination image group of about 5. This is because there is a high possibility that a better determination image can be obtained by selecting the determination image from a large number of subject images. In addition, by using a plurality of determination image groups for determination processing to be described later, determination accuracy can be further improved as compared with a case where only one determination image is used. Furthermore, as another example, every time the subject image is imaged, the imaged subject image may be transmitted to the processing device 100, and then the candidate image and the determination image may be selected, or the candidate image and the determination image may be selected in the imaging device 200, and the imaging may be terminated when a predetermined number (for example, about 5) of the determination images have been acquired. In this way, it is possible to minimize the time related to the imaging of the subject image while maintaining the improvement in the determination accuracy as described above. That is, discomfort to the user such as vomiting reflex can be reduced.
According to
First, the processing of acquiring the first positive rate will be described. The processor 111 reads the determination image associated with the user ID information of the user to be determined from the memory 112 (S611). Then, the processor 111 performs predetermined preprocessing on the read determination image. Such preprocessing is selected from filter processing such as a bandpass filter including a high-pass filter and a low-pass filter, an averaging filter, a Gaussian filter, a Gabor filter, a Canny filter, a Sobel filter, a Laplacian filter, a median filter, and a bilateral filter, blood vessel extraction processing using a Hessian matrix and the like, segmentation processing of a specific region (for example, follicles) using machine learning, trimming processing for a segmented region, defogging processing, super-resolution processing, and combinations thereof according to purposes such as high definition, region extraction, noise removal, edge enhancement, image correction, and image conversion. In this manner, by executing the preprocessing, it is possible to improve the determination accuracy by extracting or emphasizing in advance an attention region that is important in diagnosis for a disease, such as follicles in influenza.
Here, a case where the defogging processing, the super-resolution processing, and the segmentation processing are performed as such preprocessing will be specifically described below as an example. First, as an example, the defogging processing uses a learned defogging image model obtained by giving a set of learning deterioration images to which fogging is added from the learning subject image and the learning subject image by application of a fogging addition filter or the like to a learning device and performing machine learning. The processor 111 inputs the read determination image as an input to the learned defogging image model stored in the memory 112, and acquires the determination image from which the fogging has been removed as an output. Furthermore, in the super-resolution processing, a set of a high-resolution image of the subject and a low-resolution image obtained by performing degradation processing such as scale reduction processing or blurring processing on the high-resolution image is given to a learning device as learning images, and a learned super-resolution image model obtained by machine learning is used. The processor 111 inputs the determination image subjected to the defogging processing to the learned super-resolution image model stored in the memory 112, and acquires the determination image subjected to the super-resolution processing as an output. Furthermore, in the segmentation processing, a set of the learning subject image and the position information of the label obtained by assigning the label to the attention region (for example, follicles) on the basis of the operation input by the doctors to the learning subject image is given to the learning device, and the learned segmentation image model obtained by machine learning is used. The processor 111 inputs the learned segmentation image model stored in the memory 112 using the determination image on which the super-resolution processing has been performed as an input, and acquires the determination image in which the attention region (for example, follicles) has been segmented. Then, the processor 111 stores the determination image preprocessed in this manner in the memory 112. Here, the defogging processing, the super-resolution processing, and the segmentation processing are performed in this order, but may be performed in any order, or only at least one of the processing may be performed. In addition, although the processing using the learned model has been exemplified in any processing, processing such as a defogging filter, scale enlargement processing, and sharpening processing may be used.
Then, the processor 111 gives the determination image after the preprocessing to the feature amount extractor as an input (S613), and acquires the image feature amount of the determination image as an output (S614). Furthermore, the processor 111 gives the feature amount of the acquired determination image to the classifier as an input (S615), and acquires the first positive rate indicating a first possibility of morbidity for influenza as an output (S616). Note that, in the feature amount extractor, a predetermined number of feature amounts such as the presence or absence of follicles and the presence or absence of redness in the determination image can be obtained as vectors. As an example, 1024 dimensional feature amount vectors are extracted from the determination image, and these are stored as feature amounts of the determination image.
Here,
According to
When the determination image for learning and the correct answer label information associated with the determination image are obtained, the processor executes a step of performing machine learning of the positive rate determination pattern using the determination image and the correct answer label information (S714). As an example, the machine learning is performed by assigning a set of the determination image for learning and the correct answer label information to the classifier including the feature amount extractor including a convolutional neural network and the neural network, and repeating learning while adjusting parameters of each neuron so that an output from the classifier becomes the same as the correct answer label information. Then, a step of acquiring a learned positive rate determination model is executed (S715). The acquired learned positive rate determination model may be stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.
Returning to
Next, processing of acquiring the second positive rate will be described. The processor 111 reads, from the memory 112, the interview information including the biometric detection information associated with the user ID information of the user to be determined, and the attribute information as necessary (S617). The processor 111 also reads, from the memory 112, the feature amount of the determination image calculated in S614 and stored in the memory 112 in association with the user ID information (S614). Then, the processor 111 assigns the interview information including the read biometric detection information and the feature amount of the determination image as inputs to the learned positive rate determination model (S618), and acquires the second positive rate indicating the second possibility of morbidity for influenza as an output (S619).
Here,
According to
When the learning feature amount of the determination image, the interview information and the attribute information including the biometric detection information, and the correct answer label information associated therewith are obtained, the processor executes a step of performing machine learning of the positive rate determination pattern using them (S724). As an example, the machine learning is performed by assigning a set of these pieces of information to a neural network in which neurons are combined, and repeating learning while adjusting parameters of each neuron so that an output from the neural network becomes the same as correct answer label information. Then, a step of acquiring the learned positive rate determination model is executed (S725). The acquired learned positive rate determination model may be stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.
Returning to
Next, processing of acquiring the third positive rate will be described. The processor 111 reads, from the memory 112, the interview information including the biometric detection information associated with the user ID information of the user to be determined, and the attribute information as necessary (S617). Further, the processor 111 reads the first positive rate calculated in S616 and stored in the memory 112 in association with the user ID information from the memory 112. Then, the processor 111 assigns, as an input, the interview information including the read biometric detection information and the first positive rate to the learned positive rate determination model (S620), and acquires, as an output, a third positive rate indicating the third possibility of morbidity for influenza (S621).
Here,
According to
When the first positive rate information, the interview information and the attribute information including the biometric detection information, and the correct answer label information associated therewith are obtained, respectively, the processor executes a step of performing machine learning of the positive rate determination pattern using them (S734). As an example, the machine learning is performed by assigning a set of these pieces of information to a neural network in which neurons are combined, and repeating learning while adjusting parameters of each neuron so that an output from the neural network becomes the same as correct answer label information. Then, a step of acquiring the learned positive rate determination model is executed (S735). The acquired learned positive rate determination model may be stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.
Returning to
When the first positive rate, the second positive rate, and the third positive rate are calculated in this manner, the processor 111 reads each positive rate from the memory 112 and performs ensemble processing (S622). As an example of the ensemble processing, the obtained first positive rate, second positive rate, and third positive rate are assigned as inputs to a ridge regression model, and a result of ensemble of the respective positive rates is acquired as a determination result of the possibility of morbidity for influenza (S623).
The ridge regression model used in S622 is generated by machine learning by the processor 111 of the processing device 100 or a processor of another processing device. Specifically, the processor acquires each of the first positive rate, the second positive rate, and the third positive rate from the determination image for learning. In addition, the processor assigns the correct answer label assigned in advance on the basis of a result of an influenza rapid test, a PCR test, a virus isolation culture test, or the like by immunochromatography to the user who is the subject of the determination image for learning. Then, the processor assigns the ridge regression model to a set of each positive rate and the correct answer label associated therewith, and repeats learning while adjusting a parameter assigned to each positive rate so that an output becomes the same as the correct answer label information of the ridge regression model. As a result, the ridge regression model used for the ensemble processing is obtained and stored in the memory 112 of the processing device 100 or another processing device connected to the processing device 100 via a wired or wireless network.
In addition, although the case of using the ridge regression model has been described as an example of the ensemble processing, any method such as processing of acquiring an average value of each positive rate, processing of acquiring a maximum value, processing of acquiring a minimum value, processing of performing weighted addition, and processing using other machine learning methods such as bagging, boosting, stacking, lasso regression, and linear regression may be used.
The processor 111 stores the determination result thus obtained in the user table of the memory 112 in association with the user ID information (S624). As a result, the processing flow ends.
In
Furthermore, as illustrated in
As described above, in the present embodiment, it is possible to provide the processing device, the processing program, the processing method, and the processing system suitable for processing the image obtained by imaging the inside of the oral cavity for use in diagnosis of the inside of the oral cavity. In particular, in the present embodiment, the biometric detection information of the biological body that is the user is detected using the information detection sensor 232 arranged in the imaging device 200 along with the imaging of the oral cavity, and the possibility of the disease is determined on the basis of the information. Therefore, it is possible to use the biometric detection information acquired at a position closer to the region imaged by the imaging device 200, that is, the attention region and at the same timing for determination, and to perform more accurate determination.
In the example of
In the example of
In the example of
Hereinafter, a case of obtaining the feature amount of the follicle of the pharynx will be described as a specific example of the interview information. As described above, the follicle appearing in the pharyngeal portion is a characteristic sign of influenza, and is also confirmed by visual inspection in diagnosis by a doctor. Therefore, processing of assigning a label to an attention region such as follicles is performed on the learning subject image by an operation input by doctors. Then, the position information (shape information) of the label in the learning subject image is acquired as the learning position information, and a set of the learning subject image and the learning position information labeled thereto is machine-learned by the neural network to obtain a learned region extraction model. Then, the processor 111 outputs the position information (shape information) of the attention region (that is, the follicle) by assigning the subject image to the learned region extraction model as an input. Thereafter, the processor 111 stores the obtained follicular position information (shape information) as the interview information.
In the example of
In addition, even in a case where the preprocessing is performed, the processor 111 may provide both the determination image that has been subjected to the preprocessing and the determination image that has not been subjected to the preprocessing to the feature amount extractor as inputs.
In addition, also in the generation of each learned model illustrated in
Each learned model described in
In the example of
Instead of the example of
An example of the distribution example of the above processing will be specifically described. In the terminal device 810 such as a smartphone held by a patient or the like, a tablet used in a medical institution or the like, and a laptop PC used by a doctor or the like, the processing according to S11 to S15 illustrated in
Note that the above is merely an example of dispersion of processing. In the present disclosure, the processing device 100 is referred to as a processing device. However, since various types of processing related to the determination processing and the like is simply executed, the processing device 100 is merely referred to as a processing device. For example, in a case where the imaging device 200, the terminal device 810, the electronic medical record device 820, the server device 830, and the like execute various types of processing, these also function as processing devices and may be referred to as processing devices.
In the example of
Note that these modifications are similar to the configurations, processes, and procedures in one embodiment described with reference to
The processes and procedures described in the present specification can be realized not only by those explicitly described in the embodiments but also by software, hardware, or a combination thereof. Specifically, the processing and procedures described in the present specification are realized by mounting logic corresponding to the processing on a medium such as an integrated circuit, a volatile memory, a nonvolatile memory, a magnetic disk, or an optical storage. In addition, the processing and procedures described in the present specification can be implemented as a computer program and executed by various computers including a processing device and a server device.
Even if it is described that the processing and procedures described in this specification are executed by a single device, software, component, or module, such processing or procedures can be executed by a plurality of devices, a plurality of pieces of software, a plurality of components, and/or a plurality of modules. Furthermore, even if it is described that various types of information described in the present specification are stored in a single memory or storage unit, such information can be stored in a distributed manner in a plurality of memories provided in a single device or a plurality of memories arranged in a distributed manner in a plurality of devices. Furthermore, the software and hardware elements described in the present specification can be realized by integrating them into fewer components or decomposing them into more components.
The present application is a continuation application of International Application No. PCT/JP2021/039715, filed on Oct. 27, 2021, which is expressly incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2021/039715 | Oct 2021 | WO |
| Child | 18643098 | US |
