Patent Application
20230177690
The present invention relates to a system for detecting bacteria in an image and classifying the bacteria.
In conventional infectious disease therapies, there has been a case where a wide variety of antibacterial agents are administered without identifying the bacterial species. If an antibacterial agent is unnecessarily administered, drug resistance bacteria resistant to the antibacterial agent are generated, which has become a social problem in recent years.
Examples of methods for solving such a problem include bacteria classification using Gram staining. By classifying bacteria by bacterial species based on Gram staining, it is possible to administer an antibacterial agent suitable for each bacterial species.
For example, Japanese Patent Application Laid-Open No. 2007-121282 discusses a technique for detecting bacteria in a specimen by using protein that links to the bacteria cell wall, and identifying whether the detected bacteria are Gram-positive or Gram-negative.
Japanese Patent Application Laid-Open No. 2007-232560 discusses a configuration of a Gram staining apparatus provided with a staining fluid and a cleaning fluid for quickly performing staining and cleaning operations in Gram staining.
Classifying bacteria by using Gram staining requires a series of operations including performing a staining operation, observing a specimen with a microscope to detect bacteria after the staining operation, and classifying the detected bacteria by shape and color.
Further, detecting the bacterial species from a Gram-stained specimen requires special knowledge and rich experience in Gram staining. This makes it necessary to rely on persons having such knowledge and experience, resulting in the concentration of burden on the specific persons.
According to an apsect of the present invention, an image processing apparatus includes an acquisition unit configured to acquire image data as result of imaging a Gram-stained specimen, and a generation unit configured to generate a display image by superimposing a position where a bacterium classified by Gram staining exists and a type of the bacterium on an image that is based on the image data.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
A series of processing in which a doctor or a nurse collects a specimen from a patient, and classifies bacterial species through Gram staining will be described below.
In Gram staining, bacteria which are stained can be classified into four different types “GNR”, “GNC”, “GPR”, and “GPC” by color and shape. Further, “GPC” can be classified into two different types “GPC Chain” and “GPC Cluster” by shape. In this system, detected bacteria are classified into five different types: “GNR”, “GNC”, “GPR”, “GPC Chain”, and “GPC Cluster”. In bacteria detection and classification, general object detection by Deep Learning is used.
As preparation for performing Gram staining in the classification system illustrated in
In the Gram staining apparatus 101, an optical system 201 includes a lens and a diaphragm and forms an image on an image sensor 202, such as a Charge Coupled Device (CCD) or Complementary Metal Oxide Semiconductor (CMOS) sensor, with a suitable amount of light from a subject. The image sensor 202 converts light that has passed through the optical system 201 and focused thereon into an image.
A central processing unit (CPU) 203 controls the operation of each component of the Gram staining apparatus 101. A secondary storage device 204 such as a hard disk drive stores programs used by the CPU 203 to control the operation of each component of the Gram staining apparatus 101. A primary storage device 205 such as a random access memory (RAM) stores a program loaded from the secondary storage device 204. The CPU 203 reads the program stored in the primary storage device 205.
The glass slide on which a specimen is smeared is fixedly placed on a sample fixing apparatus 206. The specimen is imaged by the optical system 201 and the image sensor 202. A hot air spraying apparatus 207 generates hot air to dry the specimen. A reservoir 208 is filled with the methanol fluid used to fix the specimen to the glass slide placed on the sample fixing apparatus 206. A reservoir 209 is filled with the Gram staining fluid used for Gram staining. A reservoir 210 is filled with the cleaning fluid used to clean the specimen during Gram staining. A communication apparatus 211 performs wireless or wired data communication with the working computer 102 for Gram staining.
The working computer 102 is configured by a personal computer or an edge computer and controls the operations of the Gram staining apparatus 101. The working computer 102 temporarily stores results of bacteria detection and classification performed by the Gram staining apparatus 101. A CPU 221 receives an instruction input by a user using a mouse, a keyboard, or a touch panel via an instruction input apparatus 225, and controls the operation of each component of the working computer 102. A secondary storage device 223 such as a hard disk drive stores a program used by the CPU 221 to control the operation of each component of the working computer 102. A primary storage device 222 such as a RAM stores a program loaded from the secondary storage device 223. The CPU 221 reads the program stored in the primary storage device 222.
The CPU 221 generates image and text data necessary for the user to operate an application for Gram staining, and transmits the generated image and text data to the display 104 via a display output terminal 224. Although the display 104 and the working computer 102 are described above as different apparatuses here, the working computer 102 may be provided with the display 104 like a tablet computer.
A communication apparatus 226 is wirelessly or wiredly connected to the Gram staining apparatus 101 and the server 103 and performs data communication therewith. The CPU 221 transmits an instruction related to the operations of the Gram staining apparatus 101 to the CPU 203 of the Gram staining apparatus 101 via the communication apparatus 226 and the communication apparatus 211.
The server 103 stores electronic medical charts. In a hospital, an electronic medical chart viewer application is installed in in-house computers used by doctors and nurses. The viewer application accesses the server 103 to acquire and display patient information. A CPU 231 controls the operation of each component of the server 103. A secondary storage device 233 such as a hard disk drive stores programs used by the CPU 231 to control the operation of each component of the server 103 and also stores electronic medical chart data as patient information. A primary storage device 232 such as a RAM stores a program and the electronic medical chart data loaded from the secondary storage device 233. The CPU 231 reads the program and data stored in the primary storage device 232. The CPU 231 receives a request from the CPU 221 of the working computer 102 via a communication apparatus 234 and transmits electronic medical chart data consistent with the request via the communication apparatus 234.
The working computer 102 is an image processing apparatus that activates an application for operating the Gram staining apparatus 101 in response to an instruction from the user.
The screen 300 displayed when the application is activated includes a “Full Automatic Mode” button 301, an “Individual Mode” button 302, a “Check Past Results” button 303, and a “Setting” button 304. The full automatic mode will be first described below.
When the user operates the instruction input apparatus 225 to select the button 301, the screen 310 illustrated in
In step S400, when the CPU 221 of the working computer 102 detects that the user selects the “Start” button 311, the CPU 221 transmits an operation start instruction to the CPU 203 of the Gram staining apparatus 101.
In step S410, the CPU 203 of the Gram staining apparatus 101 receives the operation start instruction from the CPU 221 of the working computer 102.
In step S411, the CPU 203 detects the number of glass slides set on the Gram staining apparatus 101. The CPU 203 may detect the number of glass slides by using an optical or mechanical sensor provided on the sample fixing apparatus 206 or by analyzing an image of the surface of the sample fixing apparatus 206 on which glass slides are placed. Alternatively, the user may input the number of glass slides.
In step S412, the hot air spraying apparatus 207 sprays hot air to the glass slides set on the sample fixing apparatus 206 to dry the specimens.
In step S413, the CPU 203 drops the methanol fluid contained in the reservoir 208 on the glass slides set on the sample fixing apparatus 206 by using an apparatus (not illustrated) and then fixes the specimens to the glass slides.
In step S414, the CPU 203 subjects the specimens on the glass slides to gram staining. In Gram staining, two different staining methods, Faber method and Bermy method, are often used. These two methods use different staining fluids but share common operation procedures. The methods use three to four different staining fluids. The specimens are stained with one staining fluid for a predetermined time and then cleaned. Then, the specimens are stained with another staining fluid and then cleaned again. The above-described procedure is repeated for the remaining staining fluids. In Gram staining, the Gram staining fluid contained in the reservoir 209 and the cleaning fluid contained in the reservoir 210 are used.
In steps S415 to S421, the specimens are sequentially observed. In step S415, the CPU 203 moves the sample fixing apparatus 206 to select the glass slide on which the first specimen is smeared. In the second and subsequent round of operations, the CPU 203 sequentially selects other glass slides in predetermined order.
In step S416, the CPU 203 moves the sample fixing apparatus 206 to change the observation position with respect to the currently selected glass slide.
In step S417, the CPU 203 determines whether the selected observation region is suitable for bacteria classification. This region determination processing will be described in detail below.
In step S418, when the selected observation region is suitable for bacteria classification (YES in step S418), the processing proceeds to step S419. On the other hand, when the selected observation region is not suitable for bacteria classification (NO in step S418), the processing returns to step S416. In step S416, the CPU 203 selects the next observation region.
In step S419, the CPU 203 performs bacteria detection and classification. This bacteria detection and classification processing will be described in detail below.
In step S420, the CPU 203 determines whether all of the plurality of observation regions on the glass slide have been selected. When all of the observation regions have been selected (YES in step S420), the processing proceeds to step S421. On the other hand, when not all of the observation regions have been selected (NO in step S420), the processing returns to step S416.
In step S421, the CPU 203 determines whether all of the glass slides set on the sample fixing apparatus 206 have been selected. When all of the glass slides have been selected (YES in step S421), the processing proceeds to step S422. On the other hand, when not all of the glass slides have been selected (NO in step S421), the processing returns to step S415.
In step S422, the CPU 203 transmits bacteria detection and classification results to the working computer 102. Data to be transmitted will be described in detail below.
In step S401, the CPU 221 of the working computer 102 receives the bacteria detection and classification results via the communication apparatus 226.
In step S402, the CPU 221 stores the received results in the secondary storage device 223.
In step S403, the CPU 221 generates display data indicating the bacteria detection and classification results and displays the display data on the display 104 to allow the user to view the results.
The results of bacteria detection and classification performed in the above-described manner are illustrated in
A bacterial species count 612 for each detected bacterial species is displayed below the image 609. The bacterial species count 612 indicates that the number of GNRs is 16, the number of GPC Clusters is 12, and the numbers of remaining bacteria are 0 in the image 609. There are check boxes next to the names of bacterial species, which enable the detection results to be filtered and displayed. In the screen 600, the detection result of GNR is not displayed since the check box of GNR is OFF. When the user presses a button 613 or 614, all of the check boxes are collectively turned ON or OFF, respectively.
The user can enlarge or reduce the displayed image 609 by operating buttons 611. A specimen number 601 is set for each specimen. The user can select a specimen by operating the up and down buttons displayed to the right of the specimen number 601. When three different glass slides are set on the Gram staining apparatus 101, any one of the three different specimens can be selected. When the specimen is switched to another, an image 602 indicating a position on the glass slide and the image 609 are updated. The image 602 indicates a position in the entire area of the glass slide which corresponds to the image 609. A button 603 is used to select another region on the same specimen that is determined to be suitable for bacteria classification. In
In the flowchart of
In step S700, the CPU 203 drives the optical system 201 and the image sensor 202 to capture an image of the observation region.
In step S701, the CPU 203 detects a specimen region where bacteria exist in the observation region. In the image of an observation region 500 in
In step S702, the CPU 203 calculates the average density of the specimen region 501. The CPU 203 obtains the average density to determine whether the specimen is thinly smeared.
In step S703, the CPU 203 determines whether the calculated average density is equal to or less than a threshold value. When the average density is equal to or less than the predetermined threshold (YES in step S703), then in step S704, the CPU 203 determines that the specimen region 501 is suitable for bacteria classification. On the other hand, when the average density is larger than the predetermined threshold (NO in step S703), then in step S705, the CPU 203 determines that the specimen region 501 is not suitable for bacteria classification.
When no specimen region is detected in the observation region in step S701, the CPU 203 determines that the observation region is not suitable for bacteria classification. The above-described method for determining a region suitable for bacteria classification is to be considered as illustrative, and other methods are also applicable. For example, the CPU 203 may determine whether the specimen region 501 is suitable for bacteria classification by using a trained model generated by machine learning in advance.
The flowchart in
In step 800, the CPU 203 increases the imaging magnification of the observation region. Generally, an imaging magnification of about 1,000 times is required to image bacteria, so that the CPU 203 drives the optical system 201 to increase the imaging magnification to 1,000 times.
In step 801, the CPU 203 images the specimen region 501 at the set imaging magnification by using the optical system 201 and the image sensor 202. For example, in a case of the observation region 500 in
In step 802, the CPU 203 performs bacteria detection and classification. In this step, a technique for detecting target objects from a captured image and classifying the target objects by using a trained model obtained by performing machine learning based on Deep Learning. In general object detection based on Deep Learning, a learning image group labeled with a target object position is prepared in advance and machine learning is performed using the learning image group to create a trained model. Then, by causing the created trained model to read an image to be determined, target objects can be detected from the image and classified. In this system, a trained model that has been trained using a number of labeled images of Gram-stained bacteria is created, and the trained model is stored in the Gram staining apparatus 101.
Referring back to
The screen 1000 allows the user to make various settings of the application. This screen allows the user to set an application to be activated upon selection of an “Open Medical Chart” button 604 in
A radio button 1005 having the options of ON and OFF is used to select whether to automatically transmit an image indicating the bacteria detection and classification results to the server 103 storing the electronic medical chart data, upon completion of bacteria classification in the full automatic mode. When the radio button 1005 is ON, data is automatically transmitted to the server 103 upon completion of the processing in the full automatic mode. When the radio button 1005 is OFF, data is not automatically transmitted.
When the option “ON” of the radio button 1005 is selected, a server name 1006 and image transmission options 1007 are enabled, and the image is transmitted in accordance with the server and options set here. A field 1006 is used to input a server name. The image transmission options 1007 allow the user to select whether to superimpose a “Detection Frame”, “Reliability”, and “Name of Bacterial Species” on the image when the image indicating the bacteria detection and classification results is transmitted. In order to change the data storage location in the working computer 102, the user uses a button 1008. The folder path of the data storage location is displayed in a field 1009.
In
In step 1100, the CPU 221 of the working computer 102 generates an image in accordance with the settings of the transmission options 1007 illustrated in
In step 1101, the working computer 102 transmits the generated image to the server 103 via the communication apparatus 226.
Next, in step 1110, the server 103 receives the image transmitted from the working computer 102, via the communication apparatus 234.
In step 1111, the CPU 231 of the server 103 stores the received image in the secondary storage device 233. When the received image is not provided with data for associating with the patient information in the electronic medical chart, the CPU 231 stores the image in the primary storage device 232 to allow a doctor and a nurse to associate the image stored in the primary storage device 232 with the patient information in the electronic medical chart afterwards.
In step 1112, the CPU 231 transmits a storage completion notification to the working computer 102.
In step 1102, the CPU 221 of the working computer 102 receives the storage completion notification. In this way, images can be automatically transmitted to the server 103 storing the electronic medical chart data.
Referring back to
When the user selects the button 605, the screen transitions to a screen 1200 illustrated in
Transmission options 1203 have functions similar to the functions of the transmission options 1007 in
The button 606 in
The first method is a method of moving the observation region to be displayed as an image 1304 by operating a mouse on the image 1304, like a typical image viewer. The user can increase or decrease the display magnification of the image 1304 by scrolling the mouse wheel. The image 1304 can also be enlarged or reduced by operating a button 1305. The second method is a method of moving the position of the observation region to be displayed on the glass slide by operating a button 1303. The user can also specify a desired position on an image 1306 of the entire glass slide to determine the observation region to be displayed as the image 1304.
Each time the user changes the observation region on the glass slide to be displayed as the image 1304, the Gram staining apparatus 101 performs bacteria detection and classification. This processing will be described below with reference to
In step 1400, the CPU 221 of the working computer 102 transmits information about the movement of the observation region specified by the user and information about the magnification to the Gram staining apparatus 101.
In step 1410, the CPU 203 of the Gram staining apparatus 101 receives the information about the movement of the observation region and the information about the magnification.
In step 1411, in accordance with the information about the movement of the observation region, the CPU 203 drives the sample fixing apparatus 206 to move the imaging position on the glass slide.
In step 1412, the CPU 203 changes the imaging magnification of the optical system 201 in accordance with the information about the magnification.
In step 1413, the CPU 203 captures a still image via the image sensor 202.
In step 1414, the CPU 203 performs bacteria detection and classification by using a trained model based on a method similar to that in step 802 in
In step 1415, the CPU 203 transmits the bacteria detection and classification results to the working computer 102 via the communication apparatus 211.
In step 1401, the CPU 221 of the working computer 102 receives the bacteria detection and classification results transmitted from the Gram staining apparatus 101, via the communication apparatus 226.
In step 1402, the CPU 221 generates display data indicating the bacteria detection and classification results and displays the display data on the display 104.
In the above-described manner, bacteria detection and a classification is also performed on a desired region specified by the user.
A method for checking past bacteria detection and classification results will be described below. The user can check the past bacteria detection and classification results by selecting the button 303 in the screen 300 in
When the user selects the button 303 in
A procedure for performing a search by a bacterial species will be described below. To perform a search by a bacterial species, the user turns a check box 1503 ON and then specifies bacterial species to be searched in the image by turning check boxes 1504 ON. In
Results of the search performed in the above-described manner are displayed in a list 1506. In the list 1506, search results are displayed for each specimen in a row. For example, for specimen No. 5 in a row 1507, an inspection for detecting and classifying is performed at 16:23 on Jun. 28, 2020, and the specimen has seven different target portions as regions suitable for bacteria detection and classification. The row 1507 also displays the number of bacteria for each bacterial species reflected in all of the regions suitable for bacteria detection and classification. It can be seen that, in specimen No. 5, 417 GNR bacteria are shown, and no other bacteria are shown.
In a case where there are many specimens, the user can switch the page of the list by using buttons 1508. When the user desires to display the details of a result, the user specifies a desired specimen from the list 1506 and then selects a button 1509, which causes the screen to transition to the screen 600 in
Although, in the above-described full automatic mode, all of operations related to Gram staining are automatically performed, there may be cases in which the user wants to perform only a specific operation. In such a case, the user can specify a desired operation by selecting an individual mode.
When the user selects the button 302 in the screen 300 in
The screen 1610 allows the user to check the progression rate of processing and the remaining processing time, like the screen 320 in
While the present invention has specifically been described based on exemplary embodiments, the present invention is not limited to these specific exemplary embodiments, and various embodiments not departing from the spirit and scope of the present invention are also included in the present invention. Parts of the above-described exemplary embodiments may be suitably combined.
For example, an apparatus integrating the configurations of both the Gram staining apparatus 101 and the working computer 102 is also applicable. According to the above-described exemplary embodiments, the CPU 203 of the Gram staining apparatus 101 performs bacteria classification by using a learning model. However, the classification processing may be performed by the working computer 102 or the server 103.
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
The present invention is not limited to the above-described exemplary embodiments but can be modified and changed in various ways without departing from the spirit and scope thereof. Therefore, the following claims are appended to disclose the scope of the present invention.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-123747 | Jul 2020 | JP | national |
| 2021-104154 | Jun 2021 | JP | national |
This application is a Continuation of International Patent Application No. PCT/JP2021/025782, filed Jul. 8, 2021, which claims the benefit of Japanese Patent Applications No. 2020-123747, filed Jul. 20, 2020, and No. 2021-104154, filed Jun. 23, 2021, all of which are hereby incorporated by reference herein in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2021/025782 | Jul 2021 | WO |
| Child | 18155277 | US |
