Patent Application
20230248209
The present disclosure relates to an assistant device, endoscopic system, assistant method, and computer-readable recording medium, which performs image processing on an imaging signal obtained by capturing a subject and outputs the result accordingly.
The technique of transurethral resection of bladder tumors (TUR-Bt) is known in the related art, where a surgical endoscope (resectoscope) is inserted through the urethra of a subject. Then, a surgical operator uses an excision treatment instrument such as an energy device to excise a living tissue including a nidus site, while observing the nidus site through the eyepiece of the surgical endoscope (see, e.g., JP2008-246111A).
In some embodiments, an assistant device includes a processor configured to: generate a first image including one or more characteristic regions requiring excision by a surgical operator; generate a second image including one or more cauterized regions cauterized by an energy device; generate information regarding presence or absence of a first region that is included in the characteristic regions and that is not included in the cauterized region based on the first image and the second image; and output information indicating the presence of the characteristic regions that requires to be cauterized when the first region is present.
In some embodiments, an endoscopic system includes: an endoscope configured to be insert into a lumen of a subject; a light source configured to apply excitation light that excites advanced glycation end products produced by subjecting a living tissue to thermal treatment; and a controller that is be detachable from the endoscope, the endoscope including an image sensor and an optical filter, the image sensor being configured to generate an imaging signal by capturing fluorescence emitted by the excitation light, the optical filter being provided on a light-receiving surface side of the image sensor, the optical filter being configured to block light on a short wavelength side including a part of a wavelength band of the excitation light, the controller including a processor configured to: generate a first image including one or more characteristic regions requiring excision by a surgical operator; generate a second image including one or more cauterized regions cauterized by an energy device; generate information regarding presence or absence of a first region that is included in the characteristic regions and that is not included in the cauterized region based on the first image and the second image; and output information indicating the presence of the characteristic regions that requires to be cauterized when the first region is present.
In some embodiments, provided is an assistant method executed by an assistant device. The method includes: generating a first image including one or more characteristic regions requiring excision by a surgical operator; generating a second image including one or more cauterized regions cauterized by an energy device; generating information regarding presence or absence of a first region that is included in the characteristic regions and that is not included in the cauterized region based on the first image and the second image; and outputting information indicating the presence of the characteristic regions that requires to be cauterized when the first region is present.
In some embodiments, a non-transitory computer-readable recording medium with an executable program stored thereon, the program causing an assistant device to: generate a first image including one or more characteristic regions requiring excision by a surgical operator; generate a second image including one or more cauterized regions cauterized by an energy device; generate information regarding presence or absence of a first region that is included in the characteristic regions and that is not included in the cauterized region based on the first image and the second image; and output information indicating the presence of the characteristic regions that requires to be cauterized when the first region is present.
The above and other features, advantages and technical and industrial significance of this disclosure will be better understood by reading the following detailed description of presently preferred embodiments of the disclosure, when considered in connection with the accompanying drawings.
Embodiments for carrying out the present disclosure are now described in detail with reference to drawings. Moreover, the present disclosure should not be construed as being limited to the following embodiments. In addition, each drawing referred to in the following description only schematically illustrates the shape, dimension, and relative positional relationship to the extent that the contents of the present disclosure can be understood. In other words, the present disclosure should not be construed as being limited only to the shapes, dimensions, and relative positional relationships illustrated in each drawing. Furthermore, in the description with reference to the drawings, the same parts are denoted by the same reference numerals. Moreover, an endoscopic system provided with a rigid endoscope and a medical imaging device will be described as an example of an endoscopic system according to the present disclosure.
The endoscopic system 1 illustrated in
The insertion portion 2 has a rigid or at least partially flexible elongated shape. The insertion portion 2 is inserted into a subject of a patient or the like through a trocar. The insertion portion 2 is provided therein with an optical system including a lens for forming an image for observation.
The light source 3 is connected to one end of the light guide 4 and supplies illumination light to be applied to a subject on one end of the light guide 4 under the control of the controller 9. The light source 3 is implemented employing a light source, a processor, and a memory. The light source of the light source 3 is one or more light sources such as a light-emitting diode (LED) light source, a xenon lamp, and a semiconductor laser device, including a laser diode (LD). The processor is a processing device that includes hardware such as a field programmable gate array (FPGA) or central processing unit (CPU), while the memory is a temporary storage area for the processor. Moreover, the light source 3 and the controller 9 can communicate separately, as illustrated in
The light guide 4 has one end detachably connected to the light source 3 and the other end detachably connected to the insertion portion 2. The light guide 4 guides illumination light supplied from the light source 3 from one end thereof to the other end to be supplied to the insertion portion 2.
The endoscopic camera head 5 is detachably connected to an eyepiece 21 of the insertion portion 2. The endoscopic camera head 5 receives an observation image formed by the insertion portion 2 and subjects the received light to photoelectric conversion to generate an imaging signal (raw data), and outputs the imaging signal to the controller 9 via the first transmission cable 6, under the control of the controller 9.
The first transmission cable 6 has one end detachably connected to the controller 9 via a video connector 61 and the other end detachably connected to the endoscopic camera head 5 via a camera head connector 62. The first transmission cable 6 transmits an imaging signal output from the endoscopic camera head 5 to the controller 9 and transmits setting data, electric power, or the like that is output from the controller 9 to the endoscopic camera head 5. The setting data is herein a control signal, synchronization, clock, or similar signals used to control the endoscopic camera head 5.
The display 7 displays an observation image and various types of information under the control of the controller 9. The observation image is based on an imaging signal subjected to image processing in the controller 9. The various types of information are related to the endoscopic system 1. The display 7 is implemented employing a monitor of liquid crystal display, organic electroluminescence (EL) display, or the like.
The second transmission cable 8 has one end detachably connected to the display 7 and the other end detachably connected to the controller 9. The second transmission cable 8 transmits the imaging signal subjected to image processing by the controller 9 to the display 7.
The controller 9 is implemented employing a processor and a memory. The processor in the controller 9 is a processing device having the hardware of a graphics processing unit (GPU), FPGA, CPU, or the like. The memory is a temporary storage area used by the processor. The controller 9 centrally controls operations of the endoscopic camera head 5, the display 7, and the light source 3 via the first transmission cable 6, the second transmission cable 8, and the third transmission cable 10, respectively, in accordance with a program recorded in the memory. In addition, the controller 9 performs various types of image processing on the imaging signal that is input via the first transmission cable 6 and outputs the processed imaging signal to the second transmission cable 8.
The third transmission cable 10 has one end detachably connected to the light source 3 and the other end detachably connected to the controller 9. The third transmission cable 10 transmits control data from the controller 9 to the light source 3.
Next, a functional configuration of main components of the above-described endoscopic system 1 will be described.
The configuration of the insertion portion 2 is now described. The insertion portion 2 has an optical system 22 and an illumination optical system 23.
The optical system 22 converges light including reflected light that is reflected from a photographic subject, return light from the photographic subject, excitation light from the photographic subject, light emitted by the subject, or the like to form a photographic subject image. The optical system 22 is implemented employing one or more lenses or similar components.
The illumination optical system 23 irradiates the photographic subject with illumination light supplied through the light guide 4. The illumination optical system 23 is implemented employing one or more lenses or similar components.
The configuration of the light source 3 is now described. The light source 3 includes a condenser lens 30, a first light source unit 31, a second light source unit 32, a third light source unit 33, and a light source control unit 34.
The condenser lens 30 converges each ray of light emitted from the first light source unit 31, the second light source unit 32, and the third light source unit 33 and emits the light through the light guide 4.
The first light source unit 31 emits visible white light (normal light) to cause the light guide 4 to be supplied with the white light as illumination light, which is performed under the control of the light source control unit 34. The first light source unit 31 includes a collimator lens, a white LED lamp, a driver, and similar components. Moreover, the first light source unit 31 can supply a visible wavelength spectrum of white light by simultaneous emission of light using a red LED lamp, a green LED lamp, and a blue LED lamp. The first light source unit 31 can be configured to use a halogen lamp, a xenon lamp, or similar components.
The second light source unit 32 emits a first narrow-band light beam with a predetermined wavelength band to cause the light guide 4 to be supplied with the first narrow-band light beam as illumination light, which is performed under the control of the light source control unit 34. The first narrow-band light beam herein has a wavelength band ranging from 530 to 550 nanometers (nm) (with a central wavelength of 540 nm). The second light source unit 32 uses a green LED lamp, a collimator lens, a transmission filter that allows light with a wavelength ranging from 530 nm to 550 nm to pass, a driving driver, and the like.
The third light source unit 33 emits a second narrow-band light beam with a wavelength band different from that of the first narrow-band light beam, causing the light guide 4 to be supplied with the second narrow-band light beam as illumination light, which is performed under the control of the light source control unit 34. The second narrow-band light beam herein has a wavelength band ranging from 400 nm to 430 nm (with a central wavelength of 415 nm). The third light source unit 33 is implemented employing a collimator lens, a semiconductor laser such as a violet laser diode (LD), a driver, and similar components. Moreover, in the first embodiment, the second narrow-band light beam functions as excitation light to excite advanced glycation end products that are produced by living tissue subjected to thermal treatment.
The light source control unit 34 is implemented employing a processor and a memory. The processor is a processing device that includes hardware such as an FPGA or CPU, while the memory is a temporary storage area for the processor. The light source control unit 34 controls the timing, duration, or the like of light emission for the first light source unit 31, the second light source unit 32, and the third light source unit 33 on the basis of the control data that is input from the controller 9.
Wavelength characteristics of light emitted from each of the second light source unit 32 and the third light source unit 33 are now described.
In
Thus, the second light source unit 32 and the third light source unit 33 emit the first narrow-band light beam and the second narrow-band light beam (excitation light), respectively, in different wavelength bands.
Referring back to
The configuration of the endoscopic camera head 5 is now described. The endoscopic camera head 5 includes an optical system 51, a driver 52, an image sensor 53, a cut filter 54, an analog-to-digital (A/D) converter 55, a parallel-to-serial (P/S) converter 56, an imaging recorder 57, and an imaging control unit 58.
The optical system 51 forms a photographic subject image condensed by the optical system 22 of the insertion portion 2 on a light-receiving surface of the image sensor 53. The optical system 51 is capable of modifying the focal length and focal position. The optical system 51 includes a plurality of lenses 511. The optical system 51 changes the focal length and position by causing the driver 52 to move each of the plurality of lenses 511 along an optical axis L1.
The driver 52 shifts the plurality of lenses 511 of the optical system 51 along the optical axis L1 under the control of the imaging control unit 58. The driver 52 includes a motor and a transmission mechanism. Examples of the motor include a stepping motor, a DC motor, and a voice coil motor. The transmission mechanism can be a gear or the like that transmits the rotation of the motor to the optical system 51.
The image sensor 53 is implemented employing a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) image sensor having a plurality of pixels arranged in a two-dimensional matrix. The image sensor 53 receives the photographic subject image (rays of light), which is formed by the optical system 51 and passes through the cut filter 54, subjects the received image to photoelectric conversion to generate an imaging signal (raw data), and outputs the generated image to the A/D converter 55, which is performed under the control of the imaging control unit 58. The image sensor 53 has a pixel array unit 531 and a color filter 532.
As indicated by curve LB in
The image sensor 53 configured as described above allows the generation of color signals (G, R, and B signals) for the respective G, R, and B pixels in a case of receiving the photographic subject image formed by the optical system 51, as illustrated in
Referring back to
The cut filter 54 is arranged on the optical axis L1 between the optical system 51 and the image sensor 53. The cut filter 54 is provided on the light-receiving surface side (incident surface side) of the G pixel provided with the G-filter that is included in the color filter 532 and that allows light in at least the green wavelength band to pass through. The cut filter 54 blocks light in the short wavelength band including the wavelength band of the excitation light and allows light in the longer wavelength band than the wavelength band of the excitation light including narrow-band light to pass through.
As illustrated in
Referring back to
The analog-to-digital (A/D) converter 55 performs A/D conversion processing on the analog imaging signal input from the image sensor 53 and outputs the result to the P/S converter 56 under the control of the imaging control unit 58. The A/D converter 55 is implemented employing an A/D conversion circuit or the like.
The parallel-to-serial (P/S) converter 56 performs parallel-to-serial conversion on a digital imaging signal input from the A/D converter 55 and outputs the imaging signal subjected to the parallel-to-serial conversion to the controller 9 via the first transmission cable 6 under the control of the imaging control unit 58. The P/S converter 56 is implemented employing a P/S conversion circuit or the like. Moreover, in the first embodiment, instead of the P/S converter 56, an E/O converter that converts an imaging signal into an optical signal can be provided, outputting the imaging signal to the controller 9 as an optical signal. Alternatively, the imaging signal can be sent to the controller 9 by wireless communication such as Wi-Fi (a registered trademark of the Wi-Fi Alliance).
The imaging recorder 57 records various types of information regarding the endoscopic camera head 5 (e.g., pixel information of the image sensor 53 or characteristics of the cut filter 54). In addition, the imaging recorder 57 records various setting data and control parameters transmitted from the controller 9 via the first transmission cable 6. The imaging recorder 57 includes at least one of a non-volatile memory or a volatile memory.
The imaging control unit 58 controls each operation of the driver 52, the image sensor 53, the A/D converter 55, and the P/S converter 56 on the basis of the setting data received from the controller 9 via the first transmission cable 6. The imaging control unit 58 is implemented employing a timing generator (TG), a processor, and a memory. The processor is a processing device that includes hardware such as a CPU, while the memory is a temporary storage area for the processor.
The configuration of the controller 9 is now described.
The controller 9 includes a serial-to-parallel (S/P) converter 91, an image processing unit 92, an input unit 93, a recorder 94, and a control unit 95.
The S/P converter 91 performs serial-to-parallel conversion on image data received from the endoscopic camera head 5 via the first transmission cable 6 and outputs the result to the image processing unit 92, which is performed under the control of the control unit 95. Moreover, in a case where the endoscopic camera head 5 outputs an imaging signal in the form of an optical signal, an optical-to-electrical (O/E) converter that converts optical signals into electrical signals can be provided in place of the S/P converter 91. In addition, in a case where the endoscopic camera head 5 sends an imaging signal by wireless communication, a communication module capable of receiving a wireless signal can be provided in place of the S/P converter 91.
The image processing unit 92 performs predetermined image processing on the imaging signal in the form of parallel data input from the S/P converter 91 and outputs the result to the display 7, which is performed under the control of the control unit 95. The predetermined image processing herein includes demosaicing, white balancing, gain adjustment, gamma (γ) correction, format conversion processing, and the like. The image processing unit 92 is implemented employing a processor and a memory. The processor is a processing device that includes hardware such as a GPU or FPGA, while the memory is a temporary storage area for the processor. Moreover, in the first embodiment, the image processing unit 92 functions as an assistant device. The image processing unit 92 has a generation unit 921, an identification unit 922, a determination unit 923, and an output unit 924.
The generation unit 921 generates a first image including one or more characteristic regions that need to be excised by a surgical operator and a second image including one or more cauterized regions cauterized by the energy device. Specifically, the generation unit 921 generates the first image on the basis of an imaging signal obtained by capturing reflected light and return light from the living tissue in the case where the living tissue is irradiated with narrow-band light with a narrower wavelength band than white light. More specifically, the generation unit 921 generates the first image, which is a pseudo-color image including one or more characteristic regions (lesion sites) that need to be excised by the surgical operator on the basis of an imaging signal, which is obtained by capturing reflected light and return light from the living tissue in the case where the living tissue is irradiated with the first narrow-band light and the second narrow-band light in the narrow-band imaging observation mode of the endoscopic system 1, which will be described later. In addition, the generation unit 921 generates the second image on the basis of an imaging signal obtained by capturing fluorescence generated by excitation light applied to excite advanced glycation end products produced by subjecting the living tissue to thermal treatment in the thermal treatment imaging observation mode of the endoscopic system 1, which will be described later.
The identification unit 922 calculates a hue H of each pixel in the first image, which is the pseudo-color image generated by the generation unit 921, and identifies a pixel having a brown color (e.g., a hue H of 5 to 35) as a characteristic region (lesion site). The hue H is herein one of the attributes of color (hue, saturation, and brightness) and the aspect of color represented by a numerical value in the range of 0 to 360 using what is called the Munsell color wheel (e.g., red, blue, and yellow). Moreover, the identification unit 922 can determine whether or not each pixel of the first image, which is the pseudo-color image generated by the generation unit 921, has a predetermined luminance (gradation value) or more, identifying the characteristic region (lesion site) by extracting a pixel with a brightness higher than a predetermined value. In addition, the identification unit 922 determines whether or not the luminance value (gradation value) of each pixel of the second image, which is the fluorescence image generated by the generation unit 921, is equal to or greater than a predetermined threshold value, identifying a pixel equal to or greater than a predetermined threshold as a cauterized region.
The determination unit 923 determines whether or not the characteristic region is included in the cauterized region on the basis of the first image and the second image generated by the generation unit 921. Specifically, the determination unit 923 determines whether or not the entire characteristic region is included in the cauterized region on the basis of the first image and the second image generated by the generation unit 921.
The output unit 924 outputs information indicating that there is a characteristic region (lesion site) that still needs to be cauterized in the case where the determination unit 923 determines that the characteristic region (lesion site) is not included in the cauterized region.
The input unit 93 receives inputs for various operations related to the endoscopic system 1 and outputs the received operations to the control unit 95. Examples of the input unit 93 include a mouse, foot switch, keyboard, buttons, switches, touch panel, and the like.
The recorder 94 is implemented employing recording media such as volatile memory, non-volatile memory, solid-state drive (SSD), hard disk drive (HDD), or the like, and memory cards. The recorder 94 is used to record data including various parameters required for the operation of the endoscopic system 1. In addition, the recorder 94 also has a program recording unit 941 that records various programs used for endoscopic system 1 to operate.
The control unit 95 is implemented employing a processor and a memory. The processor is a processing device that includes hardware such as an FPGA or CPU, while the memory is a temporary storage area for the processor. The control unit 95 centrally controls each unit that configures the endoscopic system 1.
An overview of observation modes executable by the endoscopic system 1 is now described. Moreover, the description below is given in the order of a narrow-band imaging observation mode, a thermal treatment imaging observation mode, and a normal light imaging observation mode.
First, the narrow-band imaging observation mode will be described.
The narrow-band imaging (NBI) observation mode is an imaging observation technology that enhances the visibility of the capillaries and mucosal surface structure of the mucosal surface of living tissue by using the properties of hemoglobin in blood that strongly absorbs light at a wavelength of around 415 nm. In other words, in the narrow-band imaging observation mode, a subject of living tissue or the like is irradiated with two narrow-banded light beams more likely to be absorbed by hemoglobin in blood. These two narrow-banded light beams are a first narrow-band light beam (with a wavelength band ranging from 530 nm to 550 nm) and a second narrow-band light beam (with a wavelength band ranging from 390 nm to 445 nm). This configuration employing the narrow-band imaging observation mode makes it possible to enhance the visibility of capillaries on the mucosal surface and micro patterns of the mucous membrane, which are difficult to recognize visually with normal light (white light).
Specifically, as illustrated in the graph G1 of
More specifically, as illustrated by the polygonal line LF of the graph G2 in
Furthermore, the cut filter 54 allows the reflected light beam WG1 in the wavelength band longer than the wavelength band of the second narrow-band light beam W2, which includes the first narrow-band light beam W1 to pass through. In addition, the reflected light beams (reflected light beams WR1, WR2, WB1, and WB2) obtained by reflecting the first narrow-band light beam W1 and the second narrow-band light beam W2 from the subject are incident on the corresponding R and B pixels.
Subsequently, in
The image processing unit 92 then acquires an imaging signal (raw data) from the image sensor 53 of the endoscopic camera head 5 and performs image processing on each of the signal values of the G pixels and B pixels included in the acquired imaging signal to produce a pseudo-color image (narrow-band image). In this case, the signal value of the G pixel includes mucosal deep layer information of the subject. In addition, the signal value of the B pixel includes mucosal surface information of the subject. For this reason, the image processing unit 92 performs image processing such as gain control processing, pixel interpolation processing, and mucosa enhancement processing on each signal value of the G pixels and B pixels included in the imaging signal to generate and output a pseudo-color image to the display 7. The pseudo-color image is herein an image generated using only the signal value of the G pixel and the signal value of the B pixel. In addition, the image processing unit 92 acquires the signal value of the R pixel, but deletes the signal value of the R pixel without using the signal value of the R pixel for generating the pseudo-color image.
As described above, the narrow-band imaging observation mode makes it possible to enhance the visibility of capillaries on the mucosal surface and micro patterns of the mucous membrane, which are difficult to recognize visually with white light (normal light).
Next, the thermal treatment imaging observation mode will be described.
In recent years, minimally invasive treatments using an endoscope, laparoscope, or the like have become widely used in the medical field. Examples of minimally invasive treatments using an endoscope, laparoscope, or the like include endoscopic submucosal dissection (ESD), laparoscopy and endoscopy cooperative surgery (LECS), non-exposed endoscopic wall-inversion surgery (NEWS), transurethral resection of bladder tumors (TUR-bt), or the like, which are widely practiced.
In such minimally invasive treatments, a surgical operator, including a surgeon, performs an excision by cauterization, a marking treatment by thermal treatment, or the like on a characteristic region (pathogenic region) having a lesion in living tissue using an energy device treatment instrument that emits energy such as high frequency, ultrasonic wave, microwave, or the like in performing the treatment, which is intended to mark biologically a surgical target region as pretreatment. Furthermore, the surgical operator uses an energy device or the like to perform the treatment such as excision and coagulation of the living tissue of the subject, even in actual treatment.
The actual situation is that the degree of thermal treatment applied to the living tissue by an energy device is checked by the surgical operator relying on naked eye, tactile, and intuition. For this reason, the treatments using an energy device, or the like in the related art is difficult for the surgical operator to check in real time the degree of thermal treatment to be applied during procedures such as surgery, resulting in a medical task requiring a great deal of skill. This situation has led to the desire of the surgical operator or the like for a technique capable of visualizing the cauterized state in the thermally treated region by the thermal treatment in thermally treating the living tissue using the energy device.
Moreover, a glycation reaction (Maillard reaction) occurs in the case where amino acids and reducing sugars are heated. The end products resulting from this Maillard reaction are generally called advanced glycation end products (AGEs). AGEs are known to include a substance having fluorescence characteristics.
In other words, AGEs are produced when the living tissue is thermally treated with an energy device, the amino acids and reducing sugars in the living tissue are heated, and the Maillard reaction occurs. The AGEs produced by this heating enable visualization of the state of the thermal treatment by fluorescence imaging observation. Furthermore, AGEs are known to emit stronger fluorescence than autofluorescent substances originally present in living tissue.
In other words, the thermal treatment imaging observation mode is an imaging observation technique that visualizes the thermally treated region by utilizing the fluorescence characteristics of AGEs produced in living tissue subjected to thermal treatment with an energy device or the like. For this reason, in the thermal treatment imaging observation mode, the living tissue is irradiated with blue light having a wavelength of around 415 nm for exciting AGEs from the light source 3. This configuration allows a thermal treatment image (fluorescence image) obtained by image-capturing the fluorescence (e.g., green light with a wavelength ranging from 490 nm to 625 nm) produced from AGEs to be observed in the thermal treatment imaging observation mode.
Specifically, as illustrated in the graph G11 of
More specifically, as illustrated in the graph G12 of
Further, as illustrated by the polygonal line LNG of the fluorescence characteristics in the graph G12 of
The image processing unit 92 then acquires image data (raw data) from the image sensor 53 of the endoscopic camera head 5 and performs image processing on each signal value of the G pixel and B pixel included in the acquired image data to generate a fluorescence image (pseudo-color image). In this case, the signal value of the G pixel includes information regarding fluorescence emitted from the thermal treatment region. In addition, the B pixel includes information regarding the background, which is the living tissue surrounding the thermal treatment region. For this reason, the image processing unit 92 performs image processing such as gain control processing, pixel interpolation processing, and mucosa enhancement processing on the signal values of the G pixel and the B pixel included in the image data to generate a fluorescence image (pseudo fluorescence image). Then, the image processing unit 92 outputs the fluorescence image (pseudo-color image) to the display 7. In this case, the image processing unit 92 performs gain control processing to make the gain for the signal value of the G pixel larger than the gain for the signal value of the G pixel during the normal light imaging observation and to make the gain for the signal value of the B pixel smaller than the gain for the signal value of the B pixel during the normal light imaging observation. Furthermore, the image processing unit 92 performs gain control processing so that the signal value of the G pixel and the signal value of the B pixel are the same (1 : 1).
As described above, in the thermal treatment imaging observation mode, it is possible to easily observe the living tissue O2 (thermal treatment region) to be subjected to thermal treatment by an energy device or the like.
Next, the normal light imaging observation mode will be described.
As illustrated in
Subsequently, the image processing unit 92 acquires image data (raw data) from the image sensor 53 of the endoscopic camera head 5 and performs image processing on the signal values of each of the R, G, and B pixels included in the acquired image data to generate a white-light image. In this case, the image processing unit 92 performs white balance adjustment processing that adjusts the white balance so that the ratio of the red, green, and blue components is constant because the blue component included in the image data is smaller than that in the white-light imaging observation in the related art.
Thus, in the normal light imaging observation mode, it is possible to observe a natural white-light image (observation image) even in the case where the cut filter 54 is arranged on the light-receiving surface side of the G pixel.
The procedure for the transurethral resection of bladder tumors performed by a surgical operator is described. Moreover, the conventional transurethral resection of bladder tumors is first described, followed by a description of the novel transurethral resection of bladder tumors using the endoscopic system 1 of the present disclosure is given. The conventional transurethral resection of bladder tumors uses photodynamic diagnosis (PDD) in which treatment is conducted while observing both the tumor and treatment sites by administering a photosensitizer, such as conventional 5-aminolevulinic acid (hereinafter referred to as “5-ALA”), into the body of the subject, and then observing the fluorescence produced by irradiating protoporphyrin IX (hereinafter referred to as “PPIX”) accumulated in the tumor with excitation light.
The procedure for the conventional transurethral resection of bladder tumors using PDD is now described.
As illustrated in
Subsequently, if a bladder tumor is detected in the subject, the surgical operator identifies the bladder tumor and then administers 5-ALA to the subject (Step S2). Specifically, the surgical operator causes the subject to take a drug containing 5-ALA before surgery.
Then, the surgical operator inserts the endoscope through the urethra of the subject (Step S3) and checks the specific region (lesion site) including the tumor position in the bladder using the white light of the endoscope (Step S4). In this case, the surgical operator checks the rough specific region including the tumor position while checking the observation image displayed on the display.
Subsequently, the surgical operator cauterizes and excises the lesion site including a lesion of the subject with an energy device or the like through the endoscope while checking a fluorescence image P1 displayed on the display (Step S5).
Then, the surgical operator switches the endoscope to a PDD mode and causes the endoscope to apply the second narrow-band light to perform observation using PDD (Step S6). In this case, as illustrated in
Subsequently, the surgical operator observes the fluorescence image P1 displayed on the display to determine whether or not the excision of the tumor is completed (Step S7). If the excision for all the targeted tumors is completed (Step S7: Yes), the surgical operator ends the procedure. Specifically, while observing the fluorescence image P1 displayed on the display, the surgical operator determines whether or not the entire fluorescence region W1 of the fluorescence image P1 is excised. If the entire fluorescence region W1 is excised, then the excision of the specific region (lesion site) including the lesion such as a tumor is determined to be completed, and the procedure ends. On the other hand, if the excision of all the tumors is not completed (Step S7: No), the surgical operator returns to the previous Step S4 and continues the procedure until the fluorescence region W1 is cauterized using an energy device or the like while alternately switching the endoscope imaging observation mode between the white-light observation image and the PDD-assisted fluorescence image P1.
Thus, the procedure for conventional transurethral resection of bladder tumors using PDD requires the administration of 5-ALA to the subject.
The procedure for the transurethral resection of bladder tumors performed using the endoscopic system 1 according to the present disclosure is now described.
As illustrated in
Subsequently, the surgical operator inserts the insertion portion 2 (rigid endoscope) into the urethra of the subject (Step S11), causes the light source 3 to irradiate the inside of the subject with white light, and confirms a characteristic region (lesion site) including the tumor position while observing the observation image displayed by the display 7 (Step S12). Specifically, as illustrated in
Thereafter, the surgical operator cauterizes and excises the characteristic region (lesion site) including a lesion such as a tumor of the subject with an energy device or the like through the insertion portion 2 while checking a white-light image P2 displayed on the display 7 (Step S13). Specifically, as illustrated in
Thereafter, the surgical operator causes the light source 3 to irradiate the inside of the subject with the second narrow-band light (excitation light), which is the excitation light, and observes the fluorescence image displayed by the display 7 (Step S14).
Subsequently, the surgical operator observes the fluorescence image displayed by the display 7 to determine whether or not the excision of the characteristic region (lesion site) including the tumor position is completed (Step S15). If the excision for the characteristic region (lesion site) including the tumor position is completed (Step S15: Yes), the surgical operator ends the procedure. Specifically, as illustrated in
As described above, according to the transurethral resection of bladder tumors using the endoscopic system 1 of the present disclosure, the tumor of the subject can be excised without administering 5-ALA to the subject, and the characteristic region (lesion site) including the tumor position and the cauterized region can be easily grasped, so that it is possible to prevent the tumor from being left unremoved.
Next, processing executed by the endoscopic system 1 will be described.
As illustrated in
Subsequently, the generation unit 921 acquires an imaging signal from the image sensor 53 of the endoscopic camera head 5 to generate a white-light image (Step S102). In this case, the output unit 924 causes the display 7 to display the white-light image generated by the generation unit 921.
Thereafter, the control unit 95 controls the light source control unit 34 to cause the second light source unit 32 and the third light source unit 33 to irradiate the photographic subject with the first and second narrow-band light (Step S103).
Subsequently, the generation unit 921 acquires an imaging signal from the image sensor 53 of the endoscopic camera head 5 to generate the first image that is a pseudo-color image (Step S104).
Thereafter, the identification unit 922 identifies the characteristic region (lesion site) from the first image generated by the generation unit 921 (Step S105). Specifies, the identification unit 922 calculates a hue H of each pixel in the first image, which is the pseudo-color image generated by the generation unit 921, and identifies a pixel having a brown color (e.g., a hue H of 5 to 35) as a characteristic region (lesion site). For example, as illustrated in
Subsequently, the control unit 95 controls the light source control unit 34 to cause the third light source unit 33 to irradiate the subject with the second narrow-band light, which is the excitation light (Step S106).
Thereafter, the generation unit 921 acquires an imaging signal from the image sensor 53 of the endoscopic camera head 5 to generate the second image (Step S107).
Thereafter, the identification unit 922 identifies the cauterized region from the second image (Step S108). Specifically, the identification unit 922 determines whether or not each pixel of the second image has predetermined luminance or more, and extracts pixels having the predetermined luminance or more to identify the cauterized region. Specifically, as illustrated in
Subsequently, the determination unit 923 determines whether or not the characteristic region is included in the cauterized region (Step S109). Specifically, as illustrated in
In Step S110, the output unit 924 notifies the display 7 by outputting information indicating that there is a characteristic region (lesion site) that has not yet been cauterized. As a result, the surgical operator can grasp that there is a region that need to be cauterized by an energy device or the like with respect to the characteristic region (lesion site) of the subject, and thus, it is possible to easily grasp the characteristic region (lesion site) being left unremoved.
Subsequently, when the end signal for ending the observation of the subject is input via the input unit 93 (Step S111: Yes), the endoscopic system 1 ends the present processing. On the other hand, when the end signal for ending the observation of the subject is not input via the input unit 93 (Step S111: No), the endoscopic system 1 returns to Step S101 described above.
According to the first embodiment described above, the output unit 924 notifies the display 7 by outputting information indicating that there is a characteristic region (lesion site) that still needs to be cauterized to in the case where the determination unit 923 determines that the characteristic region is not included in the cauterized region. Therefore, the surgical operator can easily grasp the characteristic region (lesion site) being left unremoved.
Furthermore, according to the first embodiment, the generation unit 921 acquires the imaging signal from the image sensor 53 of the endoscopic camera head 5, that is, the imaging signal generated by capturing the reflected light and the return light from the living tissue at the time of irradiating the living tissue with the first and second narrow-band light having a narrower wavelength band than the white light, thereby generating the first image that is the pseudo-color image. Therefore, it is possible to generate the first image including one or more characteristic regions (lesion sites) that need to be excised by a surgical operator.
In addition, according to the first embodiment, the generation unit 921 generates the second image on the basis of an imaging signal obtained by capturing fluorescence generated by excitation light applied to excite advanced glycation end products produced by subjecting the living tissue to thermal treatment, therefore, the second image including one or more cauterized regions cauterized by the energy device.
Next, a second embodiment will be described. In the first embodiment described above, the characteristic region (lesion site) is extracted on the basis of the pseudo-color image corresponding to the imaging signal generated by irradiating the subject with the first narrow-band light and capturing the reflected light and the return light from the subject, but in the second embodiment, the characteristic region is identified from the white-light image obtained by capturing the living tissue using the learned model learned using the teacher data in which the plurality of biological images of the subject and the information to which the annotation of the characteristic region (lesion site) included in the plurality of biological images is applied are associated with each other. In the following description, the same components as those of the endoscopic system 1 according to the first embodiment described above are denoted by the same reference numerals, and a detailed description thereof will be omitted.
First, the configuration of the light source 3A is described. In the light source 3A, the second light source unit 32 capable of emitting the first narrow-band light from the light source 3A according to the above-described first embodiment is omitted.
The configuration of the controller 9A is now described. The controller 9A further includes a lesion learned model unit 96 in addition to the configuration of the controller 9 according to the first embodiment described above.
The lesion learned model unit 96 records a lesion learned model for identifying a characteristic region (lesion site) included in the white-light image. Specifically, the lesion learned model unit 96 records a learning result learned using teacher data in which a plurality biological images of a subject are associated with information to which annotation of a characteristic region (lesion site) included in the plurality of biological images is applied. The lesion learned model unit 96 inputs an imaging signal generated by capturing reflected light when white light is emitted to a living tissue or return light from the living tissue as input data, and outputs a position of a lesion site in a captured image corresponding to the imaging signal as output data. Here, the lesion learned model includes a neural network in which each layer has one or a plurality of nodes. Furthermore, the type of machine learning is not particularly limited, but for example, it is sufficient that teaching data and learning data in which a plurality of biological images of subjects are associated with information to which annotation of a characteristic region (lesion site) included in the plurality of biological images are applied are prepared, and the teaching data and the learning data are input to a calculation model based on a multilayer neural network to perform learning. Furthermore, as a machine learning method, for example, a method based on a deep neural network (DNN) of a multilayer neural network such as a convolutional neural network (CNN) or a 3D-CNN is used. Furthermore, as a machine learning method, a method based on a recurrent neural network (RNN), a long short-term memory unit (LSTM) obtained by extending the RNN, or the like may be used.
Next, processing executed by the endoscopic system 1A will be described.
In Step S203, the identification unit 922 identifies a characteristic region (lesion site) from the white-light image on the basis of the learned model recorded by the lesion learned model unit 96 and the white-light image generated by the generation unit 921. Specifically, the identification unit 922 inputs the white-light image generated by the generation unit 921 to the lesion learned model unit 96 as input data, and identifies a characteristic region (lesion site) from the white-light image on the basis of the position of the characteristic region output from the lesion learned model unit 96 as output data.
Steps S204 and S209 correspond to Steps S106 and S111 in
According to the second embodiment described above, similarly to the first embodiment described above, the surgical operator can easily grasp the characteristic region (lesion site) being left unremoved.
Next, a third embodiment will be described. In the third embodiment, the surgical operator sets the characteristic region by operating the input unit 93 while observing the white-light image displayed on the display 7 to apply the annotation to the characteristic region (tumor region) appearing in the white-light image. In the following description, the same components as those of the endoscopic system 1 according to the second embodiment described above are denoted by the same reference numerals, and a detailed description thereof will be omitted.
Next, processing executed by the endoscopic system 1B will be described.
In Step S303, in a case where the surgical operator performs the annotation insertion operation on the white-light image via the input unit 93 (Step S303: Yes), the endoscopic system 1B proceeds to Step S304 described later. On the contrary, in a case where the surgical operator does not perform the annotation insertion operation on the white-light image via the input unit 93 (Step S303: No), the endoscopic system 1B proceeds to Step S311 described later.
In Step S304, the identification unit 922 identifies a region in the white-light image designated according to the annotation insertion operation input from the input unit 93 as a specific region (lesion site). After Step S304, the endoscopic system 1 proceeds to Step S305 described later.
Steps S305 and S310 correspond to Steps S106 and S111 in
According to the second embodiment described above, similarly to the first embodiment described above, the surgical operator can easily grasp the characteristic region (lesion site) being left unremoved.
Next, a fourth embodiment will be described. In the above-described first to third embodiments, the endoscopic system includes a rigid endoscope, but in the fourth embodiment, an endoscopic system including a flexible endoscope will be described. Hereinafter, an endoscopic system according to the fourth embodiment will be described. In the fourth embodiment, the same components as those of the endoscopic system 1 according to the first embodiment described above are denoted by the same reference numerals, and a detailed description thereof will be omitted.
An endoscopic system 100 illustrated in
The configuration of the endoscope 102 is now described. The endoscope 102 generates image data by capturing the inside of the body of the subject, and outputs the generated image data to the controller 9. The endoscope 102 includes an operating unit 122 and a universal cord 123.
An insertion portion 121 has an elongated shape having flexibility. The insertion portion 121 includes a distal end portion 124 incorporating an imaging device to be described later, a bendable bending portion 125 including a plurality of bending pieces, and an elongated flexible tube portion 126 connected to a proximal end side of the bending portion 125 and having flexibility.
The distal end portion 124 is configured using glass fiber or the like. The distal end portion 124 includes a light guide 241 forming a light guide path of the light supplied from the light source 3, an illumination lens 242 provided at the distal end of the light guide 241, and an imaging device 243.
The imaging device 243 includes an optical system 244 for condensing light, the image sensor 53, the cut filter 54, the analog-to-digital (A/D) converter 55, the parallel-to-serial (P/S) converter 56, the imaging recorder 57, and the imaging control unit 58 according to the first embodiment described above. Note that, in the third embodiment, the imaging device 243 functions as a medical imaging device.
The universal cord 123 incorporates at least a light guide 241 and an assembled cable in which one or a plurality of cables are collected. The assembled cable is a signal line for transmitting and receiving signals between the endoscope 102 and the light source 3 and the controller 9, and includes a signal line for transmitting and receiving setting data, a signal line for transmitting and receiving a captured image (image data), a signal line for transmitting and receiving a driving timing signal for driving the image sensor 53, and the like. The universal cord 123 has a connector portion 127 detachable from the light source 3. The connector portion 127 has a coil-shaped coil cable 127a extending, and a connector portion 128 detachably attached to the controller 9 at an extending end of the coil cable 127a.
The endoscopic system 100 configured as described above performs processing similar to that of the endoscopic system 1 according to the first embodiment described above.
According to the fourth embodiment described above, similarly to the first embodiment described above, the surgical operator can easily grasp the characteristic region (lesion site) being left unremoved.
Next, a fifth embodiment will be described. In the above-described first to fourth embodiments, the endoscopic system is described, but in the fifth embodiment, a case where the endoscopic system is applied to a surgical microscope system will be described. In the fifth embodiment, the same components as those of the endoscopic system 1 according to the first embodiment described above are denoted by the same reference numerals, and a detailed description thereof will be omitted.
The microscope device 310 includes a microscope unit 312 that enlarges and captures a minute portion of a photographic subject, a support unit 313 that is connected to a proximal end portion of the microscope unit 312 and includes an arm that rotatably supports the microscope unit 312, and a base unit 314 that rotatably holds the proximal end portion of the support unit 313 and is movable on a floor surface. The base unit 314 includes a light source 3 that generates white light beam, first narrow-band light, second narrow-band light, and the like to be emitted from the microscope device 310 to the photographic subject, and a controller 9 that controls the operation of the surgical microscope system 300. Note that each of the light source 3 and the controller 9 has at least a configuration similar to that of the first embodiment described above. Specifically, the light source 3 includes a condenser lens 30, a first light source unit 31, a second light source unit 32, a third light source unit 33, and a light source control unit 34. Furthermore, the controller 9 includes a serial-to-parallel (S/P) converter 91, an image processing unit 92, an input unit 93, a recorder 94, and a control unit 95. The base unit 314 may be fixed to a ceiling, a wall surface, or the like to support the support unit 313 instead of being movably provided on the floor surface.
The microscope unit 312 has, for example, a cylindrical shape and includes the above-described medical imaging device inside. Specifically, the medical imaging device has a configuration similar to that of the endoscopic camera head 5 according to the first embodiment described above. For example, the microscope unit 312 includes an optical system 51, a driver 52, an image sensor 53, a cut filter 54, an analog-to-digital (A/D) converter 55, a parallel-to-serial (P/S) converter 56, an imaging recorder 57, and an imaging control unit 58. In addition, a switch that receives an input of an operation instruction of the microscope device 310 is provided on a side surface of the microscope unit 312. A cover glass for protecting the inside is provided on an aperture surface of a lower end portion of the microscope unit 312 (not illustrated).
In the surgical microscope system 300 configured as described above, a user such as a surgical operator moves the microscope unit 312, performs a zoom operation, or switches illumination light while operating various switches in a state of holding the microscope unit 312. Note that the shape of the microscope unit 312 is preferably a shape elongated in the observation direction so that the user can easily hold and change the viewing direction. Therefore, the shape of the microscope unit 312 may be a shape other than the columnar shape, and may be, for example, a polygonal columnar shape.
According to the fifth embodiment described above, also in the surgical microscope system 300, similarly to the first embodiment described above, the surgical operator can easily grasp the characteristic region (lesion site) being left unremoved.
Next, a sixth embodiment will be described. In the above-described first embodiment, the cut filter 54 is provided on the light-receiving surface side (incident surface side) of the G pixel, but in the sixth embodiment, the cut filter 54 is provided on the light-receiving surface side (incident surface side) of each of the R pixel, the G pixel, and the B pixel. Hereinafter, an endoscopic system according to the sixth embodiment will be described. Note that the same components as those of the endoscopic system 1 according to the first embodiment described above are denoted by the same reference numerals, and a detailed description thereof will be omitted.
The cut filter 54C is arranged on the optical axis between the optical system 51 and the image sensor 53. The cut filter 54C blocks most of the light in the short wavelength band including the wavelength band of the excitation light (allows a part of the excitation light to pass through) and allows light in the wavelength bands on the longer wavelength side than the wavelength band with which the most of the light is blocked to pass through.
As indicated by the polygonal line LFF in
An overview of observation modes executed by the endoscopic system 400 is now described. Moreover, the description below is given in the order of a thermal treatment imaging observation mode, and a normal light imaging observation mode.
First, the thermal treatment imaging observation mode will be described.
As illustrated in the graph G11 of
More specifically, as illustrated in the graph G121 of
Further, as illustrated by the polygonal line LNG of the fluorescence characteristics in the graph G121 of
The image processing unit 92 then acquires image data (raw data) from the image sensor 53 of the endoscopic camera head 5C and performs image processing on each signal value of the G pixel and B pixel included in the acquired image data to generate a fluorescence image (pseudo-color image). In this case, the signal value of the G pixel includes information regarding fluorescence emitted from the thermal treatment region. In addition, the B pixel includes information regarding the background from the living tissue of the subject including the thermal treatment region. Therefore, the image processing unit 92 performs processing similar to that of the first embodiment described above to generate a fluorescence image. Specifically, the image processing unit 92 generates a fluorescence image (pseudo-color image) by performing demosaic processing, processing of calculating an intensity ratio for each pixel, processing of determining a fluorescence region and a background region, and image processing of different parameters on each of a color component signal (pixel value) of a pixel located in the fluorescence region and a color component signal (pixel value) of a pixel located in the background region. Then, the image processing unit 92 outputs the thermal treatment image to the display 7. Here, the fluorescence region is a region in which fluorescence information is superior to background information. In addition, the background region refers to a region in which background information is superior to fluorescence information. Specifically, in a case where the intensity ratio between the reflected light component signal corresponding to the background information and the fluorescence component signal corresponding to the fluorescence information included in the pixel is greater than or equal to a predetermined threshold value (for example, greater than or equal to 0.5), the image processing unit 92 determines the region as the fluorescence region, and in a case where the intensity ratio is less than the predetermined threshold value, the image processing unit determines the region as the background region.
As described above, in the thermal treatment imaging observation mode, it is possible to easily observe the living tissue O1 (thermal treatment region) to be subjected to thermal treatment by an energy device or the like.
Next, the normal light imaging observation mode will be described.
As illustrated in the graph G211 of
Subsequently, the image processing unit 92 acquires image data (raw data) from the image sensor 53 of an endoscopic camera head 5A and performs image processing on the signal values of each of the R, G, and B pixels included in the acquired image data to generate a white-light image. In this case, the image processing unit 92 performs white balance adjustment processing that adjusts the white balance so that the ratio of the red, green, and blue components is constant because the blue component included in the image data is smaller than that in the white-light imaging observation in the related art.
Thus, in the normal light imaging observation mode, it is possible to observe a natural white-light image even in the case where the cut filter 54C is arranged.
That is, the endoscopic system 400 performs processing similar to that of the above-described first embodiment, determines the background region and the fluorescence region in the thermal treatment imaging observation mode, and applies different image processing parameters to each of the background region and the fluorescence region, thereby generating a fluorescence image in which the fluorescence region is emphasized from the background region and displaying the fluorescent image on the display 7. Furthermore, even in a case where the cut filter 54C is arranged in the normal light imaging observation mode and the narrow-band imaging observation mode, the endoscopic system 400 can generate a white-light image and a pseudo-color image because the component of the light in the blue wavelength band incident on the B pixel and the component of the light in the green wavelength band incident on the G pixel are only smaller than those in a state where the cut filter 54C is not arranged.
According to the sixth embodiment described above, the same effects as those of the first embodiment described above are obtained, and the cut filter 54C as an optical element is provided, so that it is possible to prevent the fluorescence from the heat treatment region from being buried in the reflected light and the return light reflected by the living tissue.
Various embodiments can be formed by appropriately combining a plurality of components disclosed in the endoscopic system according to the above-described first to fourth and sixth embodiments or the surgical microscope system according to the fifth embodiment of the present disclosure. For example, some components may be deleted from all the components described in the endoscopic system or the surgical microscope system according to the embodiment of the present disclosure described above. Furthermore, the components described in the endoscopic system or the surgical microscope system according to the embodiment of the present disclosure described above may be appropriately combined.
In addition, in the first to sixth embodiments of the present disclosure, an example of being used for transurethral resection of bladder tumors has been described, but the disclosure is not limited thereto, and can be applied to various treatments of excision of a lesion by, for example, an energy device or the like.
Furthermore, in the endoscopic system or the surgical microscope system according to the first to sixth embodiments of the present disclosure, the above-described “unit” can be replaced with “means”, “circuit”, or the like. For example, the control unit can be replaced with a control means or a control circuit.
Note that, in the description of the flowcharts in the present specification, the context of processing between steps is clearly indicated using expressions such as “first”, “thereafter”, and “subsequently”, but the order of processing necessary for implementing the disclosure is not uniquely determined by these expressions. That is, the order of processing in the flowcharts described in the present specification can be changed within a range without inconsistency.
Note that the present disclosure can also take the following procedures.
A procedure for the transurethral resection of bladder tumors, including:
According to the present disclosure, it is possible to easily recognize a characteristic region being left unremoved.
Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the disclosure in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.
This application is a continuation of International Application No. PCT/JP2020/036993, filed on Sep. 29, 2020, the entire contents of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2020/036993 | Sep 2020 | WO |
| Child | 18127051 | US |
