Patent Grant
10111614
The present invention relates to a device for observing a sentinel lymph node (SLN) in a human body. More particularly, the present invention relates to a device for observing an SLN by detecting near-infrared (NIR) fluorescence caused by a fluorescent material such as indocyanine green (ICG) at the SLN and a method for detecting NIR fluorescence at an SLN.
Sentinel lymph node (SLN) is a lymph node that cancer cells firstly reach when the tumor is directly metastasized via lymph nodes. SLN biopsy is a method for identifying metastasis by finding an SLN through injection of a color pigment into cancer tissue, excising the SLN and then performing a histopathologic examination on the excised SLN. If a cancer is detected in the SLN, all lymph nodes around the cancer are excised. However, if the cancer is not detected in the SLN, it is determined that the cancer is not metastasized to the SLN, and the excision of the SLN can be minimized.
As such, the SLN biopsy can minimize, through minimum excision of lymph nodes, side effects and complications, which may be generated by completely excising peripheral lymph nodes together with cancer tissues in the existing operations. The SLN biopsy has been already performed as a standard surgical technique in breast cancer, melanoma, etc. In addition, the SLN biopsy is being extended to surgical operations in the fields of all cancers including lung cancer, esophageal cancer, stomach cancer, thyroid cancer, gynecological cancer, urologic cancer, laryngeal cancer, etc.
In the SLN biopsy, the position of the SLN cannot be exactly detected with the naked eye. Hence, a nuclear medicine imaging method using a radioactive isotope as a tracer, an imaging method using a magnetic fluid having magnetism, an optical imaging method using a vital dye, or a method simultaneously using a radioactive isotope and a vital dye is used in the SLN biopsy.
An optical imaging method and various fluorescent materials as vital dyes have been studied to minimize radiation exposure to a patient and to detect the SLN. For example, studies on an optical imaging probe for SLN detection including a poly-gamma-glutamic acid and an optical imaging dye complex have been conducted. Among fluorescent dyes, the use of indocyanine green (ICG) is permitted in many countries including FDA in USA. The ICG allows light to be excited in a near-infrared (NIR) region, and generates fluorescent light. In addition, the internal structure of human tissue distributed up to a depth of 10 to 20 mm can be observed using the ICG, and NIR fluorescent light can be observed even at a place where white visible light is thrown, such as an operating room.
However, such an NIR fluorescent dye cannot be seen with the naked eyes, and thus devices capable of observing NIR fluorescent light have been developed. As a result, a device called hyper eye medical system (HEMS) was recently developed for observing NIR during surgery.
The HEMS device is an imaging device for observing ICG fluorescent light. The HEMS device simultaneously measures visible light and NIR using a single camera 3 installed therein, and NIR fluorescent light can be observed even in an environment with bright external illumination. The HEMS device is shown in
The HEMS device employs, together with a white light source 1, a light source (NIR LED) having a wavelength of 760 nm as an excitation light source 2. However, through the device having the structure described above, the external appearance of an opened organ can be seen only during an abdominal surgery, and there often occurs a confusion among the color of visible light, an ICG fluorescent image in an NIR combined image and an image caused by glare of white light reflected on the surface of a human body.
Meanwhile, Japanese Patent Application Publication No. 2006-340796 has disclosed a system for detecting an SLN from a fluorescent image. Particularly, in Japanese Patent Application Publication No. 2006-340796, white light including excitation light is emitted by a xenon lamp, and an excitation light filter is set to allow light of a wavelength band of 385 to 435 nm to be transmitted therethrough. In the case of fluorescent light and background light, obtained from an object to be measured, a light-shielding filter is inserted between an object to be observed and a single CCD chip to allow the fluorescent light and the background light to be transmitted therethrough. Thus, an image is picked up in the CCD chip. In addition, the image is processed by a TV camera to be shown as a fluorescent image through a monitor. However, the system is a device for a contrast medium which emits fluorescent light in visible light such as 5-ALA. The device is not suitable for observing fluorescent light in NIR, such as ICG.
In relation to this, U.S. Patent Application Publication No. 2011/0063427 discloses an imaging system for providing full-color reflection light and NIR image. The imaging system for obtaining the NIR and full-color image includes a light source which supplies visible light and NIR light to an object to be observed, and a camera having a plurality of image sensors which independently detect blue reflection light and green reflection light from the object to be observed, and alternately detect red reflection light and NIR light generated from the object to be observed.
A controller for transmitting a signal to the light source and the camera controls consecutive blue and green lights to be irradiated onto the object to be observed, and red light and NIR excitation light are synchronized by periodically switching on-off the light source and the camera so that red and NIR fluorescent images are alternately obtained from the camera.
A red reflection light spectrum and an NIR light spectrum are alternately obtained from the same image sensor through the switching synchronization between the light source and the camera. Thus, the red reflection light spectrum provides, together with the blue and green reflection lights, a full-color image, or the NIR light spectrum provides an NIR fluorescent image. However, the synchronization between the light source and the camera makes the device complicated.
Meanwhile, in an imaging system for simultaneously observing a wide range of spectra from visible light (400 to 700 nm) to NIR light (700 to 900 nm), a chromatic aberration correction is required to adjust the focus on the focal plane of an image obtaining chip such as a CCD sensor. U.S. Patent Application Publication No. US2011/0249323 A1 discloses a special optical coupler for correcting a chromatic aberration in an endoscope device. The disclosed optical coupler is configured with an afocal prism assembly and an imaging optics. The afocal prism assembly is configured with prisms having different refractive indices, and dichroic coating is performed at the boundary between the prisms, so that an incident wavelength is incident onto an appropriate prism. The chromatic aberration between visible light and NIR light, which pass through prisms having different refractive indices, is corrected by correcting the difference in light path length between the visible light and the NIR light. However, a specific optical system is required to remove the chromatic aberration in such a manner, and the existing optical couplers cannot be used.
In addition, when a visible light image and an NIR image are respectively displayed in two different screen windows of the same monitor or when the two images are displayed to overlap with each other, it is difficult to distinguish the visible light image from the NIR image.
Basically, the distinguishment of the SLN from a non-SLN depends on the intensity of a fluorescent signal. Even when the same device is used, the intensity of the fluorescent signal is considerably changed depending on a distance to an object to be observed, parameters (gain, shutter and frame) set in the detection sensitivity of a TV system, the intensity of excitation light, etc. Therefore, a standard measuring method is essentially required to ensure the reliability of a detection result.
The present invention provides a device and a method for detecting near-infrared (NIR) fluorescence at a sentinel lymph node (SLN), which can detect NIR fluorescence and an SLN with high accuracy in the implementation of a composite image obtained by reproducing NIR fluorescence emitted by excitation light from a fluorescent material such as indocyanine green (ICG) together with a visible light image.
In one aspect, the present invention provides a device for detecting NIR fluorescence at an SLN, the device including: a white light source configured to emit white light; an NIR excitation light source configured to NIR excitation light; an optical analyzing assembly configured to transmit reflection light and NIR fluorescence from an object to be observed; a multispectral image processing unit configured to detect white reflection light and NIR fluorescence, transmitted from the optical analyzing assembly, and process the detected white reflection light and NIR fluorescence as a visible (VIS) reflection light image signal and an NIR fluorescence image signal; and a display unit configured to output a composite image obtained by combining the VIS reflection light image signal and the NIR fluorescence image signal, processed by the multispectral image processing unit, wherein the multispectral image processing unit splits the detected white reflection light into red (R), green (G) and blue (B) image signals, and then performs image processing so that the VIS reflection light image signal is implemented with red (R) and green (G) in a pixel from which an NIR fluorescence image signal is not detected, and the NIR fluorescent image signal is displayed with only blue (B) in a pixel from which the NIR fluorescent image signal is detected.
In another aspect, the present invention provides a device for detecting NIR fluorescence at an SLN, the device including: a white light source configured to emit white light; an NIR excitation light source configured to NIR excitation light; an optical analyzing assembly configured to transmit reflection light and NIR fluorescence from an object to be observed; a multispectral image processing unit configured to detect white reflection light and NIR fluorescence, transmitted from the optical analyzing assembly, and process the detected white reflection light and NIR fluorescence as a VIS reflection light image signal and an NIR fluorescence image signal; and a display unit configured to output a composite image obtained by combining the VIS reflection light image signal and the NIR fluorescence image signal, processed by the multispectral image processing unit, wherein the multispectral image processing unit splits the detected white reflection light into red (R), green (G) and blue (B) image signals, and relatively compares the intensity of a blue (B) image signal of the white reflection light with that of the NIR fluorescence image signal for each pixel, wherein, when the intensity of the blue (B) image signal of the white reflection light is greater than that of the NIR fluorescence image signal, the multispectral image processing unit performs image processing so that the blue (B) image signal of the white reflection light is displayed together with red (R) and green (G) image signals in a corresponding pixel, and wherein, when the intensity of the NIR fluorescence image signal is greater than that of the blue (B) image signal of the white reflection light, the multispectral image processing unit performs image processing so that the NIR fluorescence image signal is displayed with only blue (B) in the corresponding pixel.
In an exemplary embodiment, the multispectral image processing unit may include a beam splitter configured to form a VIS channel and an NIR channel by splitting a light path of the white reflection light and the NIR fluorescence, transmitted from the optical analyzing assembly.
In another exemplary embodiment, the beam splitter may be a beam splitting prism that selectively transmits or reflects light in a selected wavelength range.
In still another exemplary embodiment, the beam splitter may be a dichroic prism.
In still another exemplary embodiment, the multispectral image processing unit may be configured to have two image sensors respectively disposed on the light paths split by the beam splitter.
In still another exemplary embodiment, the two image sensors may be a color image sensor disposed on the VIS channel and a monochrome image sensor disposed on the NIR channel.
In still another exemplary embodiment, the monochrome image sensor may be disposed to be spaced apart from the color image sensor at a predetermined distance (A) to correct an axial chromatic aberration.
In still another exemplary embodiment, optical filters may be disposed between the respective image sensors and the beam splitter.
In still another exemplary embodiment, a gain amplifier and an analog/digital (A/D) converter may be connected to each image sensor.
In still another exemplary embodiment, the multispectral image processing unit may include digital image processor configured to generate a VIS reflection light image signal and an NIR fluorescence image signal by analyzing and processing a collected image signal.
In still another exemplary embodiment, the multispectral image processing unit may further include a common timing generator for the two image sensors, and the digital image processor may perform an image processing operation in synchronization with the timing generator.
In still another exemplary embodiment, the digital image processor may automatically control parameters of the gain amplifier so that reference light is constant, using reflection excitation light or white reflection light as the reference light.
In still another exemplary embodiment, the device may further include a computer configured to form a composite image by combining the VIS reflection light image signal and the NIR fluorescence image signal, analyzed and processed by the digital image processor.
In still another exemplary embodiment, the device may further include a transmitter/receiver configured to transmit, to the computer, the VIS reflection light image signal and the NIR fluorescence image signal, analyzed and processed by the digital image processor.
In still another exemplary embodiment, the white light source may be a metal halide lamp or xenon lamp.
In still another exemplary embodiment, the NIR excitation light source may be a laser light source that emits light of 800±20 nm.
In still another exemplary embodiment, the optical analyzing assembly may be an optical analyzing assembly that constitutes any one imaging system selected from a laparoscope, a hard endoscope, a soft endoscope, a camera and an operating microscope.
In still another exemplary embodiment, the device may further include an optical adaptor configured to connect the optical analyzing assembly to the multispectral image processing unit.
In still another exemplary embodiment, the fluorescent material used to observe NIR fluorescence may be indocyanine green (ICG).
In still another exemplary embodiment, the output of the excitation light irradiated from the NIR excitation light source may be controlled to have a pulse mode with a frequency lower than the frame rate of the multispectral image processing unit.
In still another exemplary embodiment, the device may further include a standard sample configured to have a fluorescence intensity corresponding to that of the SLN under the same condition.
In still another exemplary embodiment, the intensity of a fluorescence signal from the standard sample may be compared with that of a fluorescence signal from the object to be observed, and the object to be observed may be Determined as the SLN only when the intensity of the fluorescence signal from the object to be observed is greater than that of the fluorescence signal from the standard sample.
In still another exemplary embodiment, the device may further include a light-shielding filter configured to shield reflection excitation light transmitted to the multispectral image processing unit.
In still another aspect, the present invention provides a method for detecting NIR fluorescence at an SLN, the device including: irradiating white light and NIR excitation light onto an object to be observed; collecting white reflection light and NIR fluorescence from the object to be observed; splitting the white reflection light into red (R), green (G) and blue (B) light signals; generating the NIR fluorescence as a blue (B) image signal, and generating a composite image, using the generated blue (B) image signal together with red (R) and green (G) signals of the white reflection light; and outputting the generated composite image to a display unit.
In a further aspect, the present invention provides a method for detecting NIR fluorescence at an SLN, the device including: irradiating white light and NIR excitation light onto an object to be observed; collecting white reflection light and NIR fluorescence from the object to be observed; splitting the white reflection light into red (R), green (G) and blue (B) light signals; relatively comparing the intensity of a blue (B) image signal of the white reflection light with that of the NIR fluorescence image signal for each pixel; selecting any one signal of which intensity is greater than the other as a blue (B) light signal of a corresponding pixel, based on the compared result, wherein, when the intensity of the blue (B) image signal of the white reflection light is greater than that of the NIR fluorescence mage signal, the blue (B) image signal of the white reflection light is displayed together with red (R) and green (G) signals of the white reflection light in the corresponding pixel, and wherein, when the intensity of the NIR fluorescence mage signal is greater than that of the blue (B) image signal of the white reflection light, the NIR fluorescence image signal is displayed with only blue (B) in the corresponding pixel; and generating a composite image and outputting the generated composite image to a display unit.
In an exemplary embodiment, the irradiating of the white light and the NIR excitation light further may include controlling the output of NIR laser irradiated onto the object to be observed to have a pulse mode with a frequency lower than the frame rate of a multispectral image processing unit, so that the NIR excitation light has a discontinuous light irradiation output.
In another exemplary embodiment, in the collecting of the white reflection light and the NIR fluorescence, a fluorescence signal of a standard sample may be simultaneously collected as well as the object to be observed.
In still another exemplary embodiment, the method may further include analyzing the composite image output to the display unit. In the analyzing of the composite image, the object to be observed may be finally Determined as the SLN when the intensity of a fluorescence signal of the object to be observed is greater than that of a fluorescence signal of the standard sample, and the object to be observed may not be Determined as the SLN when the intensity of the fluorescence signal of the standard sample is greater than that of the fluorescence signal of the object to be observed.
In still another exemplary embodiment, the method may further include measuring fluorescence of the standard sample before the irradiation of the white light and the NIR excitation light.
In still another exemplary embodiment, the method may further include irradiating white light and excitation light onto the standard sample and setting white reflection light and reflection excitation light from the standard sample as reference light, before measuring the fluorescence of the standard sample.
In still another exemplary embodiment, the irradiating of the white light and the NIR excitation light may further include controlling parameters so that the intensity of the reference light is constant.
In still another exemplary embodiment, the method may further include analyzing the composite image output to the display unit. In the analyzing of the composite image, the object to be observed may be finally Determined as the SLN when the intensity of a fluorescence signal of the object to be observed is greater than that of a fluorescence signal of the standard sample, and the object to be observed may not be Determined as the SLN when the intensity of the fluorescence signal of the standard sample is greater than that of the fluorescence signal of the object to be observed.
In still another exemplary embodiment, the generating of the composite image may include generating the composite image by performing image processing so that the blue (B) of the NIR fluorescence is displayed in a pixel from which the NIR fluorescence is detected, and the red (R) and the green (G) of the white reflection light are displayed in a pixel from which the NIR fluorescence is not detected.
Other aspects and exemplary embodiments of the invention are discussed infra.
As described above, the device and the method for detecting the NIR fluorescence at the SLN according to the present invention have advantages as follows.
First, although the existing imaging system is used, it is possible to detect an SLN with high accuracy for a short period of time. Accordingly, it is possible to improve the accuracy in deciding the presence of cancer metastasis and to minimize the excision of an SLN.
Second, since it is unnecessary that the positions of focal surfaces of the NIR and VIS sensors should correspond to each other, it is unnecessary to correct a chromatic aberration in the laparoscope having an optical system, particularly an optical coupler.
Third, since the device and the method can be easily apply to existing laparoscopes, etc., it is possible to minimize additional cost consumption for device improvement.
The above and other features of the invention are discussed infra.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
The above and other features of the present invention will now be described in detail with reference to certain exemplary embodiments thereof illustrated the accompanying drawings which are given hereinbelow by way of illustration only, and thus are not limitative of the present invention, and wherein:
It should be understood that the appended drawings are not necessarily to scale, presenting a somewhat simplified representation of various preferred features illustrative of the basic principles of the invention. The specific design features of the present invention as disclosed herein, including, for example, specific dimensions, orientations, locations, and shapes will be determined in part by the particular intended application and use environment.
In the figures, reference numbers refer to the same or equivalent parts of the present invention throughout the several figures of the drawing.
Hereinafter reference will now be made in detail to various embodiments of the present invention, examples of which are illustrated in the accompanying drawings and described below. While the invention will be described in conjunction with exemplary embodiments, it will be understood that present description is not intended to limit the invention to those exemplary embodiments. On the contrary, the invention is intended to cover not only the exemplary embodiments, but also various alternatives, modifications, equivalents and other embodiments, which may be included within the spirit and scope of the invention as defined by the appended claims.
Hereinafter, a device for detecting near-infrared (NIR) fluorescence at a sentinel lymph node (SLN) according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.
Although it has been illustrated in
Hereinafter, the case where the laparoscope is implemented as the optical analyzing assembly will be illustrated with reference to
As shown in
In the device configured as described above, excitation light in an NIR wavelength band and white light, which are emitted from the combined white-NIR illuminator 10, is transmitted to an object A to be observed through an optical transmission module 31 included in the laparoscope 30 as the optical analyzing assembly by passing through an optical guide 20. Various biological objects to be observed may be considered as the object A to be observed, and a standard sample is used to be compared and observed with these objects. The combined white-NIR illuminator 10 provided with a lamp for emitting white light and a laser for emitting NIR light is used to obtain the combined white-NIR light.
Preferably, a white light source of the combined white-NIR illuminator 10 may be configured with a metal halide lamp or xenon lamp, and an NIR excitation light source of the combined white-NIR illuminator 10 may be configured with an NIR laser light source that emits light of 800±20 nm.
In this embodiment, the combined white-NIR illuminator integrally provided with the white light source and the NIR excitation light source has been described as an example, but the present invention is not limited to the form of the combined white-NIR illuminator. That is, any form of the may be used as long as the combined white-NIR illuminator provides the white light and the NIR excitation light to the object to be observed.
As such, the white light and the NIR excitation light from the combined white-NIR illuminator 10 are irradiated onto the object to be observed through the optical transmission module 31 of the laparoscope 30, and visible (VIS) reflection light, laser excitation light and NIR fluorescence are emitted from the object to be observed.
In relation to this,
In this embodiment, the device is configured so that the light (the VIS reflection light+the NIR laser excitation light and fluorescence) emitted from the object to be observed can be transmitted to the multispectral image processing system 50. Hence, the device may be configured to include the laparoscope 30 and an optical adaptor 40 as shown in
Thus, the light (the VIS reflection light+the NIR laser excitation light and fluorescence) emitted from the object to be observed is transmitted to the multispectral image processing system 50 through an optical imaging module 32 included in the laparoscope and an optical coupler 41 included in the optical adaptor 40.
Meanwhile, in this embodiment, two image sensors for respectively processing an image in VIS and NIR regions are included as shown in
Here, a light-shielding filter 42 may be installed to prevent the laser excitation light in the NIR wavelength band from being penetrated into the multispectral image processing system 50 through the path of secondary lights reflected from the object to be observed and to allow light in the other wavelength bands to be transmitted therethrough.
Since the white reflection light for a background of the object to be observed and the fluorescence to be detected are required to be processed in the multispectral image processing system, the light-shielding filter 42 is used to shield a large amount of reflection excitation light to be detected.
Meanwhile, a beam splitter 51 may be installed in the multispectral image processing system 50. The beam splitter 51 splits the secondary light from the object to be observed into two lights, i.e., VIS and NIR. In addition, optical filters 52 and 53 are used to select one of spectra of the lights divided into two channels.
Here, a beam splitting prism using a prism may be used as the beam splitter. More preferably, a dichroic prism may be used as the beam splitter.
In the device according to this embodiment, an image sensor capable of sensing the wavelength of the split light is installed in each channel. That is, a color image sensor 54 and the optical filter 52 are installed in the VIS channel where the VIS is split from the beam splitter 51, and a monochrome image sensor 55 and an optical filter 53 are installed in the NIR channel where the NIR is split from the beam splitter 51.
The two sensors are controlled by the same video processing and control unit 60 including a common timing generator 61. A first gain amplifier 62 and a first analog/digital (A/D) converter 64 for the color image sensor 54 and a second gain amplifier 63 and a second A/D converter 65 for the monochrome image sensor 55 are installed in the video processing and control unit 60. As a result, a digital image including a white light image and an NIR fluorescence image is formed from these components of the video processing and control unit 60.
In relation to this, the device in this embodiment is configured to include a digital image processor 66 for analyzing and processing image signals collected in the multispectral image processing system and generating a VIS reflection light image signal and an NIR fluorescence image signal.
That is, the two-channel digital image processor 66 included in the video processing and control unit 60 generates a control signal for independently controlling amplification coefficients in the first and second gain amplifiers 62 and 63, and the control signal is required to perform an automatic gain control (AGC) condition. Here, if the AGC condition is set, the gain can be controlled so that the intensity of a predetermined reference light such as reflection excitation light or white reflection light.
The digital image processor 66 performs an operation in synchronization with the timing generator 61. The digital image processor 66 performs processing on a video signal and then transmits the video signal to computer 70 through a receiver/transmitter 67. The computer 70 performs image-processing on a VIS reflection light image and an NIR fluorescence image and makes the two images as a composite image to be displayed on a screen of the display unit 80.
As shown in
In each apparatus, a multispectral 2-chip TV system may be used as the multispectral image processing system as shown in
Particularly, in the ICG laparoscope, a combined VIS-NIR light source 10 including a VIS light source 12 as the white light source and a laser 11 as the NIR light source is used as the white light source in order to irradiate NIR excitation light and the white light at the same time.
In the ICG videoscope, a separated surgical lamp and an NIR laser 11 are used rather than the combined VIS-NIR light source, and a camera lens 33 is used as the imaging system rather than the laparoscope.
Meanwhile, when image spectra in two wavelength bands of the VIS and the NIR are detected and recorded at the same time, a problem of chromatic aberration occurs in the imaging system designed and manufactured suitable for a VIS spectrum region. The chromatic aberration may be divided into an axial (longitudinal) chromatic aberration and a transverse (lateral) chromatic aberration, and a problem of the axial chromatic aberration is particularly important.
Referring to
In an embodiment of the present invention, an image sensor is installed in each of the VIS and NIR channels so as to correct such a chromatic aberration.
Particularly, as shown in
Thus, the beam splitter splits the light into VIS and NIR, and the split VIS and NIR are respectively incident onto the color image sensor and the NIR image sensor.
According to this embodiment, in order to correct the axial chromatic aberration, the position of the NIR image sensor is controlled to be relatively moved by a value of Δ, as compared with that of the color image sensor.
As a result, the axial chromatic aberration of each of the color and NIR image sensors having the controlled distance is corrected according to the moved position.
Thus, in the device according to this embodiment, the axial chromatic aberration is corrected without any increase in cost, caused as the separate VIS and NIR imaging system is used, and installation of a complicated optical module, so that it is possible to precisely control the focuses of the two images at the same time.
Meanwhile, the present invention provides a new method for simultaneously implementing NIR and VIS images and a device for detecting NIR fluorescence at an SLN, which is manufactured by the method.
In order to implement two images overlapped with each other using a general monitor, there occurs a problem how to distinguish an NIR fluorescence image belonging to a VIS reflection light image. In the present invention, a new method according to color contrast is used to easily distinguish the NIR fluorescence image.
In the color contrast, characteristics of a biological tissue having a yellow-red color tone in white reflection light are used.
That is,
An image (NIR Fluorescence) obtained from the NIR fluorescence is also shown in
When comparing the images of
Thus, in an embodiment of the present invention, an image of blue (B) is substituted as the NIR fluorescence image in the white reflection light, so that it is possible to implement the NIR fluorescence image together with the white reflection light.
In the configuration of a composite image under the condition of a mode in which white reflection light and ICG fluorescence are combined (Reflected White Image+ICG Fluorescence Image), the NIR image fluorescence image is selected other than the image of blue (B) image in a reflection light image background of red (R) and green (G).
The blue obtained from the white reflection light is not a characteristic element for the image of a tissue, and thus the SLN displayed by the NIR fluorescence is easily distinguished from other tissues. In addition, the SLN is distinguished from hot spots shown in the white reflection light. The white reflection light image forming the image background in the composite image is expressed by only the channels of red (R) and green (G). Since the hot spot has no blue, the hot spot is shown as yellow. Meanwhile, the NIR fluorescence image is shown as blue (B).
That is, a composite image output using the white reflection light (RG) and the NIR fluorescence (B) is shown at the right lower end of
Accordingly, it is possible to easily observe an SLN related to the NIR fluorescence together with the exact structure of a tissue.
A method for detecting NIR fluorescence at an SLN according to this embodiment will be described in connection with the device shown in
Each of the signals respectively converted through the image sensors passes through the gain amplifier and the A/D converter. Then, the signals are respectively processed as a VIS reflection light image signal and an NIR fluorescence image signal in the digital image processor. In this embodiment, the white reflection light and the NIR fluorescence are respectively formed with an image of RG and an image of B through the digital image processor. In this case, the synchronization is performed by the timing generator 61 in the process of forming the images, and thus the fluorescence image signal is simultaneously obtained together with a background caused by white light so that it is possible to obtain VIS reflection light and NIR fluorescence image signals for implementing a composite image.
The obtained are transmitted to the computer through the transmitter/receiver, and the computer combines the transmitted VIS reflection light and NIR fluorescence image signals, thereby implementing a composite image through the display unit.
In relation to this,
As shown in
Next, the signal of the fluorescence image is generated as a signal of B (Fluorescence Image[i, j]B), and a composite image is generated using the signal of B together with signals of R and G, thereby outputting the composite image.
That is, in this embodiment, when the composite image is generated, the blue (B) of the NIR fluorescence is displayed in the pixel where the NIR fluorescence is detected, and the red (R) and green (G) of the white reflection light are displayed in the pixel where the NIR fluorescence is not detected.
Meanwhile, unlike Embodiment 1, in another preferred embodiment, the amplitudes of the blue (B) of the white reflection light and the NIR fluorescence are relatively compared. Thus, one of both, of which intensity is relatively strong, is selected for each pixel, thereby implementing a composite image.
Basically, since the intensity of the blue (B) of the white reflection light is different from that of the NIR fluorescence, the comparison between the absolute amplitudes of the blue (B) of the white reflection light and the NIR fluorescence is not appropriate in detecting the position of an SLN. Therefore, the method of relatively comparing the amplitudes of the blue (B) of the white reflection light and the NIR fluorescence may be used in the preferred embodiment of the present invention.
For example, since the intensity of the NIR fluorescence signal is much smaller than that of the white reflection light, the NIR fluorescence signal may be amplified by multiplying the NIR fluorescence signal by a predetermined gain, and the amplitude of the amplified NIR fluorescence signal may be then relatively compared with that of the white reflection light.
In this embodiment, the intensities of the blue (B) of the white reflection light and the NIR fluorescence are relatively compared for each pixel, so that one of both the blue (B) of the white reflection light and the NIR fluorescence, of which intensity is strong, is selected and output as the blue (B).
Thus, when the intensity of the blue (B) of the white reflection light is stronger than that of the NIR fluorescence, the blue (B) of the white reflection light is selected, thereby implementing RGB colors of the white reflection light together with the red (R) and the green (G). When the intensity of the NIR fluorescence is stronger than that of the blue (B) of the white reflection light, the NIR fluorescence is selected as the blue (B). Hence, the NIR fluorescence is output as a composite image together with the red (R) and the green (G) of the white reflection light.
As shown in
The composite image obtained according to this embodiment is shown in
In this case, the hot spot shown in the white reflection light is shown as white as the original color at the early stage, and the color tone of the tissue is not changed at a part where the intensity of the NIR signal is weaker than that of the blue signal in the white reflection light.
Meanwhile, in the device and the method according to the present invention, the SLN is more exactly identified through an additional process of analyzing a composite image provided to the display unit.
Particularly, in the present invention, a standard sample may be used to more exactly identify the SLN.
Specifically, in the present invention, the ICG is basically used as a contrast medium to distinguish the SLN from non-SLNs, and the SLN is distinguished from the non-SLNs through a difference in intensity between fluorescence signals generated from the ICG when laser excitation light is irradiated. The intensity of the fluorescence signal depends on several elements. That is, the intensity of the fluorescence signal depends on the intensity of light irradiated onto the object to be observed, the sensitivities of VIS and NIR detection sensors, parameters (gain and shutter) of the video processing and control unit, the measurement distance between the device and the object to be observed, and the like. Among these elements, it is difficult to control the measurement distance. This is because the measurement distance is frequently changed due to histological characteristics of the object to be observed, close observation of a required part to be observed, or the like.
Therefore, a fluorescence standard sample having a fluorescence intensity similar to that of the SLN is required to exactly identify the SLN. The fluorescence standard sample may be prepared by basically using an ICG solution in which serum of blood or albumin is added. The ICG solution is generally used as a fluorescence activator for increasing fluorescence intensity. As the activator, other materials other than an organic material which can increase fluorescent intensity and more stably maintain fluorescence may be used. When a beauty shampoo is used as a candidate of the activator in an experiment performed by this research, it can be seen that the fluorescence intensity of the ICG solution in a sealed plastic container is maintained without any change for at least one month. When the ICG solution with a predetermined concentration is used, the fluorescence intensity of the standard sample may be controlled similarly to that of the SLN.
The method of identifying the SLN may be divided into two different methods depending on whether the standard sample is disposed at a side of the object to be observed or not.
First, when considering that the standard sample is disposed at a side of the object to be observed, the standard sample may be disposed at a side of an SLN to be observed in order to compare the fluorescence intensity of the standard sample with that of a part to be operated. The example in which the standard sample is disposed at a side of the SLN that is an object to be observed is shown in
In this case, the comparison between the intensity of a fluorescence signal of the object to be observed (lymph node) and the intensity of a fluorescence signal of the standard sample may be performed with the naked eyes through a monitor or by quantitative estimation. If the fluorescence intensity of the lymph node (FI Lymph Node) is greater than that of the standard sample (FI Standard Sample), the lymph node that is the object to be observed is an SLN. If the fluorescence intensity of the lymph node (FI Lymph Node) is smaller than that of the standard sample (FI Standard Sample), the lymph node is a non-SLN. Here, the fluorescence intensity of the lymph node means a fluorescence signal of the object to be observed.
In relation to this,
That is, referring to the block diagram of
Meanwhile, it is considered when the standard sample is not disposed at a side of the object to be observed.
The comparison between the intensity of the fluorescence signal of the lymph node and the intensity of the fluorescence signal of the standard sample is a direct method. However, when the standard sample is not used, a change in measurement distance from the object to be observed may be automatically calculated using a white reflection light signal.
Therefore, if the parameters (gain and shutter) are automatically controlled so that reference light is always constant, using the reflection excitation light or white reflection light as the reference light, it is possible to obtain the same intensity of the fluorescence, regardless of the distance.
That is, when considering the device of
In relation to this, in this embodiment, white reflection light is used rather than reflection excitation light so as to correct the measurement distance. In this case, a signal generated from light of red (R), green (G) or blue (B) may be used as the reference light. Preferably, an optical signal generated from the channel R that receives less influence from structural characteristics of the tissue to be observed may be used as the reference light.
When the standard sample is not disposed at the side of the object to be observed, a process of identifying an SLN is shown in
First, a standard sample is disposed, and a correcting operation of the system is performed based on measurement values of VIS and NIR signals with respect to the standard sample. The proportional expression according to the distance between white and NIR standard samples is performed by measuring a light intensity of each standard sample, and a diffused reflection value is measured.
After the laparoscope is inserted into a human body, the parameters (gain and shutter) are changed suitable for the intensity of white light, and accordingly, the signal of the white light channel (VIS light channel) is automatically amplified and controlled. In this case, the parameters are equally changed in not only the white light channel but also an NIR channel, thereby removing an error of the measurement distance.
If an increase in fluorescence at a lymph node is observed, the fluorescence intensity of the lymph node is measured.
If the fluorescence intensity of the lymph node to be observed is higher than that of the standard sample, it is determined that the lymph node is an SLN. If the fluorescence intensity of the lymph node to be observed is lower than that of the standard sample, it is determined that the lymph node is a non-SLN.
Meanwhile, the detection of the fluorescence signal at the SLN may be considered through a temporal modulation method together with the color contrast method.
Referring to
Thus, the NIR fluorescence image is periodically glittered in the VIS background image caused by the continuously output white reflection light, so that it is possible to easily identify the NIR fluorescence image.
The temporal modulation method may be independently used. Preferably, the temporal modulation method is used together with the color contrast method of Embodiment 1 or 2, so that it is possible to easily detect a fluorescence signal from the SLN.
Through the temporal modulation method, the output mode of continuous laser is changed into a pulse mode having a repetition rate of 1 to 2 Hz that is much smaller than 15 Hz that is the frame rate of the light detection sensor of the TV system used in the endoscope.
Thus, the NIR image in the continuous VIS image background is periodically glittered due to the pulse mode.
In this embodiment, the temporal modulation method is performed separately from the color contrast method, and thus the VIS image maintains perfect colors (RGB). On the other hand, when the temporal modulation method is used together with the color contrast method of Embodiments 1 and 2, the blue (B) can be replaced in the NIR fluorescence image, and thus the VIS image may be formed with the red (R), the green (G), and the NIR fluorescence image. Alternatively, when the intensity of a blue (B) light signal is greater than that of the NIR fluorescence image, the VIS image may be perfectly formed with the red (R), the green (G) and the blue (B).
The invention has been described in detail with reference to exemplary embodiments thereof. However, it will be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2013-0081980 | Jul 2013 | KR | national |
This application is a continuation of U.S. patent application Ser. No. 14/327,797 filed Jul. 10, 2014, which claims under 35 U.S.C. § 119(a) the priority benefit of Korean Patent Application No. 10-2013-0081980 filed Jul. 12, 2013, the disclosures of all applications of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6059720 | Furusawa et al. | May 2000 | A |
| 6293911 | Imaizumi et al. | Sep 2001 | B1 |
| 20020013531 | Hayashi | Jan 2002 | A1 |
| 20030078477 | Kang et al. | Apr 2003 | A1 |
| 20040010192 | Benaron et al. | Jan 2004 | A1 |
| 20040225222 | Zeng et al. | Nov 2004 | A1 |
| 20070203413 | Frangioni | Aug 2007 | A1 |
| 20090270678 | Scott | Oct 2009 | A1 |
| 20110063427 | Fengler et al. | Mar 2011 | A1 |
| 20110104071 | Lee et al. | May 2011 | A1 |
| 20110249323 | Tesar et al. | Oct 2011 | A1 |
| Number | Date | Country |
|---|---|---|
| 0792618 | Sep 1997 | EP |
| 2006340796 | Dec 2006 | JP |
| Entry |
|---|
| UK Combined Search and Examination Report for application No. GB1411868.1 dated Dec. 1, 2014. |
| Number | Date | Country | |
|---|---|---|---|
| 20180020965 A1 | Jan 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14327797 | Jul 2014 | US |
| Child | 15708144 | US |
