Patent Application
20250031971
The present disclosure relates generally to medical imaging, and more specifically to techniques for identifying at least one lymph node of a subject using fluorescence images.
When a patient is given a diagnosis of cancer, one of the first questions to be answered is whether the cancer has spread from a primary site (e.g., breast, colon, lung) or if it is still contained in the detected area. The answer can have implications in the prognosis of the patient and the treatment prescribed. For the majority of solid tumors, the most powerful indicator of metastatic spread is the presence or absence of cancer cells in the regional lymph nodes.
Lymph nodes are part of the lymphatic system that is widely present throughout the body. The lymphatic system collects fluid that escapes from the cells, arteries, and veins, and returns the fluid, called lymph, back to the heart. The lymph nodes serve as filters in the lymphatic system to filter any foreign bodies from the lymph before it returns to the heart. As tumors secrete fluid and shed cells (i.e., metastasize), the lymphatic system is a likely reservoir for those fluids and cancer cells to deposit. Thus, if the lymph nodes in the area of the tumor can be localized, surgically removed, and evaluated to determine if there are any cancer cells trapped there, the extent of the spread of the cancer can be determined.
Sentinel lymph node biopsy is a method that involves localization of the lymph node that is the first in the lymph node chain to receive lymphatic drainage from a primary tumor site. This first draining node is referred to as the sentinel lymph node (SLN) and has been found to be a highly accurate indicator of the metastatic involvement of the entire lymph node basin in which it is situated. In short, if the sentinel node does not contain cancer, then the rest of the lymph nodes are also likely negative for cancer. Thus, instead of removing all of the lymph nodes in the lymphatic basin in the area of the tumor, only the sentinel node needs to be removed for further analysis.
Currently, the most common techniques for SLN mapping employed include blue dye staining and gamma imaging, which are frequently used together to improve the accuracy of the results.
The first technique involves the injection of a blue dye around the tumor site. The dye flows into the lymphatic basin, which receives lymphatic drainage from the tumor, and the nodes are stained blue. The surgeon can proceed to carefully dissect the nodal basin and visually identify blue-stained lymphatic vessels. These lymphatic vessels are followed to the SLN, the first blue node in the lymphatic chain from the tumor. The SLN is then excised, and its histologic status is determined by pathology. Although this method has been used successfully by several researchers, its success varies with the experience level of the surgeon and can result in a procedure that approaches the invasive level of a full dissection of the nodal basin.
The second technique uses a radiocolloid as the agent injected around the tumor. This radioactive tracer flows into the lymphatics in the same manner as the blue dye and deposits into the SLN. The advantage to this method is that a gamma detection probe is utilized to detect and mark the SLN at the skin pre-incision and intra-operatively. This method allows a small incision to be used, unlike the blue dye, when visual search for nodes is necessary. Once localized, the SLN is excised and its histologic status can be determined.
Recently, in addition to these two established procedures, the indocyanine green (ICG) fluorescence technique for the detection of SLN is found to be a valid and feasible method in clinical practice. This method, however, has limited efficacy in patients with a high BMI as lymph drainage channels and sentinel nodes cannot be adequately visualized due to high degree of signal absorption. Further, the lymph nodes in certain locations (e.g., armpit) are surrounded by layers of fat and are thus hard to visualize. As a result, a surgeon may need to dig through the tissue until a lymph node is visible to the naked eye. Another challenge with the ICG fluorescence technique is that it may be hard to tell from the fluorescence image whether a fluorescence signal is from a lymph node or another source in the body. The identification is done based on a surgeon's knowledge and is thus error-prone.
Disclosed herein are exemplary devices, apparatuses, systems, methods, and non-transitory storage media for automatically identifying one or more lymph nodes of a subject based on fluorescence images such as near-infrared (NIR) images. An exemplary system can perform template matching techniques, machine learning techniques, or any combination thereof. The systems, devices, and methods may be used in association with surgical procedures. Imaging and analysis may be performed pre-operatively, intra-operatively, post-operatively, and during diagnostic imaging sessions and procedures. For example, the system can automatically identify lymph nodes in images acquired in the course of a SLN mapping procedure (such as ICG fluorescence or scintigraphy) and enhance the visualization to allow physicians and surgeons to quickly and efficiently map the relevant sentinel nodes. For example, the system can automatically locate areas containing sentinel lymph nodes in NIR image frames, and these areas can be further visually enhanced to indicate both the shape and the depth of the located nodes.
An exemplary method for identifying at least one lymph node of a subject comprises: obtaining a fluorescence image of a field of view including the at least one lymph node of the subject; obtaining a template image; comparing the template image with the fluorescence image to obtain one or more similarity values; and identifying at least one portion of the fluorescence image that corresponds to a location of the at least one lymph node based on the one or more similarity values.
In some aspects, the method further comprises displaying the identified at least one portion of the fluorescence image.
In some aspects, displaying the identified at least one portion of the fluorescence image comprises displaying the fluorescence image and at least one bounding box indicative of the identified at least one portion of the fluorescence image.
In some aspects, the fluorescence image is displayed as part of an intraoperative video stream during a surgical procedure.
In some aspects, the surgical procedure comprises a sentinel lymph node mapping procedure.
In some aspects, the fluorescence image is a first fluorescence image, the field of view is a first field of view, the subject is a first subject, and the method further comprises: obtaining a second fluorescence image of a second field of view including a lymph node of a second subject different from the first subject; and cropping a portion of the second fluorescence image as the template image.
In some aspects, the cropped portion of the second fluorescence image depicts a center portion of the lymph node of the second subject.
In some aspects, the template image is generated based on a predefined intensity distribution pattern.
In some aspects, the template image is generated based on a plurality of candidate template images.
In some aspects, the plurality of candidate template images corresponds to a plurality of subjects.
In some aspects, comparing the template image with the fluorescence image comprises: comparing the template image with a plurality of patches of the fluorescence image; and generating a matrix of similarity values, each similarity value in the matrix indicative of a difference between the template image and a respective patch of the plurality of patches of the fluorescence image.
In some aspects, each similarity value in the matrix of similarity values is calculated based on a pixel-wise comparison between the template image and the respective patch of the plurality of patches of the fluorescence image.
In some aspects, the plurality of patches of the fluorescence image is a first plurality of patches of the fluorescence image and the method further comprises: resizing the fluorescence image; and comparing the template image with a second plurality of patches of the resized fluorescence image.
In some aspects, identifying the at least one portion of the fluorescence image corresponding to the at least one lymph node comprises: comparing each similarity value of the one or more similarity values with a predefined threshold.
In some aspects, the method further comprises performing contrast enhancement on the identified at least one portion of the fluorescence image to obtain an enhanced version of the identified at least one portion of the fluorescence image.
In some aspects, the method further comprises displaying the enhanced version of the identified at least one portion of the fluorescence image according to a color scheme.
In some aspects, the contrast enhancement comprises histogram equalization.
In some aspects, the template image is associated with a metastatic node, and the method further comprises determining whether the at least one lymph node is metastatic based on the one or more similarity values.
In some aspects, the method further comprises displaying a visual indication of the metastatic determination for each lymph node of the at least one lymph node.
In some aspects, the template image is generated based on a plurality of time-intensity curves.
In some aspects, the fluorescence image is generated based on a time series of signal intensity data.
An exemplary method for identifying at least one lymph node of a subject comprises: obtaining a fluorescence image of a field of view including the at least one lymph node of the subject; obtaining a template image; and identifying at least one portion of the fluorescence image that corresponds to a location of the at least one lymph node by inputting the template image and the fluorescence image into one or more trained neural networks.
In some aspects, the method further comprises displaying the identified at least one portion of the fluorescence image.
In some aspects, displaying the identified at least one portion of the fluorescence image comprises displaying the fluorescence image and at least one bounding box indicative of the identified at least one portion of the fluorescence image.
In some aspects, the fluorescence image is displayed as part of an intraoperative video stream during a surgical procedure.
In some aspects, the surgical procedure comprises a sentinel lymph node mapping procedure.
In some aspects, the fluorescence image is a first fluorescence image and the subject is a first subject, and the method further comprises: obtaining a second fluorescence image of a second field of view including a lymph node of a second subject different from the first subject; and cropping a portion of the second fluorescence image as the template image.
In some aspects, the cropped portion of the second fluorescence image depicts a center portion of the lymph node of the second subject.
In some aspects, the template image is generated based on a predefined intensity distribution pattern.
In some aspects, the template image is generated based on a plurality of candidate template images.
In some aspects, the plurality of candidate template images corresponds to a plurality of subjects.
In some aspects, the one or more trained neural networks comprise a Siamese network backbone.
In some aspects, the one or more trained neural networks comprise a cross-correlation module connected with the Siamese network backbone.
In some aspects, the one or more trained neural networks comprise a template localization subnetwork connected with the cross-correlation module.
In some aspects, the method further comprises performing contrast enhancement on the identified at least one portion of the fluorescence image to obtain an enhanced version of the identified at least one portion of the fluorescence image.
In some aspects, the method further comprises displaying the enhanced version of the identified at least one portion of the fluorescence image according to a color scheme.
In some aspects, the contrast enhancement comprises histogram equalization.
In some aspects, the one or more trained neural networks are configured to output a classification confidence value for each location of a plurality of locations in the fluorescence image and a template bounding box at the respective location.
In some aspects, the template image is associated with a metastatic node, and the method further comprises determining whether the at least one lymph node is metastatic based on an output of the one or more trained neural networks.
In some aspects, the method further comprises displaying a visual indication of the metastatic determination for each lymph node of the at least one lymph node.
In some aspects, the template image is generated based on a plurality of time-intensity curves.
In some aspects, the fluorescence image comprises a time series of signal intensity data.
An exemplary system for identifying at least one lymph node of a subject comprises: one or more processors; one or more memories; and one or more programs. The one or more programs are stored in the one or more memories and configured to be executed by the one or more processors. The one or more programs include instructions for: obtaining a fluorescence image of a field of view including the at least one lymph node of the subject; obtaining a template image; comparing the template image with the fluorescence image to obtain one or more similarity values; and identifying at least one portion of the fluorescence image that corresponds to a location of the at least one lymph node based on the one or more similarity values.
In some aspects, the one or more programs further include instructions for: displaying the identified at least one portion of the fluorescence image.
In some aspects, displaying the identified at least one portion of the fluorescence image comprises: displaying the fluorescence image and at least one bounding box indicative of the identified at least one portion of the fluorescence image.
In some aspects, the fluorescence image is displayed as part of an intraoperative video stream during a surgical procedure.
In some aspects, the surgical procedure comprises a sentinel lymph node mapping procedure.
In some aspects, the fluorescence image is a first fluorescence image, the field of view is a first field of view, the subject is a first subject, and the one or more programs further include instructions for: obtaining a second fluorescence image of a second field of view including a lymph node of a second subject different from the first subject; and cropping a portion of the second fluorescence image as the template image.
In some aspects, the cropped portion of the second fluorescence image depicts a center portion of the lymph node of the second subject.
In some aspects, the template image is generated based on a predefined intensity distribution pattern.
In some aspects, the template image is generated based on a plurality of candidate template images.
In some aspects, the plurality of candidate template images corresponds to a plurality of subjects.
In some aspects, comparing the template image with the fluorescence image comprises: comparing the template image with a plurality of patches of the fluorescence image; and generating a matrix of similarity values, each similarity value in the matrix indicative of a difference between the template image and a respective patch of the plurality of patches of the fluorescence image.
In some aspects, each similarity value in the matrix of similarity values is calculated based on a pixel-wise comparison between the template image and the respective patch of the plurality of patches of the fluorescence image.
In some aspects, the plurality of patches of the fluorescence image is a first plurality of patches of the fluorescence image, and the one or more programs further include instructions for: resizing the fluorescence image; and comparing the template image with a second plurality of patches of the resized fluorescence image.
In some aspects, identifying the at least one portion of the fluorescence image corresponding to the at least one lymph node comprises: comparing each similarity value of the one or more similarity values with a predefined threshold.
In some aspects, the one or more programs further include instructions for: performing contrast enhancement on the identified at least one portion of the fluorescence image to obtain an enhanced version of the identified at least one portion of the fluorescence image.
In some aspects, the one or more programs further include instructions for: displaying the enhanced version of the identified at least one portion of the fluorescence image according to a color scheme.
In some aspects, the contrast enhancement comprises histogram equalization.
In some aspects, the template image is associated with a metastatic node, and the one or more programs further include instructions for: determining whether the at least one lymph node is metastatic based on the one or more similarity values.
In some aspects, the one or more programs further include instructions for: displaying a visual indication of the metastatic determination for each lymph node of the at least one lymph node.
In some aspects, the template image is generated based on a plurality of time-intensity curves.
In some aspects, the fluorescence image is generated based on a time series of signal intensity data.
An exemplary system for identifying at least one lymph node of a subject comprises: one or more processors; one or more memories; and one or more programs. The one or more programs are stored in the one or more memories and configured to be executed by the one or more processors. The one or more programs include instructions for: obtaining a fluorescence image of a field of view including the at least one lymph node of the subject; obtaining a template image; and identifying at least one portion of the fluorescence image that corresponds to a location of the at least one lymph node by inputting the template image and the fluorescence image into one or more trained neural networks.
In some aspects, the one or more programs further include instructions for: displaying the identified at least one portion of the fluorescence image.
In some aspects, displaying the identified at least one portion of the fluorescence image comprises: displaying the fluorescence image and at least one bounding box indicative of the identified at least one portion of the fluorescence image.
In some aspects, the fluorescence image is displayed as part of an intraoperative video stream during a surgical procedure.
In some aspects, the surgical procedure comprises a sentinel lymph node mapping procedure.
In some aspects, the fluorescence image is a first fluorescence image, and the subject is a first subject, and the one or more programs further include instructions for: obtaining a second fluorescence image of a second field of view including a lymph node of a second subject different from the first subject; and cropping a portion of the second fluorescence image as the template image.
In some aspects, the cropped portion of the second fluorescence image depicts a center portion of the lymph node of the second subject.
In some aspects, the template image is generated based on a predefined intensity distribution pattern.
In some aspects, the template image is generated based on a plurality of candidate template images.
In some aspects, the plurality of candidate template images corresponds to a plurality of subjects.
In some aspects, the one or more trained neural networks comprise a Siamese network backbone.
In some aspects, the one or more trained neural networks comprise a cross-correlation module connected with the Siamese network backbone.
In some aspects, the one or more trained neural networks comprise a template localization subnetwork connected with the cross-correlation module.
In some aspects, the one or more programs further include instructions for: performing contrast enhancement on the identified at least one portion of the fluorescence image to obtain an enhanced version of the identified at least one portion of the fluorescence image.
In some aspects, the one or more programs further include instructions for: displaying the enhanced version of the identified at least one portion of the fluorescence image according to a color scheme.
In some aspects, the contrast enhancement comprises histogram equalization.
In some aspects any one of claims, the one or more trained neural networks are configured to output a classification confidence value for each location of a plurality of locations in the fluorescence image and a template bounding box at the respective location.
In some aspects, the template image is associated with a metastatic node, and the one or more programs further include instructions for: determining whether the at least one lymph node is metastatic based on an output of the one or more trained neural networks.
In some aspects, the one or more programs further include instructions for: displaying a visual indication of the metastatic determination for each lymph node of the at least one lymph node.
In some aspects, template image is generated based on a plurality of time-intensity curves.
In some aspects, the fluorescence image comprises a time series of signal intensity data.
Also disclosed herein is a kit for imaging tissue in a surgical site, the kit comprising a fluorescence imaging agent and any of the system described herein.
Also disclosed herein is use of the kit for lymphatic imaging.
Also disclosed herein is a fluorescence imaging agent for use with any of the surgical systems described herein for imaging tissue in a surgical site.
Optionally, imaging tissue in the surgical site comprises imaging lymphatic tissue.
Also disclosed herein is use of any of the system described herein for lymphatic imaging.
In some aspects, the method further comprises administering a fluorescence agent to the subject.
An exemplary non-transitory computer-readable storage medium stores one or more programs. The one or more programs comprise instructions, which when executed by one or more processors of an electronic device, cause the electronic device to perform any of methods described herein.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
The invention will now be described, by way of example only, with reference to the accompanying drawings, in which:
Reference will now be made in detail to implementations and various aspects and variations of systems and methods described herein. Although several exemplary variations of the systems and methods are described herein, other variations of the systems and methods may include aspects of the systems and methods described herein combined in any suitable manner having combinations of all or some of the aspects described. Examples will now be described more fully hereinafter with reference to the accompanying drawings; however, they may be embodied in different forms and should not be construed as limited to the examples set forth herein. Rather, these examples are provided so that this disclosure will be thorough and complete, and will fully convey exemplary implementations to those skilled in the art.
Disclosed herein are exemplary devices, apparatuses, systems, methods, and non-transitory storage media for automatically identifying one or more lymph nodes of a subject based on fluorescence images such as near-infrared (NIR) images. An exemplary system can perform template matching techniques, machine learning techniques, or any combination thereof. The systems, devices, and methods may be used in association with surgical procedures. Imaging and analysis may be performed pre-operatively, intra-operatively, post-operatively, and during diagnostic imaging sessions and procedures. For example, the system can automatically identify lymph nodes in images acquired in the course of a SLN mapping procedure (such as ICG fluorescence or scintigraphy) and enhance the visualization to allow physicians and surgeons to quickly and efficiently map the relevant sentinel nodes. For example, the system can automatically locate areas containing sentinel lymph nodes in NIR image frames, and these areas can be further visually enhanced to indicate both the shape and the depth of the located nodes.
The system can obtain a fluorescence image of a field of view including the at least one lymph node of the subject and a template image. For a given imaging modality (i.e., ICG fluorescence), certain spatial intensity distribution patterns are unique to sentinel lymph nodes and are unlikely to be produced by other types of tissue, noise, or background. As described herein, in the context of fluorescence imaging, an area of fluorescence with a mountain-like intensity distribution pattern is highly likely to correspond to a sentinel lymph node. Accordingly, the template image can be generated by excising or cropping a relevant image patch from a fluorescence image where the location of the tissue of interest (e.g., lymph node) is known and/or be artificially modelled based on an identified signature pattern of a tissue of interest (e.g., lymph node).
The system can execute a template matching algorithm to compare the fluorescence image and the template image. Based on the comparison, the system can obtain a matrix of similarity values. The system can then compare the matrix with a predefined threshold to identify the most probable locations of the lymph node(s). The locations within the fluorescence image with similarity scores over the predefined threshold are marked as the areas containing lymph node(s). To account for differences in scale between the input image and the template, the system can resize the input image in successive runs of the template matching process, as described herein.
Additionally or alternatively, the system can execute one or more machine learning algorithms to identify lymph nodes. For example, the system can identify at least one portion of the fluorescence image that corresponds to a location of the at least one lymph node by inputting the template image and the fluorescence image into one or more trained neural networks, as described herein.
After the lymph node(s) are identified, the system can locally enhance the region(s) corresponding to the lymph node(s) by, for example, contrast enhancement (e.g., histogram equalization). The system can further display the region(s) according to a color scheme to convey the depth of the node. If multiple fluorescence images are obtained from a video, the timing component can also be incorporated into the node visualization. For example, the nodes that are identified earlier in the course of the imaging process can be shown differently from the ones that are identified later, because the nodes that absorb the dye sooner are more likely to be the primary sentinel nodes.
Examples of the present disclosure provide numerous technical advantages. As discussed herein, the ICG fluorescence technique for the detection of SLN has limited efficacy in patients with a high BMI as lymph drainage channels and sentinel nodes cannot be adequately visualized due to high degree of signal absorption. Further, the lymph nodes in certain locations (e.g., armpit) are surrounded by layers of fat and are thus hard to visualize. As a result, a surgeon may need to dig through the tissue until a lymph node is visible to the naked eye. Another challenge with the ICG fluorescence technique is that it may be hard to tell from the fluorescence image whether a fluorescence signal is from a lymph node or another source in the body. The identification is done based on a surgeon's knowledge and is thus error-prone. Techniques described herein can accurately identify lymph nodes based on a distinct pattern even if the fluorescence signal is weak and/or invisible to manual identification. Further, the techniques described herein can also determine the likelihood that the node is metastatic or benign. Further still, the techniques described herein can also be used to identify any tissue/structure of interest exhibiting a distinct visual and/or temporal pattern, as described herein.
In the following description, it is to be understood that the singular forms “a,” “an,” and “the” used in the following description are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is also to be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It is further to be understood that the terms “includes, “including,” “comprises,” and/or “comprising,” when used herein, specify the presence of stated features, integers, steps, operations, elements, components, and/or units but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, units, and/or groups thereof.
Certain aspects of the present disclosure include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present disclosure could be embodied in software, firmware, or hardware and, when embodied in software, could be downloaded to reside on and be operated from different platforms used by a variety of operating systems. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that, throughout the description, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” “generating” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission, or display devices.
The present disclosure in some examples also relates to a device for performing the operations herein. This device may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, computer readable storage medium, such as, but not limited to, any type of disk, including floppy disks, USB flash drives, external hard drives, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
The methods, devices, and systems described herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein.
A control or switch arrangement 17 may be provided on the camera head 16 for allowing a user to manually control various functions of the system 10, which may include switching from one imaging mode to another, as discussed further below. Voice commands may be input into a microphone 25 mounted on a headset 27 worn by the practitioner and coupled to the voice-control unit 23. A hand-held control device 21, such as a tablet with a touch screen user interface or a PDA, may be coupled to the voice control unit 23 as a further control interface. In the illustrated example, a recorder 31 and a printer 33 are also coupled to the CCU 18. Additional devices, such as an image capture and archiving device, may be included in the system 10 and coupled to the CCU 18. Video image data acquired by the camera head 16 and processed by the CCU 18 is converted to images, which can be displayed on a monitor 20, recorded by recorder 31, and/or used to generate static images, hard copies of which can be produced by the printer 33.
The light source 14 can generate visible illumination light (such as any combination of red, green, and blue light) for generating visible (e.g., white light) images of the target object 1 and, in some examples, can also produce fluorescence excitation illumination light for exciting the fluorescent markers 2 in the target object for generating fluorescence images. In some examples, the light source 14 can produce fluorescence excitation illumination light for exciting autofluorescence in the target object for generating fluorescence images, additionally or alternatively to light for exciting the fluorescent markers. Illumination light is transmitted to and through an optic lens system 22 which focuses light onto a light pipe 24. The light pipe 24 may create a homogeneous light, which is then transmitted to the fiber optic light guide 26. The light guide 26 may include multiple optic fibers and is connected to a light post 28, which is part of the endoscope 12. The endoscope 12 includes an illumination pathway 12′ and an optical channel pathway 12″.
The endoscope 12 may include a notch filter 131 that allows some or all (preferably, at least 80%) of fluorescence emission light (e.g., in a wavelength range of 830 nm to 870 nm) emitted by fluorescence markers 2 in the target object 1 to pass therethrough and that allows some or all (preferably, at least 80%) of visible light (e.g., in the wavelength range of 400 nm to 700 nm), such as visible illumination light reflected by the target object 1, to pass therethrough, but that blocks substantially all of the fluorescence excitation light (e.g., infrared light having a wavelength of 808 nm) that is used to excite fluorescence emission from the fluorescent marker 2 in the target object 1. The notch filter 131 may have an optical density of OD5 or higher. In some examples, the notch filter 131 can be located in the coupler 13.
One or more control components may be integrated into the same integrated circuit in which the sensor 304 is integrated or may be discrete components. The imager 302 may be incorporated into an imaging head, such as camera head 16 of system 10.
One or more control components 306, such as row circuitry and a timing circuit, may be electrically connected to an imaging controller 320, such as camera control unit 18 of system 10. The imaging controller 320 may include one or more processors 322 and memory 324. The imaging controller 320 receives imager row readouts and may control readout timings and other imager operations, including mechanical shutter operation. The imaging controller 320 may generate image frames, such as video frames from the row and/or column readouts from the imager 302. Generated frames may be provided to a display 350 for display to a user, such as a surgeon.
The system 300 in this example includes a light source 330 for illuminating a target scene. The light source 330 is controlled by the imaging controller 320. The imaging controller 320 may determine the type of illumination provided by the light source 330 (e.g., white light, fluorescence excitation light, or both), the intensity of the illumination provided by the light source 330, and or the on/off times of illumination in synchronization with rolling shutter operation. The light source 330 may include a first light generator 332 for generating light in a first wavelength and a second light generator 334 for generating light in a second wavelength. In some examples, the first light generator 332 is a white light generator, which may be comprised of multiple discrete light generation components (e.g., multiple LEDs of different colors), and the second light generator 334 is a fluorescence excitation light generator, such as a laser diode.
The light source 330 includes a controller 336 for controlling light output of the light generators. The controller 336 may be configured to provide pulse width modulation (PWM) of the light generators for modulating intensity of light provided by the light source 330, which can be used to manage over-exposure and under-exposure. In some examples, nominal current and/or voltage of each light generator remains constant, and the light intensity is modulated by switching the light generators (e.g., LEDs) on and off according to a PWM control signal. In some examples, a PWM control signal is provided by the imaging controller 336. This control signal can be a waveform that corresponds to the desired pulse width modulated operation of light generators.
The imaging controller 320 may be configured to determine the illumination intensity required of the light source 330 and may generate a PWM signal that is communicated to the light source 330. In some examples, depending on the amount of light received at the sensor 304 and the integration times, the light source may be pulsed at different rates to alter the intensity of illumination light at the target scene. The imaging controller 320 may determine a required illumination light intensity for a subsequent frame based on an amount of light received at the sensor 304 in a current frame and/or one or more previous frames. In some examples, the imaging controller 320 is capable of controlling pixel intensities via PWM of the light source 330 (to increase/decrease the amount of light at the pixels), via operation of the mechanical shutter 312 (to increase/decrease the amount of light at the pixels), and/or via changes in gain (to increase/decrease sensitivity of the pixels to received light). In some examples, the imaging controller 320 primarily uses PWM of the illumination source for controlling pixel intensities while holding the shutter open (or at least not operating the shutter) and maintaining gain levels. The controller 320 may operate the shutter 312 and/or modify the gain in the event that the light intensity is at a maximum or minimum and further adjustment is needed.
Examples of the present disclosure can identify one or more lymph nodes of a subject in an image based on the unique morphology of a typical lymph node.
At block 802, an exemplary system (e.g., one or more electronic devices) obtains a fluorescence image of a field of view including the at least one lymph node of the subject. The fluorescence image may be captured before, during, and/or after a surgical procedure.
At block 804, the system obtains a template image. The template image is the image to which the fluorescence image is to be compared against to identify the at least one lymph node. The generation of the template image can be based on one or more other images or a predefined intensity distribution pattern (e.g., one that follows a mountain-shaped intensity distribution pattern). The generation of the template image can occur before the step 804.
The generation of the template image can be based on a second fluorescence image different from the fluorescence image in block 802, and the second fluorescence image can be of a second subject different from the subject in block 802. The second subject may be a human or an animal (e.g., porcine). To generate the template image, the system can obtain the second fluorescence image having a field of view including a lymph node of the second subject and crop a portion of the second fluorescence image as the template image. The cropped portion of the second fluorescence image may depict a center portion of the lymph node of the second subject. In some examples, the template image is generated based on a plurality of candidate template images, which may correspond to a plurality of subjects. An exemplary template image is shown in
At block 806, the system compares the template image with the fluorescence image to obtain one or more similarity values. The comparison may be performed using template matching techniques, which can involve searching and finding the location of a template image in a larger image (i.e., the input fluorescence image). For example, the system can slide the template image over the input image and compare the template image and the patch of input image under the template image. Specifically, to perform the comparison, the system can compare the template image with a plurality of patches of the fluorescence image and generating a matrix of similarity values, each similarity value in the matrix indicative of a difference between the template image and a respective patch of the plurality of patches of the fluorescence image. Each similarity value in the matrix of similarity values is calculated based on a pixel-wise comparison between the template image and the respective patch of the plurality of patches of the fluorescence image. For example, a location in the similarity matrix may correspond to a pixel location in the fluorescence image, and the similarity value at the location in the similarity matrix indicates the level of similarity between the template image and an image patch of the fluorescence image in which the pixel is at the center of the image patch.
In one exemplary implementation, the system receives a source image I, which is the fluorescence image obtained in block 802, and a template image T obtained in block 804. For each location of T over I, a Result Metric (R) is calculated using a matching method. It should be appreciated by one of ordinary skill in the art that many matching methods can be used, such as cross-correlation and sum of absolute differences. In the exemplary implementation, the following formula for R can be used.
The higher the R score (i.e., the similarity score), the more similar the examined image patch is to the template image. Optionally, the similarity matrix may be normalized based on normalized cross correlation such that each similarity value in the matrix is in the range of zero to one.
At block 808, the system identifies at least one portion of the fluorescence image that corresponds to a location of the at least one lymph node based on the one or more similarity values. The system can examine the similarity matrix and identify one or more similarity values in the similarity matrix that exceed a predefined threshold. The predefined threshold is indicative of the minimum similarity score to be identified as a lymph node. Based on the one or more similarity values, the system can then identify one or more corresponding portions or patches of the fluorescence image (i.e., the image patch that, when compared against the template image, has produced the similarity score). The predefined threshold may optionally be empirically defined.
To improve the accuracy of the result, the system may use multiple templates and/or different orientations of the same template. Further, to address the issue of different scales between the input fluorescence image and the template, the system may execute the template matching algorithm repeatedly on sub-sampled fluorescence image frames. Specifically, the system can resize the fluorescence image and compare the template image with patches of the resized fluorescence image. In one exemplary implementation, the system can start by comparing the template image against patches (having the same size as the template image) of the original fluorescence image to obtain a first similarity matrix. If there is no similarity score in the first similarity matrix that exceeds the predefined threshold, the system may resize the fluorescence image, and compare the template image against patches (having the same size as the template image) of the resized fluorescence image to obtain a second similarity matrix. The system can iteratively repeat the process until it obtains a similarity score that exceeds the predefined threshold or when the fluorescence image is resized to be the same size as the template image.
After the at least one lymph nodes are identified, the system can display the identified at least one portion of the fluorescence image. For example, the system can indicate the location of the at least one lymph node using a bounding box indicative of the identified at least one portion of the fluorescence image. The identification of the lymph nodes in the fluorescence image can be used to guide a surgical procedure, evaluate a patient, diagnose a disease, recommend a treatment, or any combination thereof. For example, the fluorescence image can be displayed as part of an intraoperative video stream during a surgical procedure, such as a sentinel lymph node mapping procedure.
The system can enhance the visualization of the identified lymph nodes. For example, the system can perform contrast enhancement on the identified at least one portion of the fluorescence image to obtain an enhanced version of the identified at least one portion of the fluorescence image. The contrast enhancement may include localized histogram equalization. To perform localized histogram equalization, the system can apply a histogram equalization algorithm to each region of the image where the similarity values exceed the pre-defined threshold. However, it should be appreciated that other visualization enhancement techniques may be used.
After enhancing the contrast for each identified lymph node, the system may apply a color scheme to further improve the visualization. To visually convey the impression of depth, the system may use the measured average fluorescence intensity of the identified region containing the lymph node as a proxy and apply a color to the region depending on its average intensity value. Specifically, different color schemes can be associated with the different ranges of the average intensity (e.g., red for intensity values between 0-50, orange for 51-101, yellow for 102-152, green for 153 and above). In other words, the system may visualize how dim the original fluorescence signal was, which may be correlated with the depth of the node under the tissue. Accordingly, displaying the enhanced version according to a color scheme may inform the user how deep the lymph node is under the tissue.
Optionally, the color of the lymph node may change dynamically as the lymph node becomes more exposed (and thus its average fluorescence intensity increases) in the process of the surgery. For example, the system may first apply a red color when the original fluorescence signal from the lymph node is very dim. As the surgeon removes the tissue covering the lymph node, the fluorescence signal becomes brighter and the system can switch to a different color (e.g., orange). Finally, when the lymph node is fully exposed, it can be displayed as green.
It should be appreciated that the described visualization technique may not be based on actual depth information. Rather, it may operate under the assumption that the signal intensity is inversely proportional to the node's distance under the tissue surface. There may be other factors that can affect the measured fluorescence intensity, such as ICG dosage. To mitigate this effect, the system may apply normalization factors to the average intensity that take into account, for example, the injected dye dose and the patient's BMI.
At block 1302, an exemplary system (e.g., one or more electronic devices) obtains a fluorescence image of a field of view including the at least one lymph node of the subject. The fluorescence image may be captured before, during, and/or after a surgical procedure. At block 1304, the system obtains a template image. The template image can be generated using the techniques described herein. As described herein, multiple templates and/or different orientations of the same template may be used.
At block 1306, the system identifies at least one portion of the fluorescence image that corresponds to a location of the at least one lymph node by inputting the template image and the fluorescence image into one or more trained neural networks.
The one or more trained neural networks may comprise an object detection neural network, which can be trained using labeled training images in which lymph nodes are annotated. The annotation may be performed manually or via supervised or self-supervised learning techniques. Optionally, the one or more trained neural networks may comprise deep convolutional neural networks (DNNs) such as object tracking DNNs. The one or more neural networks may comprise a Siamese network backbone, a cross-correlation module connected with the Siamese network backbone, and/or a template localization subnetwork connected with the cross-correlation module. The one or more trained neural networks may be configured to output a classification confidence value for each location of a plurality of locations in the fluorescence image and a template bounding box at the respective location. Additional details of exemplary neural networks are provided below with reference to
After the at least one lymph nodes are identified, the system can display the identified at least one portion of the fluorescence image and/or enhance the visualization using any of the techniques described herein.
Optionally, in addition to identifying the location of a lymph node using any of the techniques described herein (e.g.,
Optionally, the techniques described herein can be used to identify tissue/structure of interest exhibiting a distinct temporal pattern. Certain tissue types (e.g., a specific tumor, a specific lymph node) exhibit distinct temporal patterns of fluorescence dye absorption. For example, a specific type of tumor may absorb the fluorescent dye much faster than healthy tissues. Thus, the system can generate the template image based on a plurality of intensity-over-time curves. Specifically, the system can obtain a time-series sequence of fluorescence images depicting a known tissue of interest over time (e.g., as the fluorescence dye is accumulating, staying, and/or leaving the tissue). The system can crop out the region in each image where the tissue is depicted to obtain a time-series sequence of image patches. Each image location in the time-series sequence is associated with an intensity-over-time curve comprising pixels in the same location across the time-series sequence of images. The template image may be generated based on the plurality of intensity-over-time curves. Specifically, the template image can be generated as a visual map in which the value of each pixel corresponds to a parameter derived from the corresponding intensity-over-time curve for that pixel. The parameter may comprise an ingress rate (i.e., the rate or the slope of the curve at which the intensity of the fluorescence signal is growing), an egress rate (i.e., the rate or the slope of the curve at which the intensity of the fluorescence signal is decreasing), a duration of the stable phase (e.g., the length of the plateau of the curve), or any combination thereof.
After the template image is generated, the system can compare the template image against a given time-series sequence of fluorescence images to detect the tissue of interest. Specifically, the system can calculate an input visual map based on the given time-series sequence of fluorescence images (e.g., by calculating an input visual map of ingress rates) and use the template-matching techniques described herein to compare the template image with the input visual map to identify an area exhibiting a similar visual/temporal pattern as the location of the tissue of interest.
Optionally, the techniques described herein can be used to identify other tissue or anatomical structures of interest exhibiting a distinct visual pattern, such as ureters. Existing techniques for ureter visualization during minimally invasive surgeries include injecting a fluorescent dye (e.g., Methylene Blue (MB) or ICG) and using an NIR camera to visualize the presence of the dye in the ureters. However, there are unique constraints associated with each of the dyes. MB is excreted through the kidneys and is concentrated in the urine. Due to the peristaltic nature of urine movement through the ureters, the latter are only visible periodically when the urine is passing through the ureters. ICG, on the other hand, has a unique property of binding to and staining the proteins of the ureteral epithelium for the entire procedure. However, it requires a cystoscopy-guided ICG instillation to the ureters (vs. a much faster and safer intravenous injection of MB).
When ICG is administered intravenously, however, it can be detected in blood vessels and is often used for intra-operative perfusion assessment. Accordingly, ureters can be identified due to the unique vasculature network surrounding them.
The foregoing description, for the purpose of explanation, has been described with reference to specific examples or aspects. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. For the purpose of clarity and a concise description, features are described herein as part of the same or separate variations; however, it will be appreciated that the scope of the disclosure includes variations having combinations of all or some of the features described. Many modifications and variations are possible in view of the above teachings. The variations were chosen and described in order to best explain the principles of the techniques and their practical applications. Others skilled in the art are thereby enabled to best utilize the techniques and various variations with various modifications as are suited to the particular use contemplated.
Although the disclosure and examples have been fully described with reference to the accompanying figures, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the disclosure and examples as defined by the claims. Finally, the entire disclosure of the patents and publications referred to in this application are hereby incorporated herein by reference.
This application claims the benefit of U.S. Provisional Application No. 63/515,621 filed on Jul. 26, 2023, the entire content of which is incorporated herein by reference for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 63515621 | Jul 2023 | US |
