Patent Application
20130274596
1. Field of the Invention
The present embodiments relate generally to apparatuses and methods for tracking and control in surgery and interventional medical procedures.
2. Description of the Related Art
There is currently no technology for robust image-guidance in automated surgery. What is available in the market as so called “robotic surgery” is truly just robot-assisted surgery because the robot only follows direct commands of the surgeon with very little intelligence or autonomy. Some research groups have looked into closing the loop of control for surgical robots with existing sensors, however special conditions and considerations applied to operations in-vivo, make it extremely difficult to achieve such goals.
The present embodiments address at least this problem by introducing a robust tracking technique which requires minimal changes to the current robot-assisted surgical workflow and closing the loop with an effector function.
A more complete appreciation of the embodiments described herein, and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein
According to one embodiment of the present disclosure there is described a system for tracking and control in medical procedures. The system includes a device configured to deploy fluorescent material on at least one of an organ under surgery and a surgical tool, a visual light source, a fluorescent light source corresponding to an excitation wavelength of the fluorescent material, an image acquisition and control element configured to control the visual light source and the fluorescent light source, and configured to capture and digitize at least one of resulting visual images and fluorescent images, and an image-based tracking module configured to apply image processing to the visual and fluorescent images, the image processing detecting fluorescent markers on at least one of the organ and the surgical tool.
According to another embodiment of the system, there is further included in the system a surgical robot, and a visual servoing control module configured to receive tracking information from the image-based tracking module and to control the surgical robot, based on the tracking information, to perform a surgical operation.
According to another embodiment of the system, there is further included in the system a surgical robot, and a visual servoing control module configured to receive tracking information from the image-based tracking module and to control the surgical robot, based on the tracking information, to perform a surgical operation.
According to another embodiment of the system, there is further included in the system a manual control module configured to enable manual control of the surgical robot in place of control by the visual servoing control module.
According to another embodiment of the system, the visual servoing control module is further configured to receive manual input and to control the surgical robot, based on the manual input, to perform a surgical operation.
According to another embodiment of the system, there is further included in the system a surgical robot, and a manual control module configured to receive manual input and execute master-slave control of the surgical robot.
According to another embodiment of the system, there is further included in the system a display configured to display at least one of the visual images and the fluorescent images.
According to another embodiment of the system, the image-based tracking module further identifies the organ or the surgical tool based on the detected fluorescent markers.
According to another embodiment of the system, the image acquisition and control element further includes a dynamic tunable filter configured to alternatively pass visual light and light emitted by the fluorescent material, and a charged coupled device configured to capture at least one of visual images and fluorescent images.
According to another embodiment of the system, the display is stereoscopic or monoscopic.
According to another embodiment of the system, the image acquisition and control element generates stereoscopic or monoscopic images.
According to another embodiment of the system, the stereoscopic display is further configured to display visual images and a color coded overlay of fluorescent images.
According to another embodiment of the system, the stereoscopic display is further configured to display an augmented reality image by overlaying target points detected by the image-based tracking module.
According to another embodiment of the system, the system is configured to provide at least one of visual, audio, and haptic feedback to a system operator, based on information provided by the image-based tracking module.
According to another embodiment of the system, the system is configured to operate in each of a manual mode, a semi-autonomous mode, and an autonomous mode.
According to another embodiment of the system, the image-based tracking module identifies virtual boundaries based on the detected fluorescent markers to designate critical structures.
According to another embodiment of the system, the system further includes a detection device configured to determine whether a surgical tool has passed a boundary and to provide constraints on motion or provide alarms when the boundary has been crossed in order to protect the critical structures.
According to another embodiment of the system, the fluorescent light source is a near-infrared (NIR) light source.
According to another embodiment of the system, the image acquisition and control element includes two charge coupled devices (CCDs), one assigned to a visual spectrum and one assigned to a NIR spectrum.
According to another embodiment of the system, light generated by the visual light source and the fluorescent light source is split by either a beam-splitting or a dichromatic prism.
According to another embodiment of the system, light generated by the visual light source and the fluorescent light source are provided separate light paths to the two CCDs.
According to one embodiment of the present disclosure there is described a method for performing a medical procedure. The method includes the steps of deploying fluorescent material on at least one of an organ under surgery and a surgical tool, illuminating the organ, the surgical tool, or both, with a visual light source and a fluorescent light source, the fluorescent light source corresponding to an excitation wavelength of the fluorescent material, capturing and digitizing images resulting from the illumination by the visual light source and the fluorescent light source, and applying image processing to the digitized images, the image processing detecting fluorescent markers on at least one of the organ and the surgical tool.
According to another embodiment of the method, there is further included in the method the step of generating tracking information by tracking the organ, the surgical tool, or both based on the detected fluorescent markers.
According to another embodiment of the method, there is further included in the method the step of controlling a surgical robot, based on the tracking information, to perform a surgical operation.
According to another embodiment of the method, there is further included in the method the steps of receiving manual input, and controlling the surgical robot, based on the manual input, to perform the surgical operation.
According to another embodiment of the method, there is further included in the method the steps of receiving manual input, and executing master-slave control of a surgical robot based on the on manual input.
According to another embodiment of the method, there is further included in the method the step of providing a stereoscopic or monoscopic display of the digitized images.
According to another embodiment of the method, the step of capturing and digitizing images further includes the step of generating stereoscopic or monoscopic images.
According to another embodiment of the method, there is further included in the method the step of displaying visual images and a color coded overlay of fluorescent images.
According to another embodiment of the method, there is further included in the method the step of displaying an augmented reality image by overlaying target points detected by the image-based tracking module.
According to another embodiment of the method, there is further included in the method the step of providing at least one of visual, audio, or haptic feedback to a system operator, based on the tracking information.
According to another embodiment of the method, there is further included in the method the step of identifying the organ or the surgical tool based on the detected fluorescent markers.
According to another embodiment of the method, there is further included in the method the step of performing a surgical procedure based on the detected fluorescent markers.
According to another embodiment of the method, there is further included in the method the step of designating critical structures by identifying virtual boundaries based on the detected fluorescent markers.
According to another embodiment of the method, there is further included in the method the step of determining whether a surgical tool has passed a boundary and providing constraints on motion or providing alarms when the boundary has been crossed in order to protect the critical structures.
According to one embodiment of the present disclosure there is described a system for tracking and control in medical procedures. The system includes means for deploying fluorescent material on at least one of an organ under surgery and a surgical tool, a visual light source, a fluorescent light source corresponding to an excitation wavelength of the fluorescent material, means for controlling the visual light source and the fluorescent light source, means for capturing and digitizing at least one of resulting visual images and fluorescent images, and means for applying image processing to the visual and fluorescent images, the image processing detecting fluorescent markers on at least one of the organ and the surgical tool.
The disclosed embodiments may be applied in the field automated anastomosis where tubular structures (vessels, bile ducts, urinary tract, etc.) are connected and sealed. Anastomosis is one of the four major steps in every surgery: 1) Access through incision; 2) Exposure and dissection; 3) Resection and removal of pathology; and 4) Reconstruction and closure (Anastomosis). Anastomosis is currently performed by suturing or applying clips or glue to the anastomosis site. The anastomosis procedure may be performed manually or by using robots through master-slave control, both techniques are very time consuming and cumbersome. The present embodiments make it possible for the surgeon to mark the anastomosis site by applying fluorescent markers (in terms of miniature clips, spray, paint, tapes, etc.) which can be detected and tracked using the dual-spectrum imaging technology. In addition, a robotic system can be controlled through visual servoing using this tracking information, in order to apply sutures/clips/glue or weld at specified positions.
The present embodiments have several other applications including but not limited to:
Automation of other steps of surgery: Automating all parts of surgery including exposure and dissection, and resection and removal of pathology.
Automated tumor resection/ablation: a tumor will be painted using a fluorescent dye and the robotic system will be guided/controlled to resect or ablate the tumor. This can be applied in applications such as partial nephrectomy, hepatectomy, etc.
Assisting in manual or master-slave robotic surgery: The technology can be used as visual guide to surgeons for manual surgeries and master-slave controlled robotic surgery. Critical structures can be marked by the surgeons. The tools and structures are then clearly visible to the surgeon throughout the procedure.
Pre-excisional or incisional biopsy localization of sub-surface or deep nodules or lesions in viscera.
Reference marker for accurate re-approximation, orientation of tissue or precise reconstruction of surgical area during open surgery.
Positional marker for motion tracking/memory during endoscopic procedure.
Some variants of embodiments of the technology are listed below:
The technology can be used with multiple dyes with excitation/emission at different wavelengths. This can be applied to have inherently different markers for tracking multiple objects. In one embodiment, fluorescent dyes A and B are used to mark the two sides of a tubular structure prior to automated anastomosis.
The markers can be applied to the targets both internally and externally. The fluorescent dye can be attached to the target by clips, staples, glue or can be applied by painting or spraying. The dye can also be injected to the tissue to mark specific points or can be injected through blood. The dye can be selected in order to bind with specific types of cells to mark specific structures (such as tumors).
Providing “no-fly zones” or “virtual fixtures” to prevent the surgical tools from approaching critical structures: In this embodiment, the surgeon marks the critical structures prior to the task and the marked borders will be tracked using the dual-mode imaging technology. The coordinates will be used to force constraints on the motion of the surgical tools during the automated or semi-automated task. It can also be used to provide alarms (visual/audio or haptic) in manual tasks.
The imaging system can be monoscopic and provide two-dimensional location of the tracked points which can potentially be used for image-based visual servoing. The imaging system can be stereoscopic and provide three-dimensional location of the tracked structures and therefore be used for image-based or position-based visual servoing.
The embodiments of the technology can be applied for automated or semi-automated applications. It can also provide guidance for manual operations through visual, audio or haptic feedback.
Automation of a surgical procedure is a very challenging task. The surgical scene is dynamically changing, deformable organs may occlude surgeon's view and variations in illumination make it extremely difficult to robustly track any target and object inside the patient's body. Several attempts have been made to develop image-based tracking algorithms for minimally invasive and/or open surgeries but depend on special conditions and are not robust; therefore cannot be used to control any of the surgical tools or to automate parts of a surgery.
The present embodiments address these limitations by using a dual-spectrum imaging device which can image in the visual spectrum as well as in near-infrared (NIR) spectrum. The surgeon places fluorescent markers on the locations which should be tracked (e.g., tools and tissue); The excitation light generated by the imaging device causes the fluorophores to emit NIR light which will be detected by the imaging device. As a result, the system has a high signal to noise ratio (SNR) because of (a) limited autofluorescence of the tissue compared to the fluorescent dyes, and (b) lack of other NIR sources in the patient's body. This high SNR makes any tracking algorithm more robust and reliable. NIR light has a good penetration in the tissue as opposed to the visible light; this makes it possible to track an object even if it is occluded by another organ, flipped over, covered by blood, etc. A combination of visual and NIR images can be used to make image-based tracking algorithms even more robust.
One embodiment describes a system for automation of surgical tasks. It is based on deploying fluorescent markers on the organ under surgery and/or on the surgical tool, tracking the markers in real-time and controlling the surgical tool via visually servoing.
In one embodiment, multiple different markers are used to help track multiple structures, organs, and tools. Using different markers reduces the error rate for tracking, since the number of similar markers is reduced. Differentiation of markers can be achieved by having different size or volume and/or shape of the markers and or using dyes with excitation/emission at different wavelengths. In one embodiment, markers with 3 micro liters volume and markers with 6 micro liters volume are used to mark the two sides of a tubular structure respectively prior to automated anastomosis. In another embodiment, a fluorescent dye emitting at 790 nm corresponds to the no-fly zone while a different wavelength 830 nm corresponds to an edge of a structure.
In one embodiment, each structure (i.e. organ, stream segment) is assigned a structure identification number. Likewise, when the surgeon marks a structure at the anastomoses site, each marker is automatically assigned a unique identification number and is automatically labeled with the structure identification number to which it is attached. As the markers are tracked, the label of each marker is used to determine which structure it belongs and its overlay color. This tracking may be performed using tables or databases implemented by a computer processor and corresponding software instructions.
The image-based tracking module 107 also includes a tracking module that performs pre-processing of the NIR image and visual tracking based on the processed image information. In one embodiment, the pre-processing algorithm involves image processing algorithms, such as image smoothing, to mitigate the effect of sensor noise; image histogram equalization to enhance the pixel intensity values, and image segmentation based on pixel intensity values to extract templates for the NIR markers. The visual trackers are initialized first. The initialization of the visual trackers starts by detection and segmentation of the NIR marker. Segmentation is based on applying an adaptive intensity threshold on the enhanced NIR image to obtain a binary template for the NIR markers. A two dimensional (2D) median filter and additional morphology-based binary operators (binary image processing algorithms such as image erosion and dilation) may be applied on the binary template to remove segmentation noise. The binary template may be used as a starting base for visual tracking of NIR markers using visual tracking algorithms. After pre-processing and segmentation, the NIR template is a white blob on a darker background, which represents the rest of the surgical field in the NIR image.
In
In manual mode (
In autonomous mode (
The tracked visual markers are used to guide the motion of the robot. Each visual marker is represented by a representative vector of numbers, which is typically called a visual feature. Examples of visual features are coordinates of the centers of NIR markers extracted from the binary image, and/or their higher-order image moments (such as their area in terms of number of pixels).
Robot motion is performed by transforming the sensor measurements into global Cartesian coordinate form for the robot. In one embodiment, the NIR and tool markers are tracked in the stereo images to compute the 3D coordinates of the marker or tool with respect to the surgical field, as shown in
In particular,
When the motion control feedback loop is closed in the sensor space, the effect of calibration errors is limited. This is desired for supervised autonomy. Vision-based, closed loop feedback, motion control of robots is called visual servoing. There are two main approaches to visual servoing based on control architecture: position-based visual servoing (PBVS) and image-based visual servoing (IBVS). Both approaches are viable options. In PBVS, the position of the robotic tool is estimated and the error is estimated based on the estimated position and the goal tool position. In IBVS, the image features are used directly to compute the task error in the image space, such that when the robotic tool is at the goal position the task error is zero. Both control approaches generate motions that drive the error to zero.
The NIR based robot motion control is a core technology which has not been developed in the past. Previous methods and apparatuses for NIR based imaging (without robot control, Frangioni 2012, U.S. Pat. No. 8,229,548 B2) and NIR based display (Mohr and Mohr, US 2011/0082369) fail to consider robot motion control or any control whatsoever. With a stereo imaging system consisting of two NIR cameras with appropriate filters, a properly excited NIR agent can be seen in both stereo images. Image processing and visual tracking algorithms, such as the algorithms described above as being implemented by the image-based tracking module 107, are utilized to visually track each NIR marker in the image. The 3D estimate of a marker position is found by triangulation of the NIR marker image as seen in both left 701 and right 703 NIR stereo image pairs. The 3D estimate of the NIR marker can then be re-projected as an overlay in the RGB image 702. The tool position is also found from the stereo image pair. The stereo NIR system can be replaced by a 3D sensing camera capable of NIR observation.
The embodiments described herein are also very useful in non-stereo applications. For example, the system can be implemented for mono camera applications. For manual and master-slave modes (
As is shown in
Certain portions or all of the disclosed processing, such as the image processing and visual tracking algorithms, for example, can be implemented using some form of computer microprocessor. As one of ordinary skill in the art would recognize, the computer processor can be implemented as discrete logic gates, as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Complex Programmable Logic Device (CPLD). An FPGA or CPLD implementation may be coded in VHDL, Verilog or any other hardware description language and the code may be stored in an electronic memory directly within the FPGA or CPLD, or as a separate electronic memory. Further, the electronic memory may be non-volatile, such as ROM, EPROM, EEPROM or FLASH memory. The electronic memory may also be volatile, such as static or dynamic RAM, and a processor, such as a microcontroller or microprocessor, may be provided to manage the electronic memory as well as the interaction between the FPGA or CPLD and the electronic memory.
Alternatively, the computer processor may execute a computer program including a set of computer-readable instructions that perform the functions described herein, the program being stored in any of the above-described non-transitory electronic memories and/or a hard disk drive, CD, DVD, FLASH drive or any other known storage media. Further, the computer-readable instructions may be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with a processor, such as a Xenon processor from Intel of America or an Opteron processor from AMD of America and an operating system, such as Microsoft VISTA, UNIX, Solaris, LINUX, Apple, MAC-OSX and other operating systems known to those skilled in the art.
In addition, certain features of the embodiments can be implemented using a computer based system (
The computer 1000 may also include a disk controller coupled to the bus B to control one or more storage devices for storing information and instructions, such as mass storage 1002, and drive device 1006 (e.g., floppy disk drive, read-only compact disc drive, read/write compact disc drive, compact disc jukebox, tape drive, and removable magneto-optical drive). The storage devices may be added to the computer 1000 using an appropriate device interface (e.g., small computer system interface (SCSI), integrated device electronics (IDE), enhanced-IDE (E-IDE), direct memory access (DMA), or ultra-DMA).
The computer 1000 may also include special purpose logic devices (e.g., application specific integrated circuits (ASICs)) or configurable logic devices (e.g., simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)).
The computer 1000 may also include a display controller coupled to the bus B to control a display, such as a cathode ray tube (CRT), for displaying information to a computer user. The computer system includes input devices, such as a keyboard and a pointing device, for interacting with a computer user and providing information to the processor. The pointing device, for example, may be a mouse, a trackball, or a pointing stick for communicating direction information and command selections to the processor and for controlling cursor movement on the display. In addition, a printer may provide printed listings of data stored and/or generated by the computer system.
The computer 1000 performs at least a portion of the processing steps of the invention in response to the CPU 1004 executing one or more sequences of one or more instructions contained in a memory, such as the memory unit 1003. Such instructions may be read into the memory unit from another computer readable medium, such as the mass storage 1002 or a removable media 1001. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in memory unit 1003. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.
As stated above, the computer 1000 includes at least one computer readable medium 1001 or memory for holding instructions programmed according to the teachings of the invention and for containing data structures, tables, records, or other data described herein. Examples of computer readable media are compact discs, hard disks, floppy disks, tape, magneto-optical disks, PROMs (EPROM, EEPROM, flash EPROM), DRAM, SRAM, SDRAM, or any other magnetic medium, compact discs (e.g., CD-ROM), or any other medium from which a computer can read.
Stored on any one or on a combination of computer readable media, the present invention includes software for controlling the main processing unit 1004, for driving a device or devices for implementing the invention, and for enabling the main processing unit 1004 to interact with a human user. Such software may include, but is not limited to, device drivers, operating systems, development tools, and applications software. Such computer readable media further includes the computer program product of the present invention for performing all or a portion (if processing is distributed) of the processing performed in implementing the invention.
The computer code elements on the medium of the present invention may be any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs), Java classes, and complete executable programs. Moreover, parts of the processing of the present invention may be distributed for better performance, reliability, and/or cost.
The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the CPU 1004 for execution. A computer readable medium may take many forms, including but not limited to, non-volatile media, and volatile media. Non-volatile media includes, for example, optical, magnetic disks, and magneto-optical disks, such as the mass storage 1002 or the removable media 1001. Volatile media includes dynamic memory, such as the memory unit 1003.
Various forms of computer readable media may be involved in carrying out one or more sequences of one or more instructions to the CPU 1004 for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. An input coupled to the bus B can receive the data and place the data on the bus B. The bus B carries the data to the memory unit 1003, from which the CPU 1004 retrieves and executes the instructions. The instructions received by the memory unit 1003 may optionally be stored on mass storage 1002 either before or after execution by the CPU 1004.
The computer 1000 also includes a communication interface 1005 coupled to the bus B. The communication interface 1004 provides a two-way data communication coupling to a network that is connected to, for example, a local area network (LAN), or to another communications network such as the Internet. For example, the communication interface 1005 may be a network interface card to attach to any packet switched LAN. As another example, the communication interface 1005 may be an asymmetrical digital subscriber line (ADSL) card, an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of communications line. Wireless links may also be implemented. In any such implementation, the communication interface 1005 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
The network typically provides data communication through one or more networks to other data devices. For example, the network may provide a connection to another computer through a local network (e.g., a LAN) or through equipment operated by a service provider, which provides communication services through a communications network. The local network and the communications network use, for example, electrical, electromagnetic, or optical signals that carry digital data streams, and the associated physical layer (e.g., CAT 5 cable, coaxial cable, optical fiber, etc). Moreover, the network may provide a connection to a mobile device such as a personal digital assistant (PDA) laptop computer, or cellular telephone.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions. As used herein the words “a” and “an” and the like carry the meaning of “one or more.” The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions
This application claims the benefit of priority under 35 U.S.C. §119(e) from U.S. Ser. No. 61/624,665, filed Apr. 16, 2012, the entire contents of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61624665 | Apr 2012 | US |
