The present invention generally relates to a real-time tracking of a surgical tool within an anatomical region of a body based on a preoperative scan image and endoscopic images of the anatomical region. The present invention specifically relates to a computation of an offset transformation matrix between an endoscopic camera and an electromagnetic (“EM”) tracker using the preoperative scan image and one or more endoscopic images of the anatomical region.
EM guided endoscopy has been recognized as a valuable tool for many lung applications. The advantage of this technology over conventional endoscopy is based on a real-time connection to a three-dimensional (“3D”) roadmap of the lung while the interventional procedure is being performed. This connection requires a tracking of a tip of an endoscope in a global coordinate system to thereby associate endoscopic images of the lung with a preoperative scan image of the lung (e.g., a computed tomography image, a magnetic resonance image, an X-ray image, a three-dimensional ultrasound image, etc.). The fused images are displayed to enable the surgeon to visually navigate the endoscope to a surgical site within the lung.
A key requirement of this image integration is an endoscopic calibration involving a determination of a position and an orientation of an EM tracker externally mounted to the endoscope with respect to a coordinate system of an endoscopic camera disposed within a camera channel of the endoscope. The results of this endoscopic calibration take the form of six (6) offset constants: three (3) for rotation and three (3) for translation. The goal of the endoscopic calibration in an interventional endoscopic procedure is to dynamically determine the pose of the endoscopic camera relative to the preoperative scan image based on the EM readings of the attached EM tracker.
Generally speaking, calibration parameters have been obtained in the art by using an EM-tracked endoscope to image an EM-tracked lung phantom of a particular calibration pattern that has known geometric properties. However, a phantom based endoscopic calibration involves a cumbersome engineering procedure. In one known endoscopic calibration, although a desired transformation of the endoscopic calibration is between a camera coordinate system and an EM tracker coordinate system, an array of calibration procedures are in fact needed between an endoscope, the EM tracker externally and rigidly attached to the endoscope, an EM field generator, the calibration phantom and a reference tracker. For example, the needed calibration procedures include a calibration of the EM tracker coordinate system and the reference tracker, a calibration between the calibration phantom and the reference tracker, and a calibration between the endoscopic camera and the calibration phantom to thereby arrive at the destination calibration between the camera coordinate system and the EM tracker coordinate system.
In addition, the data acquisition protocol required in collecting the calibration data is usually from a calibration phantom with a checker-board pattern. This makes the calibration impractical to be an intraoperative calibration procedure of the endoscopic application. However, an intraoperative calibration is preferred under circumstances whereby (1) intrinsic camera and distortion parameters are fixed and determined through a preoperative calibration process and (2) extrinsic camera parameters (e.g., a transformation between the coordinates of the EM tracker and the endoscopic camera) are not fixed and will change across different endoscopic applications. This change may due to the reality that the EM tracker may not be bundled permanently to the tip of the endoscope due to a variety of reasons. For example, the EM tracker may be inserted inside the working channel of the endoscope at the initial phase of the endoscopic application, removed from the working channel after the endoscope reaches the target site within the anatomical region, and replaced with a surgical instrument (e.g., a biopsy needle or forceps) for subsequent interventions.
Moreover, intraoperative calibration procedures as known in the art still utilize a calibration phantom.
The present invention provides an endoscopic calibration approach that quickly and accurately computes the desired extrinsic parameter to thereby achieve the real-time data fusion between a preoperative scan image (e.g., a CT image) of an anatomical region and endoscopic images of the anatomical region. Specifically, the endoscopic calibration method of the present invention excludes any involvement with any phantom. Instead, the endoscopic calibration method of the present invention utilizes both preoperative scan data and endoscopic video data from a patient to perform an image-based registration that yields the transformation from the preoperative scan coordinates to the endoscopic camera coordinates, which may be utilized with other known transformation matrixes to derive the desired calibration transformation matrix.
One form of the present invention is a surgical navigation system employing an endoscope and an imaging unit. The endoscope includes an electromagnetic tracker within a working channel of the endoscope for generating electromagnetic sensing signals indicative of one or more poses of the endoscope within an anatomical region, and an endoscopic camera within an imaging channel of the endoscope for generating endoscopic images of the anatomical region. In operation, the imaging unit executes an intraoperative calibration of the electromagnetic tracker and the endoscopic camera as a function of an image registration between the preoperative scan image of a calibration site within the anatomical region and one or more endoscopic images of the calibration site within the anatomical region.
In a second form of the present invention, the surgical navigation system further employs an electromagnetic tracking unit responsive to the electromagnetic signals to electromagnetically track the endoscope within the anatomical region relative to a global reference, and the intraoperative calibration of the electromagnetic tracker and the endoscopic camera is a function of both the image registration between the preoperative scan image of a calibration site within the anatomical region and one or more endoscopic images of the calibration site within the anatomical region and a function of an electromagnetic registration between the global reference and the preoperative scan image.
A third form of the present invention is a surgical navigation method involving an execution of an intraoperative calibration of the electromagnetic tracker and the endoscopic camera as a function of an image registration between the preoperative scan image of a calibration site within the anatomical region and one or more endoscopic images of the calibration site within the anatomical region, and a display of an image integration of the preoperative scan image of the anatomical region and the endoscopic image(s) of the anatomical region derived from the image registration.
For purposes of the present invention, the term “endoscope” is broadly defined herein as any device having the ability to image from inside a body and the term “endoscopic” is broadly defined herein as a characterization of any image acquired from such device. Examples of an endoscope for purposes of the present invention include, but are not limited to, any type of scope, flexible or rigid (e.g., arthroscope, bronchoscope, choledochoscope, colonoscope, cystoscope, duodenoscope, gastroscope, hysteroscope, laparoscope, laryngoscope, neuroscope, otoscope, push enteroscope, rhino laryngoscope, sigmoidoscope, sinuscope, thorascope, etc.) and any device similar to a scope that is equipped with an image system (e.g., a nested cannula with imaging). The imaging is local, and surface images may be obtained optically with fiber optics, lenses, or miniaturized (e.g. CCD based) imaging systems.
Additionally, the term “generating” and any form thereof as used herein is broadly defined to encompass any technique presently or subsequently known in the art for creating, supplying, furnishing, obtaining, producing, forming, developing, evolving, modifying, transforming, altering or otherwise making available information (e.g., data, text, images, voice and video) for computer processing and memory storage/retrieval purposes, particularly image datasets and video frames, and the term “registration” and any form thereof as used herein is broadly defined to encompass any technique presently or subsequently known in the art for transforming different sets of coordinate data into one coordinate system.
Furthermore, the term “preoperative” as used herein is broadly defined to describe any activity occurring or related to a period or preparations before an intervention of an endoscope within a body during an endoscopic application, and the term “intraoperative” as used herein is broadly defined to describe as any activity occurring, carried out, or encountered in the course of an introduction of an endoscope within a body during an endoscopic application. Examples of an endoscopic application include, but are not limited to, an arthroscopy, a bronchoscopy, a colonscopy, a laparoscopy, and a brain endoscopy.
The foregoing forms and other forms of the present invention as well as various features and advantages of the present invention will become further apparent from the following detailed description of various embodiments of the present invention read in conjunction with the accompanying drawings. The detailed description and drawings are merely illustrative of the present invention rather than limiting, the scope of the present invention being defined by the appended claims and equivalents thereof.
Referring to
More particularly, intrinsic parameters and distortion parameters of endoscopic camera 50 are unchanging and as such, these parameters only require a one-time calibration process (e.g., a preoperative intrinsic calibration as known in the art). Thus, with EM tracker 40 being inserted into a working channel of endoscope 30, the only variable of all camera parameters is the extrinsic parameters, especially an offset transformation matrix TC←E from EM tracker coordinate system 41 to camera coordinate system 51.
In practice, the present invention neither restricts or limits the manner by which registration/calibration technique 60 differentiates the disparity in the 2D projection images between endoscopic images of an anatomical region and a virtual fly-through of image frames of preoperative scan image 10 of the anatomical region.
In one embodiment, registration/calibration technique 60 involves the execution of the following equation [1]:
T
C←E=(TC←T)*(TT←R)*(TR←E) [1]
where TR←E is a transformation matrix as known in the art from EM tracker coordinate system 41 to a global coordinate system 21 of global reference 20 (e.g., a reference tracker or a EM field generator having a fixed location during the endoscopic surgical procedure),
where TT∂R is a transformation matrix as known in the art from global coordinate system 21 of global reference 20 to scan image coordinate system 11 of preoperative scan image 10,
where TC←T is a transformation matrix as taught by the present invention from scan image coordinate system 11 of preoperative scan image 10 to camera coordinate system 51 of endoscopic camera 50, and
where TC←E is the desired rigid transformation from EM tracking coordinate system 41 of EM tracker 40 to camera coordinate system 51 of endoscopic camera 50.
An execution of equation [1] results in an image registration of the endoscopic images and preoperative scan image 10 for display to enable a surgeon to visually navigate the tip of endoscope 30 to a surgical site within the anatomical region.
As shown in
Further, in practice, EM sensor tracking device 73 executes any known method for generating EM tracking data (“EMT”) 74 derived via any known registration of endoscope tracker 40 relative to EM field generator 71 or reference tracking device 72, whichever has a fixed location relative to the anatomical region within the global coordinate system.
The endoscopic navigation system further employs an endoscope imaging unit 80 having an EM reference registration device 81, an endoscopic camera calibration device 82 and an endoscopic image tracking device 83. EM tracker registration device 81 is broadly defined herein as any device structurally configured for executing any known registration of EM tracker 40 to a preoperative scan image of an anatomical region (e.g., preoperative scan image 10 of
Endoscopic camera calibration device 82 is broadly defined herein as any device structurally configured for executing a registration of a preoperative scan image of an anatomical region to endoscopic images of the anatomical region in accordance with an endoscopic camera calibration method of the present invention as will be further explained in connection with the description of
Endoscopic image tracking device 83 is broadly defined herein as any device structurally configured for generating a display of a real-time tracking of endoscope 30 within the preoperative scan image based on the image registration between the endoscopic images and the preoperative scan image achieved by endoscopic camera calibration device 82.
A flowchart 100 representative of an endoscopic surgical method of the present invention as shown in
Referring to
Referring back to
In one embodiment, a flowchart 130 representative of an image registration method of the present invention as shown in
Referring to
A stage S132 of flowchart 130 encompasses an EM reference registration involving a known computation by EM reference registration device 81 (
A stage S133 of flowchart 130 encompasses an image registration involving a computation by camera calibration device 82 of a transformation matrix TC←T as taught by the present invention from scan image coordinate system 122 of preoperative scan image 120 to camera coordinate system 51 of endoscopic camera 50 (
In one embodiment of stage S133, a flowchart 140 representative of a camera calibration method of the present invention as shown in
Referring to
Specifically, stages S142-S144 of flowchart 140 respectively encompass an acquisition of a video frame Vi of endoscopic image of the calibration site, a rendering of an scan frame Ui of an endoluminal image of the calibration site, and an image registration between scan frame Ui of an endoluminal image of the calibration site and the video frame Vi of the calibration site to identify the camera poses in the pre-operative scan space TiT←C. The endoscopic image acquisition of stage S142 involves an EM tracker reading PR<-Ei to obtain a pose of endoscope 30 associated with the endoscopic image acquisition. The endoluminal image acquisition of stage 143 involves a virtual endoscopic flythrough of the preoperative scan image of the anatomical region to thereby obtain a visual match of an endoscopic view of the calibration site as shown in a scan frame Ui of the preoperative scan image with the endoscopic image of the calibration site as shown in video frame Vi. The endoluminal image registration of stage S144 involves a computation 4×4 transformation matrix TiC←T as an inverse of matrix TiT←C hereby the camera viewing pose is expressed as M=[RxTx;0 1], where Rx is the corresponding Euler 3×3 rotation matrix of the 3D translation vector and Tx is the 3D translation vector.
Stages S142-S144 may be executed a single time whereby a stage S145 of flowchart encompasses an execution of equation [1]: TC←E=(TC←T)*(TT←R)*(TR←E) to thereby obtain the transformation matrix TC←E.
Alternatively, stages S142-S144 may be executed as a loop for a set of N image registrations, wherein N≧2. For this loop embodiment, the transformation matrixes TC-T computed during each execution of stage S144 are averaged prior to the endoscopic camera calibration computation of stage S145.
In practice, N=6 may be utilized as a sufficient number of image registrations for an accurate computation of the camera calibration.
Furthermore, in practice, a known motion compensation algorithm (e.g., respiratory gating or four-dimensional modeling) may be utilized to compensate for any respiratory motion that my degrade the computation of the camera calibration.
Referring to back to
Referring to
While various embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that the methods and the system as described herein are illustrative, and various changes and modifications may be made and equivalents may be substituted for elements thereof without departing from the true scope of the present invention. In addition, many modifications may be made to adapt the teachings of the present invention without departing from its central scope. Therefore, it is intended that the present invention not be limited to the particular embodiments disclosed as the best mode contemplated for carrying out the present invention, but that the present invention include all embodiments falling within the scope of the appended claims.
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/IB2012/050024 | 1/3/2012 | WO | 00 | 7/3/2013 |
Number | Date | Country | |
---|---|---|---|
61432298 | Jan 2011 | US |