Patent Application
20080033283
The present invention concerns an apparatus for fusion and navigation of ecographic and volumetric images of a patient which uses a combination of active and passive optical markers for localization of ecographic probes and surgical instruments with reference to the patient.
In the past years technological progress in the field of the acquisition of biomedical images has allowed to obtain more and more detailed information both on the morphology and on the functions of the different anatomical areas of a patient. In this context the fusion of the information obtained with different techniques for the acquisition of images is playing an increasingly important role, both in diagnostics and in programming and execution of surgical operations. The fusion of images coming from different technological approaches, in fact, allows to combine the specific advantages of each technique, while limiting their respective applicative constraints. In particular it seems to be very interesting the combination of systems for the acquisition of volumetric images (CAT, MRI, PET, etc.) which is capable to provide images with very high morphologic and functional details, with 2D or 3D ultrasonography, non-invasive, flexible technique and ideal instrument for support to the surgeon in the stages of planning and, above all, execution of surgical operations. The characteristics of the ecographic analysis allow the surgeon to obtain a verification of his work and, therefore, to organize and to execute an operation by adjusting in real time to the possible variations of the conditions of the subject. The recent introduction of contrast means in echography has further improved both the quality of the ecographic images and the possibility to extract information of functional type from such images. In any case, unfortunately, limits intrinsic to the ecographic technology do not allow to obtain very defined images for all the types of tissues and to localize anomalies with the same accuracy and evidence that other techniques for the acquisition of images offer. In this context a system capable to combine in real time ecographic images with 3D images previously obtained with other image acquisition techniques would be an important advance for the development of diagnostic and surgical techniques which offer a greater effectiveness and a lower impact on the patient, with consequent improvement of the quality of the life and reduction in the cost of the operation.
In order to create a complete system in support of diagnosis, planning and execution of operations by combining information obtained from 3D images acquisition techniques with ultrasonographical techniques it is necessary to solve two important technological problems: 1) the fusion of images acquired in different moments and with different techniques (which inevitably create geometric distortions in the recontructed images); 2) the real time localization of the position of the ecographic probe with reference to the patient in order to be able to obtain the information necessary to the navigation and fusion systems which combine the 3D images obtained with other methods with the 2D ecographic ones so as to obtain a new image having a higher informative content.
In addition it would be very useful to be able to localize in space not only the ecographic probe but also some surgical instruments (for example needles for thermoablation, endoscopes, etc.) used in the mini-invasive surgery. In this way it would be possible to superimpose the position of the parts of the instrument inserted inside the body and therefore not visible from the outside to the ecographic and CAT images.
Object of the present invention is consequently to provide a unit of support to diagnosis and to surgery that meets the requirements reported above.
According to the present invention such object is attained with an apparatus for fusion and navigation of ecographic and volumetric images of a patient and for localization of an ecographic probe and/or a surgical instrument operating on the same patient, characterised in that it comprises a plurality of passive or active optical markers positionable on the body of the patient and a plurality of active optical markers located on the ecographic probe and/or the surgical instrument, sensors of optical signals provided with devices for activation of said passive markers, said sensors being suitable for reception of optical signals produced by reflection of said passive markers and/or coming from said active markers, a device for turning on of said active markers and said devices for activation of the passive markers, a movement analysis device suitable to process the signals emitted by said sensors as a function of the optical signals being received in order to obtain from them the coordinates of said markers, a decoding device suitable to distinguish the coordinates of the active markers from the ones of the passive markers, a data processing device for localization of the ecographic probe and/or the surgical instrument and for definition of movement of the patient, a device for acquisition of the ecographic images in synchronism with the position of the ecographic probe and/or the surgical instrument and a device for fusion and navigation of the ecographic images with volumetric images coming from other image acquisition systems.
The characteristics of the present invention will be better understood through the following detailed description of an embodiment thereof which is illustrated as a non-limiting example with reference to the enclosed drawings, in which:
The apparatus shown in
Optical markers 4, preferably passive (that is capable to reflect appropriate optical signals), as for instance spherical or semispherical objects coated with back-reflecting materials are applied to the patient. The markers 4 get applied to the surface of the patient to correspond with detection or “repere” points in order to: a) localize the patient in the laboratory space and b) identify some useful detection points for the recording and the fusion of images acquired with other techniques for the formation of 3D images, as additionally explained hereinafter. For the scope a) at least three markers must be used. It is advisable to use a higher number since the body of the patient is not a rigid body but a relatively deformable structure. A higher number of markers allows therefore to better characterize the position of the different anatomical areas and, in addition, to consider also low deformations that occur during the measurement because of movements of the patient or the normal respiratory activities. For the scope b) too a minimum of three markers must be used, which can coincide with some or with all the ones in the scope a). The detection points utilised can be of two types, anatomical or artificial, which get applied to the body of the patient on the occasion of the acquisitions of 3D images carried out by different techniques. For example, in the case of fusion with CAT images, semi-permanent tattoos can be carried out on the skin of the subject to correspond with the position of radio-opaque markers present during CAT scan. Subsequently, before beginning the examination or operation procedures which utilises the apparatus according to the present invention, optical markers 4 are applied on the subject exactly to correspond with the tattooed signs.
To the ecographic probe 2 and/or surgical instrument 3 active optical markers 5 and 6 (
The surgical instrument 3 can in turn be provided with active markers 6 similar to the ones of the probe 2 (
Two or more optical sensors for example made up of video cameras 8 provided with devices for the activation of the passive markers 4, for example made up of illuminators 9 (as for instance infrared light LEDs) coaxially mounted with the objective of the video camera, receive the optical signals emitted by the active markers 5 and the ones reflected by the passive markers 4.
The video signal coming from each video camera 8 is received by a device 10 for the analysis of the movement, which processes them with the aim of extracting the two-dimensional coordinates of each marker 4, 5 or 6 present in each image. Starting from the two-dimensional coordinates of the markers and from the previously obtained calibration information it is possible, by applying opportune photogrammetry algorithms (as for instance the ones described in Borghese, N. A. and G. Ferrigno, An algorithm for 3-D Automatic Movement Detection by means of Standard TV Cameras, Ieee Trans Biomed Fng 37: 1221-1225, 1990), to obtain the accurate measure of the position of the markers in a reference system common to the laboratory. The markers recognized by the system can be both passive and active. In the case of the passive markers the light generated by the illuminators 9, being reflected by the markers, gets detected by the video cameras 8. In the case of the active markers, the light detected by the video cameras 8 is instead generated directly by the marker. To the purpose of reducing the interferences with the atmosphere where the measurement is carried out, it is preferable to use radiations different from the visible ones, in particular infrared light. In this case the active illuminators and/or markers are made up of sets of high luminosity LED diode valves which produce infrared radiation and the video cameras are provided with filters which allow the passage of the infrared radiation, therefore, attenuating the visible light. In order to implement the present application it is possible to adopt optoelectronic analyses devices available on the market, upon condition of being able to modify the timing logics for the turning on of the illuminators since the unit according to the invention requires the possibility to turn on and to turn off both the illuminators and the active markers in independent and coordinate way, as described hereinafter.
The simultaneous presence of markers on the surface of the patient and on the ecographic probe 2 and/or on the surgical instrument 3 operated by the physician can easily create problems in the calculation of the position of the markers in the space because of the difficulties in solving the problem known as “stereo matching”. In order to improve the reliability of the system for the calculation of the position of the markers placed on the probe and of the ones placed on the patient passive markers are used on the subject (which do not require connection cables, are simple to apply and can be single-use) and active markers on the probe. The turning on of the active markers and of the illuminators are managed by a synchronized starting device 11 in such a way so as to have the active markers turned on only when the illuminators are turned off. The illuminators 9 are therefore turned on only on the occasion of a subset of the image pictures acquired by the device 10. For example, if the device 10 for the analysis of the movement acquires at the frequency of 60 pictures per second, the illuminators 9 could be alternatively turned on (a picture on and a picture off). In this way the images of the markers applied to the subject and of the ones applied on the probe are obtained as alternate pictures. In addition, if one wants to have a higher sampling frequency for localization of the probe as compared with the one used for the localization of the patient (which is supposed to move much slower than the ecographic probe), it is possible to change the turning on sequences of the illuminators and of the active markers, as for instance by maintaining the active markers turned on in order to have more pictures so as to then turn them off and to turn the illuminators on for a single picture. The device 11 receives the necessary synchronisms from the central processing unit of the device 10 for the analysis of the movement and from it the controls for the turning on of the illuminators and of the active markers start. In the simplest of the embodiments the device 11 can for instance be made up of simple digital circuits in cabled logics (for example a Johnson counter with the outputs opportunely connected by a matrix of diode valves). The use of integrated microcontrollers, instead, allows to opportunely codify the models for the turning on of the active markers, thus facilitating their classification as described hereinafter.
A possible implementation of the turning on device 11 is represented by the circuit in
The flow of data coming from the analysis device 10 is processed by a decoding device 12 in such a way so as to possibly separate the coordinates of the active markers on the ecographic probe 2 (and on the surgical instrument 3) from those of the active markers placed on the body surface of the patient. An opportune program, being the information relative to the turning on sequence of illuminators and active markers known, decodes the flow of data coming from the device 10 so as to obtain two (or more, in the case of markers placed also on one or more surgical instruments) distinct sequences MP and MA, the first containing the 2D coordinates of the passive markers for each moment of turning on of the illuminators, the second (and possibly the other ones, in the case of the presence of surgical instruments) containing the 2D coordinates of the active markers for each moment of turning on of the LED. Once the time sequences of the coordinates of the active and passive markers have been subdivided, subsequent programs will therefore be capable to calculate the 3D coordinates and to identify automatically and with a minimum probability of error the different markers.
The calculation of the three-dimensional coordinates of the markers is based on stereo-photogrammetry algorithms, which require the valuation, carried out previously, of the position, of the orientation and of the geometric parameters which identify the optical characteristics of the different video cameras in the reference system of the laboratory, for instance as described in Cerveri, Borghese, & Pedotti, 1998, Complete calibration of a stereo photogrammetric system through control points of unknown coordinates. Journal of Biomechanics, 31, 935-940; Tsai, 1987, A versatile camera calibration technique for high accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal of Robotics and Automation, RA-3(4), 323-344.
Opportune “tracking” procedures are instead utilised in order to be able to follow the movement of the markers in real time, while maintaining the information on their classification. Such procedures can be based:
a) on prediction algorithms bases on analytical criteria (as for example, Gamage & Lasenby, 2002, New least squares solutions for estimating the average center of rotation and the axis of rotation, Journal of Biomechanics, 35, 87-93) or functional criteria (as for example, Cappozzo, Catani, Leardini, Blessed, & Groce, 1996, Position and orientation in space of bones during movement: Experimental artifacts. Clinical Biomechanics, 11, 90-100).
b) on prediction algorithms bases on the Kalman filtering (as for example, Brown & Hwang, 1997, Introduction to random signals and applied Kalman filtering (3rd ed.), New York: John Wiley & Sons Inc.), which for instance determines the temporal series of the coordinates of the different markers by means of a picture by picture minimization of the distances between the points as measured by the video cameras and the corresponding back-projected model (for example, Cerveri, Pedotti, Ferrigno, Robust recovery of human motion from video using Kalman filters and virtual humans Human Movement Science 22 (2003) 377-404). In the case of the markers placed on the probe, the model is the one of a rigid body with six degrees of freedom, in which the relative distances between markers are known. In the case of the markers placed on the patient, the model is instead the one of a deformable body, in which strong restraints are anyhow present on the distances of the markers and on their movement.
The two sequences of data MA and MP, respectively relative to the active markers and the passive markers, are received by a data processing device 13, which can be considered as being subdivided in two distinguished parts 13a and 13b, respectively for localization of the ecographic probe and/or of the surgical instrument and for the definition of the movement of the patient.
It is a device capable to process the data coming from the decoding device 12, relative to the space coordinates of the active markers placed on the echography probe and possibly on the surgical instrument and of the back-reflecting passive markers placed on the patient, with the aim of localizing probe, image plane and patient in a single absolute reference system L (laboratory), determined by a set of cartesian axes XL, YL, ZL.
The device 13 identifies with opportune processing methods the parameters of the geometric transformations (roto-translations) to be carried out on the points belonging to the plane of the ecographic image and it calculates the new co-ordinate of such points in the reference system of the laboratory according to the equation
P′(xL, yL, zL, t)=T(t)·C·P(xI, yI, t) (1)
where P(xI, yI, t) is any point belonging to the image plane, variable in time t during the image scanning, C defines the constant of geometric transformation between the reference system S relative to the image plane and the reference system S integral with the probe, T(t) defines the geometric transformation between S and L, P′ (xL, yL, t) represents each point represented in the reference system L.
The previous equation can be expressed in matrix form:
where R stands for the rotation sub-matrixes (made up of director cosines) and O for the translation or offset sub-matrixes, whereas Z for vectors of zeros.
While the identification of matrix C is predetermined by an opportune calibration, the matrix T, variable in time, is identified at each moment of measurement.
The different calibration methods which can be utilised for the determination of the matrix C can be brought back to three different categories: 1) single point (or single line); 2) 2-D alignment; 3) “freehand” methods.
Single point methods use a calibration object (“phantom”) that contains a target point (“target”) made up of a sphere, a grain or a pin (for example, Legget et al., System for quantitative three-dimensional echocardiography of the left ventricle based on a magnetic-field position and orientation sensing system IEEE Trans Biomed Eng, 1998, 45:494-504) or a cross-wire (for example, Barry et al., Three-dimensional freehand ultrasound: Image reconstruction and volume analysis. Ultrasound Med Biol, 1997; 23: 1209-1224). The target is visualized from different directions. The advantage of these methods is their semplicity, even if the number of images to be acquired must be higher than the number of possible degrees of freedoms (three rotations and three translations). Typically, a few tens of images are used for the calculation of C. A variation of this approach considers the possibility to visualize a planar structure by means of an image plane US approximately perpendicular to the same structure (for example, Hook 2003, Probe calibration for 3D ultrasound image localization. Internship report. Laboratoire TIMC Grenoble, France/University of Magdeburg, Germany), thus obtaining a line which can be easy identified. Specific points on different images are then utilised for the calibration.
The idea at the basis of the 2-D alignment methods, instead, is to manually align the image plane US with a planar set of points (for example, cross-wire or tips of toothed membranes), using as a guide and reference the display of the ecographer (for example, Berg et al., Dynamic three-dimensional freehand echiocardiography using raw digital ultrasound data. Ultrasound Med Biol, 1999, 25: 745-753). Since the points are distributed on a 2-D plane, the orientation of the plane is not ambiguous and, in principle, only one image is necessary for the calibration. However, since such plane has in fact a finished thickness, the alignment procedure can be as a result long and difficult, and the result can be not satisfactory in terms of accuracy.
In the recent past different calibration methods have been proposed which use “free-hand” scannings (for example, Bouchet et al., Calibration of threedimensional ultrasound images for image-guided radiation therapy. Phys Med Biol, 2001, 46: 559-577). The calibration objects utilised are made up of wires arranged according to certain configurations which allow, by observing the different images, a univocous determination of the position and of the orientation of the image plane US. In other words, such methods do not require burdensome alignment procedures with points, lines or planes, and only a few images are necessary.
Such procedure well adjusts to clinical conditions, in which it is not always possible to avail of long times and skilled staff.
As per what concerns instead the identification of the matrix T, different methods can be implemented, the simplest one of which is the so-called Gram-Schmidt orthonormalization. Such procedure, having available the coordinates of three active markers integral with the probe, reconstructs three axes w1, w2, W3 as it follows:
where by O the marker chosen as origin is meant, whereas A (xA, yA, zA) and B (xB, yB, zB) are the other two markers. Starting from such axes, the algorithm therefore reconstructs an orthogonal base u1, u2, u3:
The orthonormal base is finally obtained by dividing u1, u2, and u3 by their norm. The three column vectors of the orthonormal base make up the elements of the rotation matrix RT of the equation 2, whereas the vector OT is made up of the coordinates of the marker O.
As an alternative to the method previously described, in order to identify the reference system of the probe, the so-called “redundant” methods can be used which uses a number of markers higher than three. In this way it is possible not only to obtain with higher accuracy the roto-translation parameters, but also to manage possible occlusions of the markers which can occur during the acquisition.
The processing device 13, in addition, utilises the coordinates of the passive markers placed on the patient with the aim of localizing the reference system of the patient in the reference system of the laboratory, with methods similar to the ones previously described. In the case in which one utilises markers detectable both by the device 10 and by other systems for the acquisition of volumetric images (such as CAT, MRI, PET, etc.), as for instance radio-opaque spheres coated with back-reflecting material, and such markers are placed in the same positions as chosen previously, the determination of the reference system of the patient allows the fusion of the different images.
The apparatus in
Finally the apparatus in
The fusion between ecographic images and volumetric images such as CAT, MRI, PET, etc. turns out to be possible when, at the moment of the recording of the volumetric images, opportune detection points have been acquired and subsequently, during the surgical session or the ecographic analysis, passive markers detectacle by the video cameras 8 are applied to correspond with such points. In these conditions the device 15, provided with an opportune calculation and visualization program, will be capable to present to the operator in real time both the volumetric images as well as the ecographic images, represented in the same reference system, identified by the device 13. At each instant of acquisition of the ecographic images, the device 15, starting from the set of the geometric parameters coming from the device of acquisition 14 and from the data relative to the position of the patient coming from the data processing device 13, carries out the following calculations: a) it calculates the position of each element of the ecographic image in the reference system of the patient; b) it calculates the position of each element of the volumetric image in the reference system of the patient; c) it represents on a screen, in a single spacial reference, the volumetric images (represented by means of sections chosen by the operator), the ecographic images and possibly the surgical instrument.
In conclusion, the apparatus in
Particularly critical in the apparatus herein described is the stage of localization of the markers applied on the probe and/or on the instrument and of the markers applied on the subject. In fact by moving the probe and/or the instrument on the body surface there is the risk that the images of the markers located on the subject can be fused or in any case confused with the ones of the markers on the probe in the stage of 3D reconstruction. In order to improve the ability to discriminate the markers of the probe and/or of the instruments from the ones placed on the subject one synchronises, as already said, the turning on of the active markers with the system for the analyses of the movement through opportune devices (11) capable to turn the illuminators 9 on which activate the passive markers alternatively to the markers on the probe or to the ones on the instruments. In this way two or more separate flows of data are obtained, one relative to the position of the detection points which identify the position of the patient, one associated with the coordinates relative to the active markers which localize the probe, and, possibly, one or more flows of data which localize as many surgical instruments, preventing, therefore, possible calculation errors. In addition, the use of determined codes for the turning on of the active markers located on the probe and/or on the tools allows to automate the classification procedures necessary to the 3-D calculation. The coordinates of a few markers opportunely applied on the patient allow in addition to obtain useful information on his conditions, for example, the stage of the respiratory activity and the possible deformations induced on the chest by the variations of position with reference to the one assumed during the recording of the images obtained with the other methods. The methods proposed in this patent allow to obtain advanced ecographic systems for: 1) the fusion of the ecographic images with other techniques for the acquisition of volumetric images; 2) the support to surgery, by allowing the simultaneous recording, in the same reference system, of the positions of the patient, of ecographic probe and surgical tools, by compensating possible movements of the same patient. In particular, through the synchronized detection of the ultrasonographic images and through their fusion with pre-surgical images the identification of anatomicsl-functional elements evidenced by techniques such as CAT, MRI and contrast means US becomes possible thus facilitating and optimizing the localization and navigation of anatomical areas and their surgical treatment.
| Number | Date | Country | Kind |
|---|---|---|---|
| MI2004A001448 | Jul 2004 | IT | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/EP05/53490 | 7/19/2005 | WO | 1/19/2007 |
