The present invention relates to fluoroscopic image sequences, and more particularly to detecting coronary vessel layers from fluoroscopic image sequences.
Angiography is a medical imaging technique in which X-ray images are used to visualize internal blood filled structures, such as arteries, veins, and the heart chambers. Since blood has the same radiodensity as the surrounding tissues, these blood filled structures cannot be differentiated from the surrounding tissue using conventional radiology. Thus, in angiography, a contrast agent is added to the blood, usually via a catheter, to make the blood vessels visible via X-ray. In many angiography procedures, X-ray images are taken over a period of time, which results in a sequence of fluoroscopic images, which show the motion of the blood over the period of time. Such fluoroscopic image sequences contain useful information that can be difficult to decipher due to the collapsing of 3-dimensional information into the 2-dimensional images.
In traditional computer imaging problems of motion estimation, occlusion handling or motion segmentation are typically the main concerns. Accordingly, traditional techniques for extracting objects of interest from image sequences typically use intensity based approaches to differentiate between objects in the image sequences. However, such traditional techniques can yield erroneous results in medical image sequences, such as fluoroscopic image sequences, which are generated using the phenomenon of transparency. Since various internal structures have different levels of transparency in the fluoroscopic images, these structures can overlap, and it may be difficult to accurately distinguish between these structures in the fluoroscopic image sequences using the traditional intensity based approaches.
The present invention provides a method and system for extracting coronary vessels from fluoroscopic image sequences using coronary digital subtraction angiography (DSA). Embodiments of the present invention provide real-time extraction of coronary vessel layers for each frame of a fluoroscopic image sequence. Embodiments of the present invention utilize a Bayesian framework for tracking the moving layer of dynamic background structures to achieve coronary subtraction in cardiac fluoroscopic sequences. Dense motion estimation between mask images and a contrast image are used to predict the background layer of the contrast image. The predictions from multiple mask images are statistically fused to obtain the final estimation of the background layer of the contrast image, which can then be subtracted from the contrast image to generate the coronary vessel layer.
In one embodiment of the present invention, a set of mask images is received and a contrast image for the coronary region is received. The contrast image may be one of a sequence of contrast images. A dense motion field is calculated between each of the mask images and the background region of the contrast image and a covariance is calculated for each motion vector. Multiple background layer predictions are generated by generating a background layer prediction from each mask image based on the calculated motion field and covariances. The multiple background layer estimates are combined using statistical fusion to generate a final estimated background layer. The final estimated background layer is subtracted from the contrast image to extract a coronary vessel layer for the contrast image.
These and other advantages of the invention will be apparent to those of ordinary skill in the art by reference to the following detailed description and the accompanying drawings.
The present invention is directed to a method for detecting coronary vessels from fluoroscopic images. Embodiments of the present invention are described herein to give a visual understanding of the coronary vessel extraction method. A digital image is often composed of digital representations of one or more objects (or shapes). The digital representation of an object is often described herein in terms of identifying and manipulating the objects. Such manipulations are virtual manipulations accomplished in the memory or other circuitry/hardware of a computer system. Accordingly, is to be understood that embodiments of the present invention may be performed within a computer system using data stored within the computer system.
Digital subtraction angiography (DSA) is a technique for visualizing blood vessels in the human body. In DSA, a sequence of fluoroscopic images, referred to as contrast images, is acquired to show the passage of contrast medium that is injected into the vessel of interest. A sequence of fluoroscopic images contains multiple 2D X-ray images obtained in real time. The X-ray images record a certain field of view of a time period. Accordingly, motion of objects within the field of view can be observed in a sequence of fluoroscopic images. The background structures are largely removed from the contrast images by subtracting a mask image, which is an image acquired before contrast injection. However, the appearance of background structures in a contrast image and a mask image can differ due to fluctuation of radiation, acquisition noise, and patient motion. In traditional DSA where the mask image is directly subtracted from the contrast image, the difference between the background structures in the contrast images and the mask image can cause errors in detecting the blood vessels.
The main cause of differences in the appearance of background structures in a contrast image and a mask image is patient motion. Motion correction for DSA can involve various techniques for image motion estimation, where the motion between a contrast image and a mask image is obtained by warping one image to match the other. Commonly used matching criteria for motion estimation include optimization of various similarity or error measures, such as normalized cross-correlation, sum of absolute values of differences, variance of differences, etc. In the case of coronary DSA, cardiac motion and respiratory motion cause more severe differences between contrast images and mask images. Furthermore, because of the complexity of cardiac motion and breathing motion, the commonly used matching criteria are often violated in image regions of coronary vessels, making it more difficult to estimate the background motion around coronary vessels in coronary DSA.
Embodiments of the present invention provide a viable approach to coronary DSA that combines learning-based object detection, dense motion estimation, uncertainty propagation, and statistical fusion to achieve fully automatic and real-time coronary vessel subtraction. Embodiments of the present invention formulate coronary DSA as a problem to remove dynamic background structures from a contrast image. Pixel values in X-ray images are determined by the energy flux incident on the detector, which is commonly described as an exponential attenuation function as the X-ray beams pass through the layers of tissue. As a result, X-ray images are often dealt with in the logarithmic space and modeled as a linear superposition of multiple layers. In the case of coronary DSA, in an advantageous implementation of the present invention, only two layers are considered, a coronary layer and a background layer. The coronary layer is defined as a transparent image layer including coronary arteries filled with contrast medium. The background layer is defined as a transparent layer including background structures. A contrast image in a fluoroscopic sequence is denoted herein as It(x), its coronary layer is denoted as IC,t(x), and it background layer is denoted as IB,t(x), where t is a time index and x is a pixel location. The formulation of layer composition after logarithmic post-processing can be expressed as It(x)=IC,t(x)+IB,t(x). The goal of coronary DSA is to remove the background layer to extract the coronary arteries while both layers are undergoing cardiac and respiratory motion.
At step 102, multiple mask images are received. The mask images are fluoroscopic or X-ray images of a coronary region of a patient without any contrast agent injected into the coronary vessels. The mask images are acquired at different cardiac and breathing phases to serve as static masks for background estimation. For example, the mask images can be a sequence taken over the course of at least one cardiac cycle (heartbeat). Accordingly, the cardiac motion over the course of a full cardiac cycle is implicitly embedded in the set of mask images, such that background structures in various cardiac phases are captured in the set of mask images. The mask images can be received by acquiring the mask images directly from an X-ray scanning device. It is also possible that the mask images can be received by loading mask images that were previously acquired images and stored, for example, on a computer readable medium or storage of a computer system. When the set of mask images is received, the mask images are stored on a memory or storage of a computer system that is implementing the method of
At step 104, a sequence of contrast images is received. The sequence of contrast images can be electronic data representing fluoroscopic or X-ray images resulting from an X-ray procedure, such as an angiograph, in which a contrast agent is injected into the coronary vessels. The sequence of contrast images are images of the coronary region taken at a regular interval over a time frame. Each image in the sequence can be referred to as a frame. The contrast images can be received directly from an X-ray scanning device, or previously stored contrast images can be loaded. The sequence of contrast images are processed frame by frame to independently extract the coronary vessels for each contrast image in the sequence. Accordingly, steps 106-118 of
At step 106, vessel regions in the contrast image are detected using learning-based vessel segment detection. In order to detect vessel regions in the contrast image, a vessel segment detector can be implemented using a binary classifier that separated the class of vessel images from the class of non-vessel images. The binary classifier is learned from a set of vessel images and non-vessel images using a probabilistic boosting tree (PBT) algorithm. To prepare a set of training samples, coronary vessels are manually annotated in a number of contrast images, local image patches containing the annotated vessels are used as the positive set of training samples, and patches away from the vessels are used as the negative training samples. The PBT algorithm learns a binary decision tree, where each node of the tree is a binary classifier by itself and is learned using the Adaboost algorithm. An extended set of Haar features can be used for training the boosted classifiers.
Given an image patch Ipatch, the trained binary classifier calculates the posterior probability that a vessel segment appears in this image patch p(vessel|Ipatch). A detected vessel region can be defined as the image area where vessel segments are detected with high probabilities (i.e., greater than a certain threshold). The remaining area of the image includes primarily background structures and is defined as the background region. To detect vessel segments with different thicknesses, the contrast image can be rescaled several times to generate a coarse-to-fine image pyramid, and the vessel segmentation detection can be performed at multiple scales. At each scale, in order to speed up vessel detection for a contrast image, it is possible to apply a set of efficient steerable filters to identify ridge features and their different orientations in the contrast image, and then perform the vessel detection using the binary classifier only at the ridge features.
Image 206 of
To speed up the process of vessel segment detection, ridge-like image structures are first located in the contrast image through the use of steerable filters, as described above and illustrated in
Returning to
Returning to
Given the motion estimation {circumflex over (v)}(x) and its covariance C({circumflex over (v)}(x)), the probability distribution of the motion vector v(x) can be approximated as a Gaussian distribution with mean {circumflex over (v)}(x) and covariance C({circumflex over (v)}(x)), such that:
v(x)˜N({circumflex over (v)}(x),C({circumflex over (v)}(x))); E[v(x)]={circumflex over (v)}(x), Cov[v(x)]=C({circumflex over (v)}(x)). (1)
Pixel values in the background layer IB,t(x) are predicted from the mask image:
IB,t(x)=Im(x+v(x)). (2)
According to an advantageous embodiment of the present invention, second order statistics are incorporated in order to derive the prediction probability functions of pixel values in the background layer p(IB,t(x)|Im). In general, the transformation function Im(x+v(x)) is a nonlinear function of v(x) and techniques such as linearization and unscented transformation are required to parameterize the means and covariances of the probability distribution. Due to the computational complexity of the unscented transformation, the transformation function can be linearized as follows:
Im(x+v(x))≈Im(x+{circumflex over (v)}(x))+∇TIm(x+{circumflex over (v)}(x))[v(x)−{circumflex over (v)}(x)]
∇Im(x+v(x))=[∂xIm(x+{circumflex over (v)}(x)),∂yIm(x+{circumflex over (v)}(x))]T (3)
where ∇Im(x+v(x)) denotes the gradient vector of the transformed image Im(x+v(x)). The mean and variance of IB,t(x) can be approximated as:
E[IB,t(x)|Im]=Im(x+{circumflex over (v)}(x))
Var[IB,t(x)|Im]=∇TIm(x+{circumflex over (v)}(x))·C({circumflex over (v)}(x))·∇Im(x+{circumflex over (v)}(x)) (4)
Through linearization of the transformation function, the uncertainties in motion estimation are propagated to the prediction of the pixel values of the background layer. The prediction probability density function can be approximated by a Gaussian as:
p(IB,t(x)|Im)=N(IB,t(x);E[IB,t(x)|Im],Var[IB,t(x)|Im]). (5)
Images 210 show multiple background layer predictions.
Returning to
Denote {Im,i(x):i=1, . . . , ns} as the static mask images acquired at time t1, . . . tn
p(IB,t(x)|Im,t
p(IB,t(x)|ID,k)=N(IB,t(x);mt,t-k(x),σ2t,t-k(x)) (k=0, . . . , nd) (6)
where mt,t
mt,t-k(x)=E[IB,t(x)|ID,k
Equation (7) results in the final estimated background layer ÎB,t(x) of the contrast image. Image 212 of
Returning to
ÎC,t(x)=It(x)−ÎB,t(x) (8)
Image 214 of
At step 116, a coronary enhanced image is generated. Once the coronary vessel layers are extracted from the contrast image, it is possible to virtually enhance the coronary vessels in the contrast image. This is a direct clinical application of the coronary DSA method can save contrast medium and lower radiation. With the background layer separated from the coronary layer, it is straightforward to fade out the background layer or to enhance the coronary layer by layer composition. Accordingly, an enhanced contrast image can be obtained as:
αCÎC,t+αBÎB,t (αC≧1, 0≦αB≦1) (9)
For example, to fade out the background layer, αC can be set to αC=1 and αB can be decreased. To virtually enhance the contrast of the coronary vessels, αB can be set to αB=1 and αC can be increased. Image 216 of
At step 116, the extracted coronary vessel layer and the coronary enhanced image are output. The extracted coronary vessel layer can be output by displaying the coronary vessel layer as an image on a display device. Similarly, the coronary enhanced image can be output by displaying the coronary enhanced image on a display device. The coronary vessel layer and coronary enhanced image may also be output by storing the coronary vessel layer and/or the coronary enhanced image, for example, in a computer readable medium or storage or memory of a computer system.
The above-described methods for extracting coronary vessel layers from a sequence of fluoroscopic contrast images may be implemented on a computer using well-known computer processors, memory units, storage devices, computer software, and other components. A high level block diagram of such a computer is illustrated in
The foregoing Detailed Description is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the principles of the present invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention.
This application claims the benefit of U.S. Provisional Application No. 61/175,567, filed May 5, 2009, the disclosure of which is herein incorporated by reference.
Number | Name | Date | Kind |
---|---|---|---|
5557684 | Wang et al. | Sep 1996 | A |
5647360 | Bani-Hashemi | Jul 1997 | A |
6826292 | Tao et al. | Nov 2004 | B1 |
6987865 | Szeliski et al. | Jan 2006 | B1 |
7155032 | Szeliski et al. | Dec 2006 | B2 |
7995824 | Yim | Aug 2011 | B2 |
20060285747 | Blake et al. | Dec 2006 | A1 |
20070116356 | Gong et al. | May 2007 | A1 |
20090010512 | Zhu et al. | Jan 2009 | A1 |
20090080729 | Zhang et al. | Mar 2009 | A1 |
20100034446 | Zhu et al. | Feb 2010 | A1 |
Entry |
---|
Barbu, A., et al., “Hierarchical Learning of Curves Application to Guidewire Localization in Fluoroscopy”, IEEE Int'l. Conf. Comp. Vision and Pattern Rec., 2007. |
Coleman, T.F., et al., “A Reflective Newton Method for Minimizing a Quadractic Function Subject to Bounds on some of the Variables”, SIAM Journal on Optimization, 1996. |
Comaniciu, D., “Nonparametric Information Fusion for Motion Estimation”, IEEE Conf. Comp. and Pattern Rec., 2003. |
Freeman, W.T., et al., “The Design and Use of Steerable Filters”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1991. |
Tu, Z., “Probabilistic Boosting-Tree: Learning Discriminative Models for Classification, Recognition, and Clustering”, IEEE Int'l. Conf. Comp. Vision, 2006. |
Number | Date | Country | |
---|---|---|---|
20110033102 A1 | Feb 2011 | US |
Number | Date | Country | |
---|---|---|---|
61175567 | May 2009 | US |