Patent Application
20220319004
The present invention relates to automated vessel analysis from 2D image data.
Artery diseases involve circulatory problems in which narrowed arteries reduce blood flow to body organs. For example, coronary artery disease (CAD) is the most common cardiovascular disease, which involves reduction of blood flow to the heart muscle due to build-up of plaque in the arteries of the heart.
Current clinical practices used in the diagnosis of coronary artery disease include coronary angiography and/or non-invasive image-based methods such as computerized tomography (CT), which require constructing a 3D model of arteries from which a computer can create cross-sectional images (slices) of the imaged tissues, and which require an expert's interpretation of the images. Other image-based diagnostic methods typically require user input (e.g., a physician is required to mark vessels in an image) based on which further image analysis may be performed to detect pathologies such as lesions and stenoses.
Embodiments of the invention provide a fully automated solution to vessel analysis based on image data. A system, according to embodiments of the invention, detects a pathology and may provide a functional measurement value from a 2D image of a vessel, without having to construct a 3D model (i.e., without using CT techniques) and without requiring user input regarding the vessel and/or location of the pathology. Thus, embodiments of the invention enable detecting pathologies and providing a functional measurement value from 2D lengthwise images of a vessel obtained during X-ray angiography, as opposed to 2D cross section images that are used in CT procedures, such as CT angiography (CTA). Consequently, embodiments of the invention enable vessel analysis while exposing a patient to a significantly lower radiation dose compared with the level of radiation used during CT. Additionally, embodiments of the invention enable real-time analysis and treatment (e.g., stenting) of vessels, whereas analysis of CT generated images cannot be done in real-time and does not enable real-time treatment of vessels.
In one embodiment, a system for analysis of a vessel includes a processor to receive at least two 2D images of a stenosis in a patient's vessel, the images typically being 2D lengthwise images obtained during X-ray angiography, each of the images captured from a different angle. The processor determines the location of the stenosis in the vessel in each of the images and calculates an FFR value of the vessel or for the stenosis, based on a color or grayscale feature, extracted from the location of the stenosis from each of the images. Features extracted from the “location of the stenosis” may be features extracted from pixels of the stenosis itself and/or surrounding pixels and/or pixels in the vicinity of the stenosis.
The processor, which may be in communication with a user interface device, may output to a user (e.g., display on the user interface device) an indication of the FFR value.
The processor may input the color or grayscale feature into a machine learning model to predict the FFR value. In addition to the color or grayscale feature, the processor may input into the machine learning model additional features, such as, a shape feature and/or a morphological feature, to calculate the FFR value based on the color or grayscale feature and on one or more of these additional features.
In embodiments of the invention, the location of the stenosis in the vessel is a location of the stenosis relative to a structure of the vessel. The processor may apply a classifier on a first image to determine the location of the stenosis and then the processor may determine the location of the stenosis in the vessel in a second image by tracking the stenosis to the second image, based on the determined location of the stenosis relative to the structure of the vessel.
The processor may attach a virtual mark to the stenosis to track the stenosis from the first image to the second of image based on the virtual mark. The mark may be based on the location of the stenosis relative to the structure of the vessel.
Embodiments of the invention determine, in each image, a location of a pathology (e.g., stenosis) relative to a structure of the vessel. This enables tracking the same pathology throughout different angiogram images, even if the images were captured from different angles (e.g., due to rotation of the X-ray imaging device and/or rotation of the patient which causes the pathology to appear different in every image). Thus, embodiments of the invention enable automatic analysis of a vessel, such as, determination of functional measurements (e.g., FFR values) from images obtained during X-ray angiography, with no need for user (e.g., physician) input.
The invention will now be described in relation to certain examples and embodiments with reference to the following illustrative figures so that it may be more fully understood. In the drawings:
Embodiments of the invention provide methods and systems for automated analysis of vessels from images of the vessels, or portions of the vessels, and display of the analysis results.
Analysis, according to embodiments of the invention, may include diagnostic information, such as presence of a pathology, identification of the pathology, location of the pathology, etc. Analysis may also include functional measurements, such as estimates of FFR values. The analysis results, may be displayed to a user.
A “vessel” may include a tube or canal in which body fluid is contained and conveyed or circulated. Thus, the term vessel may include blood veins or arteries, coronary blood vessels, lymphatics, portions of the gastrointestinal tract, etc.
An image of a vessel may be obtained using suitable imaging techniques, for example, X-ray imaging, ultrasound imaging, Magnetic Resonance imaging (MRI) and others suitable imaging techniques. Embodiments of the invention use angiography, which includes injecting a radio-opaque contrast agent into a patient's blood vessel and imaging the blood vessel using X-ray based techniques. The images obtained, according to embodiments of the invention, are typically 2D lengthwise images of a vessel, as opposed to, for example, 2D cross section images that are used in methods that require constructing a 3D model of the vessel, such as CTA and other CT methods.
A pathology may include, for example, a narrowing of the vessel (e.g., stenosis or stricture), lesions within the vessel, etc.
A “functional measurement” is a measurement of the effect of a pathology on flow through the vessel. Functional measurements may include measurements such as an estimate of fractional flow reserve (FFR), an estimate of instant flow reserve (iFR), coronary flow reserve (CFR), quantitative flow ratio (QFR), resting full-cycle ratio (RFR), quantitative coronary analysis (QCA), and more.
In the following description, various aspects of the present invention will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the present invention. However, it will also be apparent to one skilled in the art that the present invention may be practiced without the specific details presented herein. Furthermore, well known features may be omitted or simplified in order not to obscure the present invention.
Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as “using”, “analyzing”, “processing,” “computing,” “calculating,” “determining,” “detecting”, “identifying” or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing device, that manipulates and/or transforms data represented as physical, such as electronic, quantities within the computing system's registers and/or memories into other data similarly represented as physical quantities within the computing system's memories, registers or other such information storage, transmission or display devices. Unless otherwise stated, these terms refer to automatic action of a processor, independent of and without any actions of a human operator.
In one embodiment, which is schematically illustrated in
Processor 102 performs analysis on the received image(s) and communicates analysis results and/or instructions or other communications, based on the analysis results, to a user via the user interface device 106. In some embodiments, user input can be received at processor 102, via user interface device 106.
Vessels 113 may include one or more vessel or portion of a vessel, such as a vein or artery, a branching system of arteries (arterial trees) or other portions and configurations of vessels.
Processor 102 may include, for example, one or more processors and may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a microprocessor, a controller, a chip, a microchip, an integrated circuit (IC), or any other suitable multi-purpose or specific processor or controller. Processor 102 may be locally embedded or remote, e.g., on the cloud.
Processor 102 is typically in communication with a memory unit 112. In one embodiment the memory unit 112 stores executable instructions that, when executed by the processor 102, facilitate performance of operations of the processor 102, as described below. Memory unit 112 may also store image data (which may include data such as pixel values that represent the intensity of light having passed through body tissue and/or light reflected from tissue or from a contrast agent within vessels, and received at an imaging sensor, as well partial or full images or videos) of at least part of the images 103.
Memory unit 112 may include, for example, a random access memory (RAM), a dynamic RAM (DRAM), a flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units.
The user interface device 106 may include a display, such as a monitor or screen, for displaying images, instructions and/or notifications to a user (e.g., via graphics, images, text or other content displayed on the monitor). User interface device 106 may also be designed to receive input from a user. For example, user interface device 106 may include or may be in communication with a mechanism for inputting data, such as a keyboard and/or mouse and/or touch screen, to enable a user to input data.
All or some of the components of system may be in wired or wireless communication, and may include suitable ports such as USB connectors and/or network hubs.
In one embodiment, processor 102 detects a location of a pathology, such as a stenosis, within a 2D image of a patient's vessels. Thus, processor 102 may automatically indicate the actual location of a stenosis on an image of a patient's vessels, e.g., on an X-ray image.
In one example, which is schematically illustrated in
Classifiers, such as DenseNet, CNN (Convolutional Neural Network) or EfficientNet, may be used to obtain a determination of presence of a pathology and to determine the location of the pathology. Classifiers, according to embodiments of the invention, may be pre-trained on training data that includes 2D (typically lengthwise) X-ray angiogram images of vessels which may include a pathology (e.g., stenosis). In one embodiment the training data includes X-ray angiography images that include a stenosis and, optionally, X-ray angiography images that do not include a stenosis. The neural network composing the classifier may be learned by a scheme of supervised learning, or possibly semi-supervised learning. In the training of the neural network, training data is repeatedly input to the neural network and an error of an output of the neural network for the training data and a target, is calculated, and the error of the neural network is back-propagated in order to decrease the error and update the neural network. In the case of supervised learning, training data, which includes 2D X-ray angiogram images (e.g., images of a single vessel (with and possibly without stenoses) obtained from different points of view and/or images of different vessels with and possibly without stenoses), is labelled with a correct answer (e.g., stenosis exists/does not exist in the image).
Applying a classifier on an angiography image enables detection of a pathology by using computer vision techniques, without requiring user input regarding a location of the vessels in the image and/or location of the pathology.
Processor 102 may then cause an indication of the pathology to be displayed, via the user interface device 106, on the image of the patient's vessels (e.g., image 103), at the location of the pathology (step 210). In some embodiments the indication of pathology can be displayed at a same location on a plurality of consecutive images (e.g., a video angiogram).
An indication of a pathology displayed on a display of a user interface device may include, for example, graphics, such as, letters, numerals, symbols, different colors and shapes, etc., that can be superimposed on the image or video of the patient's vessels.
In some embodiments, processor 102 determines a probability of presence of the pathology, e.g., based on output of the classifier, and causes an indication of the pathology to be displayed only if the probability is above a predetermined threshold.
In some embodiments, processor 102 obtains a vessels mask by using semantic segmentation algorithms on the image. A machine learning model can be used for the segmentation, e.g. deep learning models such as Unet or FastFCN or other deep learning based semantic segmentation techniques.
In other embodiments, one example of which is schematically illustrated in
In some embodiments, the classifier may be applied on a plurality of partially overlapping portions of an image of a vessel (possibly, a vessel mask image) and may output an analysis of the vessel based on the partially overlapping portions of image. In one embodiment, the plurality of portions 311 of image 310, on which the classifier is applied, each include a different part of the centerline 301, such as parts 301a, 301b and 301c, where each of the different parts of the centerline may partially overlap another part of the centerline. For example, part 301a partially overlaps part 301b and part 301b partially overlaps parts 301a and 301c. This ensures that each portion of image input to the classifier includes a full portion of the vessel (typically including both borders 321 and 322) such that a possible stenosis will be located more or less in the center parts of the portion of image and not at the periphery of the portion of image, where it may be cut off or otherwise not clearly presented.
Determining a centerline as well as calculating distances D1 and D2 can be done by using known algorithms for medial axis skeletonization, for example, scikit-image algorithms.
In some embodiments the 2D image (from which a vessels mask can be obtained) is an optimal image, selected from a plurality of 2D images of the patient's vessels, as the image showing the most detail. In the case of angiogram images, which include contrast agent injected to a patient to make vessels (e.g., blood vessels) visible on an X-ray image, an optimal image may be an image of a blood vessel showing a large/maximum amount of contrast agent. Thus, an optimal image can be detected by applying image analysis algorithms (e.g., to detect the image frames having the most colored pixels) on a sequence of images.
In one embodiment, an image captured at a time corresponding with maximum heart relaxation is an image showing a maximum amount of contrast agent. Thus, an optimal image may be detected based on capture time of the images compared with, for example, measurements of electrical activity of the heartbeat (e.g., ECG printout) of the patient.
In one embodiment, which is schematically illustrated in
In some embodiments, once a pathology is detected in a first image from the plurality of images, the pathology may be tracked throughout the plurality of images (e.g., video) (step 405), such that the same pathology can be detected in each of the images, even if it's shape or other visual characteristics change in between images.
One method of tracking a pathology may include attaching a virtual mark to the pathology detected in the first image. In some embodiments the virtual mark is location based, e.g., based on location of the pathology within portions of the vessel. In some embodiments, a virtual mark includes the location of the pathology relative to a structure of the vessel. A structure of a vessel can include any visible indication of anatomy of the vessel, such as junctions of vessels and/or specific vessels typically present in patients. Processor 102 may detect the vessel structure in the image by using computer vision techniques (such as by using the vessel mask described above), and may then index a detected pathology based on its location relative to the detected vessel structures.
For example, a segmenting algorithm can be used to determine which pixels in the image are part of the pathology and the location of the pathology relative to structures of the vessel can be recorded, e.g., in a lookup table or other type of virtual index. For example, in a first image a stenosis is detected at a specific location (e.g., in the distal left anterior descending artery (LAD)). A stenosis located at the same specific location (distal LAD) in a second image, is determined to be the same stenosis that was detected in the first image. If, for example, more than one stenosis is detected within the distal LAD, each of the stenoses are marked with their relative location to additional structures of the vessel, such as, in relation to a junction of vessels, enabling to distinguish between the stenoses in a second image.
Thus, the processor 102 creates a virtual mark which is specific per pathology, and in a case of multiple pathologies in a single image, distinguishes the multiple pathologies from one another. The pathology can then be detected in following images of the vessel, based on the virtual mark.
In some embodiments, processor 102 may assign a name or description to a pathology based on the location of the pathology within the vessel and the indication of pathology can include the name or description assigned to the pathology.
In one embodiment, the processor can calculate a value of a functional measurement, such as an FFR estimated value, for each pathology and may cause the value(s) to be displayed.
As schematically illustrated in
In one embodiment, a machine learning model running a regression algorithm is used to predict a value of a functional measurement (e.g., FFR estimate) from an image of the vessel, namely, based on a color feature of the image at the location of a pathology. For example, the machine learning algorithm can be implemented by using the XGBoost algorithm or other Gradient Boosted Machine or decision trees regression. In other examples, neural network or deep neural network based regression can be used.
Processor 102 then outputs an indication of the functional measurement to a user (step 508), e.g., via user interface device 106.
In some embodiments, at least two images (typically, 2D images obtained during X-ray angiography, each image captured from a different angle) of the same stenosis in a patient's vessel, are used. Processor 102 determines a location of the stenosis in the vessel in each of the images and calculates an FFR value of the vessel/stenosis based on a color or grayscale feature extracted from each of the images, at the location of the stenosis.
The image of the patient's vessels may be a grayscale image and the color feature may include shades of grey. Other features may be input to the machine learning model in addition to the color features, for example, morphological features (e.g., branching system of arteries (arterial trees) or other portions and configurations of vessels) and/or shape features.
Thus, processor 102 determines a functional measurement directly from an image, e.g., by employing machine learning models and classifiers as described above, with no need for user input.
Since, as described herein, the same stenosis may be tracked throughout different images (even images captured from different points of view or angles), color or grayscale features can be easily extracted from the location of the same stenosis in each of the different images. Thus, an FFR value (or other functional measurement) for a specific stenosis may be provided based on color features from two or more images of the specific stenosis.
In some embodiments, FFR estimate and/or other functional measurements can be obtained during or after stenting, by using the systems and methods described above, namely, obtaining an image of the patient's vessels during or after stenting and calculating a functional measurement of the vessel based on a color feature of the image at the location of the stent (e.g., at the stents ends and/or within the stent). Thus, in one embodiment, a method for analysis of a vessel during or after stenting, may include receiving an angiogram image of a patient's vessel with a stent, automatically determining a location of the stent in the vessel, e.g., based on image analysis, as described herein, and calculating an FFR value of the vessel (e.g., at the location of the stent) based on a color or grayscale feature of the image, at the location of the stent. An indication of the FFR value may then be output to the user.
Obtaining functional measurements during or after stenting provides information in real-time regarding the success of the stenting procedure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 271294 | Dec 2019 | IL | national |
| Number | Date | Country | |
|---|---|---|---|
| 62945896 | Dec 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/IL2020/051276 | Dec 2020 | US |
| Child | 17836112 | US |
