Patent Application
20240029257
The present disclosure relates to locating a vascular constriction in a temporal sequence of angiographic images. A computer-implemented method, a processing arrangement, a system, and a computer program product, are disclosed.
Vascular constrictions such as thromboses and stenoses limit the supply of blood within the vasculature. Without treatment, vascular constrictions can lead to serious medical consequences. For example, ischemic stroke is caused by a blockage of the vasculature in the brain. Stroke occurs rapidly and requires immediate treatment in order to minimize the amount of damage to the brain.
An important step in determining how to treat a suspected vascular constriction is to identify its location. This is often performed using angiographic images. For example, in the case of a suspected stroke, an initial computed tomography “CT” angiogram may be performed on the brain to try to identify the location of a suspected vascular constriction. However, the small size of vascular constrictions hampers this determination from the CT angiogram. Subsequently, a CT perfusion scan may be performed. A CT perfusion scan indicates regions of the brain that are not receiving sufficient blood flow. However, a CT perfusion scan typically only identifies a section of the brain that may be affected by the vascular constriction, rather than a specific branch of the vasculature. Subsequently, a contrast agent may be injected into the vasculature and a fluoroscopy scan may be performed on the brain to try to identify the stenotic region. The fluoroscopy scan provides a video sequence of angiographic images representing a flow of a contrast agent within the vasculature. However, locating a vascular constriction within the angiographic images is time-consuming. A radiologist may try to identify a region with interrupted flow by repeatedly zooming-in to view individual branches of the vasculature, and then zooming-out again whilst following the course of the vasculature. A small vascular constriction may easily be overlooked in this process, potentially delaying a vital intervention.
Consequently, there is a need for improvements in determining the location of vascular constrictions in angiographic images.
According to one aspect of the present disclosure, a computer-implemented method of locating a vascular constriction in a temporal sequence of angiographic images, is provided. The method includes:
According to another aspect of the present disclosure, a computer implemented method of training a neural network to locate a vascular constriction in a temporal sequence of angiographic images, is provided. The method includes:
Further aspects, features and advantages of the present disclosure will become apparent from the following description of examples, which is made with reference to the accompanying drawings.
Examples of the present disclosure are provided with reference to the following description and the figures. In this description, for the purposes of explanation, numerous specific details of certain examples are set forth. Reference in the specification to “an example”, “an implementation” or similar language means that a feature, structure, or characteristic described in connection with the example is included in at least that one example. It is also to be appreciated that features described in relation to one example may also be used in another example, and that all features are not necessarily duplicated in each example for the sake of brevity. For instance, features described in relation to a computer-implemented method may be implemented in a processing arrangement, and in a system, and in a computer program product, in a corresponding manner.
In the following description, reference is made to computer implemented methods that involve locating a vascular constriction in a temporal sequence of angiographic images. Reference is made to a vascular constriction in the form of a stenosis, i.e. a narrowing or constriction to the body or opening of a vessel conduit. The stenosis may have various origins. By way of an example, the stenosis may be caused by atherosclerosis, which causes fatty deposits and a buildup of plaque in the blood vessels, and may lead to ischemic stroke. By way of another example, the stenosis may be caused by a thrombus, wherein a blood clot develops in a blood vessel and reduces the flow of blood through the vessel. However, it is to be appreciated that the methods disclosed herein are not limited to these examples and may also be used to locate other types of vascular constrictions, and that these may have various underlying causes. Reference is also made to examples of vascular constrictions in the brain. However, it is also to be appreciated that the methods disclosed herein may also be used to locate vascular constrictions in other regions of the body, such as in the heart, the leg, and so forth. Thus, it is to be appreciated that the methods disclosed herein may be used to locate vascular constrictions in general.
It is noted that the computer-implemented methods disclosed herein may be provided as a non-transitory computer-readable storage medium including computer-readable instructions stored thereon which, when executed by at least one processor, cause the at least one processor to perform the method. In other words, the computer-implemented methods may be implemented in a computer program product. The computer program product can be provided by dedicated hardware or hardware capable of running the software in association with appropriate software. When provided by a processor, or “processing arrangement”, the functions of the method features can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which can be shared. The explicit use of the terms “processor” or “controller” should not be interpreted as exclusively referring to hardware capable of running software, and can implicitly include, but is not limited to, digital signal processor “DSP” hardware, read only memory “ROM” for storing software, random access memory “RAM”, a non-volatile storage device, and the like. Furthermore, examples of the present disclosure can take the form of a computer program product accessible from a computer usable storage medium or a computer-readable storage medium, the computer program product providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable storage medium or computer-readable storage medium can be any apparatus that can comprise, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system or device or propagation medium. Examples of computer-readable media include semiconductor or solid-state memories, magnetic tape, removable computer disks, random access memory “RAM”, read only memory “ROM”, rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read only memory “CD-ROM”, optical disk-read/write “CD-R/W”, Blu-Ray™, and DVD.
As may be appreciated, locating a vascular constriction within the angiographic images in
The inventors have determined an improved method of locating a vascular constriction in a temporal sequence of angiographic images. Thereto,
With reference to
With continued reference to
Various techniques are contemplated for computing the differential images in operation S120. In one example, a time period ΔT between the generation of a current image and the generation of an earlier image in the sequence, that is used to compute each differential image, is predetermined. For example, if, as illustrated in
In another example, the earlier image is provided by a mask image, and the same mask image is used to compute each differential image. The mask image is fluoroscopic image that is generated prior to the injection of the contrast agent into the vasculature. In this example, the differential images computed in operation S120 are so-called digital subtraction angiography “DSA” images. In both these examples, image features that are common to the current image and the earlier image, arising for example from bone, are removed from the resulting differential images.
In yet another example a combination of these techniques is used. In this example, the temporal sequence of angiographic images 110 are DSA images, and differential images are computed from the DSA images by subtracting from the image intensity values of a current DSA image, the image intensity values from an earlier DSA image in the sequence. The time period ΔT between the generation of the current DSA image and the generation of the earlier DSA image, that is used to compute each differential image, is again, predetermined. The time period ΔT may be an integer multiple of the period between successive image frames. Here, again each differential image represents a rate of change in image intensity values, in this example the rate of change being between the current DSA image and the earlier DSA image in the sequence.
Various techniques are contemplated for defining the sub-regions 120i,j in operation S120. In one example, the sub-regions 120i,j of the vasculature are defined by dividing the angiographic images in the received temporal sequence into a plurality of predefined sub-regions. This may be achieved by applying a grid of predefined regions to the angiographic images, as illustrated in the upper portion of
In another example, the sub-regions 120i,j of the vasculature are defined by segmenting the vasculature in the angiographic images and defining a plurality of sub-regions that overlap the vasculature. Various techniques for segmenting the vasculature in angiographic images are known from a document by Moccia, S. et al., entitled Blood vessel segmentation algorithms—Review of methods, datasets and evaluation metrics, Computer Methods and Programs in Biomedicine, Volume 158, May 2018, Pages 71-91. In this example, sub-regions may be defined by applying predetermined shapes, such as a rectangle or square, to sections of the vasculature such that the shapes overlap the vasculature. Continuing with this example, branches in the segmented vasculature may be identified, and an amount of contrast agent at positions along an axial length of each branch may be determined. The sub-region may then be represented as a two-dimensional graph indicating the amount contrast agent plotted against the axial length of the branch. In this example, the shape of the vasculature is essentially stretched into a straight line, and the amount of contrast agent along an axial length of each branch determined by computing the amount of contrast agent at positions along e.g. the centerline of the branch.
Returning to
The operation S130 is described further with reference to
Returning to the central portion of
Further redundancy in the temporal sequence of angiographic images used by radiologists to try to identify vascular constrictions, has also been recognized by the inventors, and in some examples, further refinements may be made to the temporal sequences of the subset of the sub-regions 120i,j that are identified in operation S130. These refinements further reduce that amount of data that is inputted into the neural network, and may further reduce the complexity and/or time taken for the neural network to analyses the angiographic images 110 and to determine a location of a vascular constriction.
In one example of these refinements, portions of the temporal sequences are identified. In this example, the operation of identifying S130 temporal sequences of a subset of the sub-regions 120i,j, comprises:
In this example, and with reference to
A further refinement may also be made to the portions of the temporal sequences that are identified in operation S130. This is described with reference to
In this example, the operation of identifying S130 temporal sequences of a subset of the sub-regions, may also include:
With reference to
Returning to the method illustrated in the flowchart of
As also illustrated in
A variety of techniques are contemplated for use in identifying a sub-region that includes the vascular constriction in the operation S150. These may for example include identifying the sub-region on a display. In one example, an image of the sub-region may be provided wherein the location is identified by means of overlaying a shape, as illustrated by the example dashed circle in
In this example, the displayed temporal sequence may be provided on a display for the identified sub-region alone, thereby permitting a user to focus attention on this sub-region, or for all sub-regions. In some examples, the displayed temporal sequence may be provided for the identified sub-region, as well as for all sub-regions. The displayed temporal sequences may correspond in time to one another, the temporal sequence for all sub-regions providing a large field of view and the temporal sequence for the identified sub-region providing a small field of view. This is illustrated in
In yet another example, the neural network 130 is trained to generate a probability score of the sub-region 120i,j of the vasculature including a vascular constriction 140. The neural network 130 may include a regression-type neural network or classification-type network for this purpose. The probability score may be identified in the operation S150. For example, an angiographic image, or a temporal sequence of angiographic images, may be displayed, indicating the probability scores of one or more of the sub-regions. This may for example be indicated with a color-coded frame around the sub-region, or by color-coding the sub-region of the vasculature. For example, regions with a relatively high probability values may be highlighted in red, and regions with relatively low probability values may be highlighted in green. Regions with intermediate probability values may be highlighted in orange.
The use of various types of neural network 130 to provide the functionality described in the above methods, is contemplated. Various types of classification neural network may be used, including a convolutional neural network “CNN”, a recurrent neural network “RNN”, such as for example a Long Short Term Memory “LSTM”, a temporal convolutional network “TCN”, a transformer, a multi-layer perceptron, a decision-tree such as for example random forest, multivariate regression (e.g. logistic regression), or a combination thereof.
In one example the neural network 130 includes a CNN, and the temporal sequences of the subset of the sub-regions 120i,j that are identified from the differential images in operation S130 are stacked, i.e. arranged, in the time dimension, with each frame as a separate channel, and this 3D data input, is inputted into the a CNN. As an alternative to using 2D filters, a CNN could include 3D filters i.e. 3D convolution kernels. Alternatively, these sequences could be initially processed by a CNN and a low-dimensional representation i.e. feature space is inputted in a sequential frame after frame manner into a RNN where each frame forms a directed graph along the temporal sequence. In an RNN the output of a current frame may be dependent on one or more previous frames. The network may include a uni- or bi-directional long short-term memory “LSTM” architecture. It is noted that the directionality of flow in the images is less important that the speed of flow in the images. In order to account for different numbers of image frames in each sequence, shorter sequences may be padded with empty images at the beginning or at the end of the sequence. Alternatively, interpolation may be used to interpolate new image frames in order to equalize the number of frames in all temporal sequences. The frame rate may be included as a parameter of the neural network.
In another example, a fully convolutional neural network may be used to provide neural network 130. Feature maps may be used to compensate for differences in inputted sequence length. In order to increase the accuracy of this neural network, during training, the inputted sequences may be resized by, for instance, by randomly dropping or interpolating frames in the sequence, so that the inputted sequence lengths capture more variance. In this example, during training, feature maps can be learned in a supervised manner by using manually annotated feature sets or features sets extracted using image processing methods such as Scale-Invariant Feature Transform “SIFT”, Speeded Up Robust Features “SURF” as ground truth. Feature maps can also be learned in an unsupervised manner using neural networks such as auto encoders to learn a fixed size feature representation.
In another example, neural network 130 may include a CNN and an RNN. In this example, each frame in the temporal sequences of the subset of the sub-regions 120i,j that are identified from the differential images in operation S130, is inputted into the CNN, and a low-dimensional representation of the frame is generated, for example a 1D feature vector. This feature vector is then inputted into an RNN, such as an LSTM or a Gated Recurrent Unit “GRU” to capture the temporal aspect of the input.
As mentioned above, in some examples the differential images are computed for a current image using an earlier image in the sequence, and wherein the time period ΔT between the generation of the current image and the generation of the earlier image in the sequence, is predetermined, such that each differential image represents a rate of change in image intensity values between the current image and the earlier image in the sequence. A benefit of inputting differential images that represent a rate of change in image intensity values between the current image and the earlier image in the sequence, into the neural network, in contrast to e.g. inputting differential images that represent e.g. DSA image data into the neural network, is that the former provides relatively higher emphasis on image frames representing dynamic changes in contrast agent flow and relatively lower emphasis on image frames representing continuous contrast agent flow, or no flow at all. In other words, it provides relatively higher emphasis on image frames Fr2, Fr3, Fr10 and Fr11 in
In some examples, additional information is inputted into the neural network 130 in the form of a first arrival time for each sub-region of the vasculature. The first arrival time represents a time at which the contrast agent enters the sub-region. In these examples the method of locating a vascular constriction in a temporal sequence of angiographic images, includes:
The first arrival time may be computed as the absolute time of the contrast agent entering each sub-region, or as the time difference with respect to another reference, such as between contrast agent entering a predetermined region of the vasculature, and the contrast agent entering each sub-region. For example, if the angiographic images represent the brain, the first arrival time may be determined as the time difference between contrast agent entering the base of the arterial tree, and the contrast agent entering each sub-region. The times of these events may, for example, be determined by segmenting the vasculature and applying a threshold to the sub-regions in order to detect the contrast agent. The first arrival time may be represented in a displayed temporal sequences of angiographic images of the vasculature, for example by color-coding the first arrival time in the displayed temporal sequence.
In some examples, additional information is inputted into the neural network in the form of corresponding image data from other imaging modalities. For example perfusion CT image data, CT angiography image data, or image data from other imaging modalities may be registered to the received temporal sequence of angiographic images 110 representing a flow of a contrast agent within a vasculature, and inputted into the neural network.
Additional information that is inputted into the neural network 130 in this manner, may assist the neural network in classifying a sub-region 120i,j of the vasculature as including a vascular constriction 140.
The angiographic image training data may for example be provided by fluoroscopic or DSA imaging datasets. The training data may be provided in the form of differential images as described above. The training data may originate from one or more clinical sites. The training data may be collected from one or more subjects. The sub-regions of the angiographic image training data may be defined using the techniques described above so as to provide temporal sequences of the sub-regions 120i,j which are inputted into the neural network in the operation S220. The ground truth classification for the sub-regions may be provided by an expert annotating the angiographic image training data with the location of any stenosis, or alternatively the absence of any stenosis.
In the training method described with reference to
Various methods are known for solving the loss minimization problem such as gradient descent, Quasi-Newton methods, and so forth. Various algorithms have been developed to implement these methods and their variants including but not limited to Stochastic Gradient Descent “SGD”, batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg Marquardt, Momentum, Adam, Nadam, Adagrad, Adadelta, RMSProp, and Adamax “optimizers” These algorithms compute the derivative of the loss function with respect to the model parameters using the chain rule. This process is called backpropagation since derivatives are computed starting at the last layer or output layer, moving toward the first layer or input layer. These derivatives inform the algorithm how the model parameters must be adjusted in order to minimize the error function. That is, adjustments to model parameters are made starting from the output layer and working backwards in the network until the input layer is reached. In a first training iteration, the initial weights and biases are often randomized. The neural network then predicts the output data, which is likewise, random. Backpropagation is then used to adjust the weights and the biases. The training process is performed iteratively by making adjustments to the weights and biases in each iteration. Training is terminated when the error, or difference between the predicted output data and the expected output data, is within an acceptable range for the training data, or for some validation data. Subsequently the neural network may be deployed, and the trained neural network makes predictions on new input data using the trained values of its parameters. If the training process was successful, the trained neural network accurately predicts the expected output data from the new input data.
Training a neural network typically involves inputting a large training dataset into the neural network, and iteratively adjusting the neural network parameters until the trained neural network provides an accurate output. Training is often performed using a Graphics Processing Unit “GPU” or a dedicated neural processor such as a Neural Processing Unit “NPU” or a Tensor Processing Unit “TPU”. Training therefore typically employs a centralized approach wherein cloud-based or mainframe-based neural processors are used to train a neural network. Following its training with the training dataset, the trained neural network may be deployed to a device for analyzing new input data; a process termed “inference”. The processing requirements during inference are significantly less than those required during training, allowing the neural network to be deployed to a variety of systems such as laptop computers, tablets, mobile phones and so forth. Inference may for example be performed by a Central Processing Unit “CPU”, a GPU, an NPU, a TPU, on a server, or in the cloud.
The above examples are to be understood as illustrative of the present disclosure and not restrictive. Further examples are also contemplated. For instance, the examples described in relation to the computer-implemented method, may also be provided by a computer program product, or by a computer-readable storage medium, or by a processing arrangement, or by a system, in a corresponding manner. It is to be understood that a feature described in relation to any one example may be used alone, or in combination with other described features, and may also be used in combination with one or more features of another of the examples, or a combination of other examples. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims. In the claims, the word “comprising” does not exclude other elements or operations, and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage. Any reference signs in the claims should not be construed as limiting their scope.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP21/85820 | 12/15/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63129292 | Dec 2020 | US |
