Patent Application
20240000413
The invention relates to a system for supporting an intervention, to a related method, to a system for training a machine learning module, to a related machine learning training method, to a computer program element, and to a computer readable medium.
Balloon angioplasty, also known as percutaneous transluminal angioplasty (PTA), is one of the most frequently used procedures in managing atherosclerotic plaque. Atherosclerosis of peripheral arteries causes peripheral artery disease, which affects approximately 8 million people in the USA alone. Typically, a balloon catheter is passed over a guide wire to the site of the lesion and is then inflated using a (mix of) contrast substance/agent and/or other fluid that is pumped into a balloon portion of the balloon catheter to cause an increase in internal pressure in the balloon portion. The inflating balloon portion can thus force an existing stenosis in a blood vessel wider and/or open. Balloon angioplasty is also commonly used in conjunction with other therapies (either before or after) such as stenting and laser atherectomy, and other.
The rate of restenosis following therapy is relatively high in the months and years following treatment leading to 20% of patients receiving repeat procedures. Studies have shown that restenosis is associated with excessive repair action of for example an artery due to mechanical damage caused during the treatment—for example, by overstretching of the arterial wall. Simulations on this topic have shown that damage is higher to certain regions of the lesion—specifically those that are not protected by a layer of calcification. The same study found that softening plaque (for example, via ballooning) could lead to lower peak stress on the vessel wall, and thus to less damage.
Recently, drug coated balloons have shown promising results in reducing restenosis and hence need for re-interventions. These balloon catheters include a coating of a drug, for example Paclitaxel, that acts to inhibit arterial smooth muscle cell proliferation in the repair action. Usage instructions (IFU) for some drug coated balloon catheters indicate that for proper drug delivery to the target lesion, inflation the balloon should be maintained for a certain period of time, such as a for a minimum of 60 seconds for example. There is some evidence that prolonged inflation times during plain balloon angioplasty can have improved outcomes.
During inflation of the balloon, there are certain adverse events that can happen. It is advantageous to prevent adverse events since they can impact the patient and procedure, but it may also be beneficial to log those events and report the data to a regulator for example, such as the FDA (Food and Drug Administration), a US federal regulatory agency. In addition, there is a desire from manufacturers to have appropriate data collected around those events. Such adverse events include balloon rupture, twisting or fold-over that prevents inflation or deflation of the balloon portion.
Currently, the medical user is responsible for integrating/interpreting fluoroscopy image feedback together with the pressure information received from the balloon catheter's inflation gauge.
There may therefore be a need for improved support for, in particular, medical interventions.
An object of the present invention is achieved by the subject matter of the independent claims where further embodiments are incorporated in the dependent claims. It should be noted that the following described aspect of the invention equally applies to the said related method, to the system for training a machine learning module, to the said related training method, to the computer program element, and to the computer readable medium.
According to a first aspect of the invention there is provided a system for supporting a balloon catheter procedure, comprising:
The input data may be received as part of one more data streams, such as in video feed (and/or a pressure measurements feed. Video feeds may be generated in certain imaging modalities, such as in fluoroscopy or others modalities. However, processing of data streams is not a requirement herein.
In one embodiment, at least one of the input data i), ii) is obtained by a sampler of the system sampling a respective original data stream, wherein a sampling frequency of the said sampler is variable. The sampling frequency may vary. In particular, the sampling frequency may depend on an earlier prediction result, in particular when the input data is received as part of such as one or more data streams as mentioned earlier.
In one embodiment, the system comprises a reporting module configured to report the prediction result numerically and/or graphically.
In one embodiment, the reporting module includes a visualizer component configure to produce a graphics display for displaying the prediction result on a display device.
In one embodiment, the graphics display further includes at least some of the image data and/or at least some of the pressure readings.
In one embodiment, the reporting module is operable to annotate at least some of the received image data with said prediction result. The annotation may be fully automatic or may rely at least in parts on user input, as through a suitable user interface (keyboard, touchscreen, pointer tool) etc.
In one embodiment, wherein, based on the prediction result, at least some of the image data and/or some of the pressure readings are stored in a memory.
In one embodiment, the machine learning module includes a trained machine learning model of the neural network type.
In one embodiment, the machine learning model includes at least one convolutional layer and/or at least one recurrent layer.
In one embodiment, the event is a rupture or twisting of a balloon portion of the balloon catheter, and/or stress on the vessel, caused by an inflation of the balloon.
In another aspect there is provided a training system configured to train, based on training data, the said machine learning module of the system as per any one of the previous embodiments.
In another aspect there is provided a method of supporting a balloon catheter procedure, comprising:
In yet another aspect there is provided a method of training, based on training data, the said machine learning module of the system as per any one of the previous embodiments mentioned above.
Using machine learning technology to process a combination of those two inputs (imagery and pressure reading(s)) allows capturing interaction between the balloon and the anatomy that can be used to predict stress on the vessel and adverse events. Machine-learning processing of both data feeds, imagery and pressure readings, allows reliable and robust prediction, and thus possibly prevention of adverse events such as malfunctioning of the medical device, such as of a balloon catheter. Adverse events may include balloon rupture, fold-over or twisting that prevents inflation or deflation. It has been found that using either data feed on its own is less desirable. For example, calcifications in the vessel walls may include sharp crystal formation that may cause rupture even for modest pressure readings. Also, imagery on its own may not capture “the whole picture” as it were. The balloon inflation may look “good” in the image feed, but in fact the internal pressure within the balloon is about to exceed its rating, etc. The event of interest may also be formulated as a predefined maximally allowed level of mechanical stress exerted by the balloon on a certain section of the vessel's wall.
The system also allows reliably logging/reporting the evens if they do indeed occur, thus providing valuable feedback to manufacturers, regulators or other parties.
Whilst processing of a single image or pressure reading datum may be sufficient in some instances, processing two data streams of image frames and/or pressure readings, may be preferred in embodiments to improve performance.
In another aspect there is provided a computer program element, which, when being executed by at least one processing unit, is adapted to cause the processing unit to perform the method as per any one of the above mentioned embodiments.
In another aspect still, there is provided a computer readable medium having stored thereon the program element.
The proposed system method, arrangement computer program element and computer readable medium can be used with mobile or fixed interventional x-ray systems. It can used for substantially all balloon catheter types, including specific relevance drug-coated angioplasty balloon catheters, scoring balloon catheters, bridge occlusion balloon catheters and others.
“User” relates to a person, such as medical personnel or other, operating the imaging apparatus and/or overseeing the imaging procedure and/or medical intervention. In other words, the user as understood herein is in general not the patient.
“Image data” as used herein includes raw or processed imagery as provided by the imaging apparatus. However, “image data” also includes parts of the original imagery, such as information extracted from the imagery via segmentation for example. The format of the image data may be matricial such as 2D, 3D or 4D (with one dimension for time as in video feeds for 2D or 3D native image data), but may also include instead or in addition other suitable functional or parametric descriptions such as segmentations. The suitable functional or parametric descriptions may include data structures that indicate or refer to shape, locale, extent etc., to so define a segmentation in image domain. A bit mask may be used for example.
In general, “machine learning” includes a computerized arrangement/module, that implements a machine learning (“ML”) algorithm. In model-based machine learning, a machine learning model is adapted to perform a task. This adaption is called “training. Task performance by the ML model or ML module improves measurably, the more (new) training data is used to train the model. The performance may be measured by objective tests applied to output generated by ML system when processing test data. The performance may be defined in terms of a certain error rate to be achieved for the given test data. See for example, T. M Mitchell, “Machine Learning”, page 2, section 1.1, McGraw-Hill, 1997.
Exemplary embodiments of the invention will now be described with reference to the following drawings, which, unless stated otherwise, are not to scale, wherein:
With reference to
The intervention is image-guided in that it is performed during operation of an imaging apparatus IA. The imaging apparatus IA is preferably an x-ray imaging apparatus including an x-ray source XS and a detector D, but other imaging modalities such as MRI, ultrasound or optical OCT and others still are also envisaged herein. The imaging apparatus mainly envisaged herein generates projection imagery.
As will be explained more fully below, the system SYS is arranged on one or more computing systems to processes data generated during the intervention. More particularly, imagery I generated by the imaging apparatus IA and pressure readings P acquired by an in-situ sensor S (to be discussed in more detail below) are processed together by the system SYS to predict one or more of i) a medical outcome in relation to the patient undergoing the intervention and ii) certain events of interest, such as adverse events. Such events may occur in relation to the patient and/or the medical equipment used in the intervention.
The medical equipment mainly envisaged herein may include a balloon catheter BC. Such catheters BC may be used in certain interventions such as angioplasty interventions. Such angioplasty interventions may be used to treat a stenosis. One such stenosis type may present in a mammal's (human or animal) vessel in the veinous or arterial system. Cardiac stenosis in the coronaries of the human heart is one example mainly envisaged herein, but other applications are not excluded. Stenosis may cause a range of medical complications due to undersupply of oxygen caused by poor blood circulation.
Broadly, and with particular reference to cardiac stenosis, this may be treated by advancing a distal portion of the balloon catheter through the patient to the lesioned site such as the strictured site ST, a portion of a lesioned blood vessel. A guide wire GW may be used to navigate and push the cathether's distal treatment portion to the site ST. A range of different types of balloon catheters are envisaged herein including those of the over-the-wire (OTW) type or rapid exchange (RX) type, or others still. In general, balloon catheters BC includes a tube with the said distal end, and a proximal end. The proximal end remains outside the patient with a user, such as an interventional cardiologist, to control the operation of the catheter. The distal portion is introduced into the patient through an entry point EP such as at the femoral artery or vein and is advanced through the vasculature to the lesioned site ST. The distal portion of the catheter tube in balloon catheters is formed into a balloon portion FP that is inflatable with liquid such as water or preferably a saline solution. The liquid is administered through a port portion at the proximal end to inflate the balloon portion FP. Upon removal of some or all of the liquid, the balloon deflates. The balloon portion FP is formed from an elastic material such as an elastomer with suitable elasticity properties. The balloon portion FP is advanced by use of the guide wire via the entry point EP to the lesioned site and during operation of the image apparatus IA to assist navigation through the vasculature. Upon arrival at the lesioned site ST, the balloon portion FP may be inflated by introduction of the liquid into the balloon FP, thus causing a force on the vessel VL walls to be exerted to so broaden the stricture thus treating same. Treatment of the stenosis as described above is mainly envisaged herein, however, so are other treatment options such as the placement of a stent for example. Whilst cardiological interventions are mainly envisaged herein, this is not to exclude other medical interventions that use balloon catheters CB such as interventions in relation to the urinary system in urological applications for instance.
The inflation of inflatable balloon portion FP causes an increase in pressure, which can be measured by a pressure sensor S integrated into the balloon catheter BC, preferably in the balloon portion FP of the catheter BC or into an inflation unit of the balloon catheter. However, the pressure sensor S may be attached at any point along the pressure inducing pathway. For example, in some embodiments, the sensor S is attached/integrated at or in an injector/inflator pump that is operable for pressurized supply of the fluid to balloon portion FP. A processing unit (not shown) of the balloon catheter arrangement BC captures and digitizes the pressure readings P. The pressure readings may be supplemented by additional information, such a volume V readings etc. In the following we will not refer specifically to such additional information and any reference below to “pressure readings” should be construed to optionally include such additional information. Equally, the pressure readings may be supplied direct in terms of pressure quantities (such as PSI, bar etc) but may be provided instead in terms of surrogate quantities, such volume, temperature etc that directly correlate with balloon portion pressure or that can be converted to such pressure quantities. As such, the processing by the proposed system SYS may well proceed on such surrogate quantities such as volume etc, and conversion to pressure quantities is not necessarily required herein. Although we will refer to herein to “pressure readings”, any such reference should be construed herein to include reference to such surrogate quantities.
Conventionally, the treatment of the stenosis ST includes multiple phases, including positioning the balloon portion FP close to the stenosis, exerting a certain push by the user to force the balloon portion FP at least partly through or past the stenosis, applying the liquid to cause the balloon portion to inflate. The inflation effects the broadening of the stenosis and/or effects the delivery of stent, or effects any other treatment to be performed at the lesion site ST. The phases are usually performed by the user under image-guidance as mentioned. The image apparatus IA produces a life feed of x-ray images I(“frames”), at a suitable frame rate such as 60 fps for example (other frame rates may also be used). The video feed may be displayed on a display device DD as a motion picture. Although capture and display of still imagery is not excluded herein, it is in particular the video feed that allows the user to ascertain the correct unfolding of the treatment procedure, such as the positioning of the catheter, its inflation, etc. Specifically, the phases of placement, push-through and deflation and/or inflation cause interaction of the balloon catheter with the vessel VL, in particular with its wall, in the course of which the balloon portion FP assumes a range of different geometrical configurations which can be ascertained through the acquired imagery. A good number of interventions run smoothly but sometimes certain adverse effects may occur such as a rupture of the balloon portion FP (“rupture event”). This can have many causes, such as unfavorable balloon-vessel interaction or unfavorable properties of the vessel. Specifically, balloon rupture event can occur due to defects in the balloon, over-inflation of the balloon beyond the rated burst pressure (RBP). Lesion morphology such as sharp calcifications can also cause balloon rupture. In most cases balloon rupture does not cause serious complications. However, at time, it can lead to air embolism, vessel dissection/perforation, balloon fragments detached into the body, and longer procedure times. Other adverse events include twists, kinks or folds in the balloon portion FP that prevents (full) inflation to the desired extent. Such adverse events such as rupture, twisting, etc are adverse as they prevent a successful conclusion of the intervention. For example, such events may result in failure to broaden the stenosis, either not at all or not sufficiently so, or that the drug could not be deposited at the site of the lesion, etc.
In addition to the imagery provided by the apparatus, there is, as mentioned briefly above, another data feed, namely a time series of the pressure readings P acquired by the sensor S. It has been found that by observing the pressure readings in conjunction with the different geometrical configurations of the balloon portions FP as captured be the video feed imagery I, a reliable prediction can be made as to whether or not an adverse effect are imminent. The adverse effect may include the said rupture event or other event, such as a twist or fold-over events, that compromise deflation and/inflation of balloon portion FP. Other malfunction events that may compromise the intervention as also envisaged herein.
Whilst the imagery I provides clues on the geometrical configuration, the pressure readings provide an additional information layer in relation to the balloon-portion-versus-vessel interaction. The two data feeds, the imagery encoding geometrical information plus the pressure readings, may be used together to predict the outcome of the treatment procedure. The outcome may include, for instance, whether or not the patient may experience re-stenosis and thus may require repeated interventions. This may be expressed as a time-defined patency, such as patency at m months, for example m=12. Other outcomes may be related to aortic insufficiency, need for follow-up surgery, even death, especially for balloon valvuloplasty for calcific aortic stenosis.
The computerized system SYS is configured to jointly process the double-data feed, the imagery I and the pressure readings P, to compute those predictions on adverse events (“E”) and/or outcome (“O”). Preferably, the system SYS uses a machine learning module or component MLM, previously trained on suitable training data. In the course of the training, and in some embodiments, a machine learning model M of the module MLM is adapted based on the training data to configure the machine learning component. Whilst there is a correlation between such adverse events/outcome and the double-feed information encoded jointly in the imagery and the pressure reading, it has been shown to be difficult to analytically and/or explicitly model the exact relationships between those events/outcomes given the imagery and pressure readings. Such analytical modelling may also suffer from low efficiency, overfitting and poor performance. Machine learning (“ML”) approaches are beneficial in that they generalize well from the training data as they do not require an explicit modeling of the said correlation. Provided a sufficiently large set of training data is available. ML has been found to provide a well-generalized approximation of the latent relationship between outcomes/events and the double-data-feed of catheter imagery I and pressure readings P.
Referring now briefly to the imaging apparatus IA, this preferably allows acquiring imagery from different projection directions d. The imaging apparatus may include an optional gantry GT to which the x-ray source XS and the detector D are connected. The projection imagery along the different projection directions may be acquired by rotation of the gantry around the lesioned site ST, and with it the source XS and detector D. Such gantry-based imaging apparatuses include C- or U-arm systems and are mainly envisaged herein, but so are (CT) computed tomography scanners. Non-gantry based imaging solutions are also envisaged, such as mobile or portable imaging devices, where there is no, or no permanent, physical connection between detector D and radiation source XS. The imagery I processed by the system SYS may be projection imagery in projection domain as recorded by the detector D, or may comprise reconstructed imagery in image domain obtained by a computed tomography algorithm.
During imaging, the x-ray source XS emits an x-ray beam which propagates along a projection direction d to interact with patient tissue and/or the balloon catheter to cause a modified x-ray beam to emerge at the far end of the patient, and be detected at detector D. Data acquisition circuitry (not shown) of the detector D converts the received modified radiation into a set of numbers (“pixel values”), preferably stored in a respective matrix per frame, with respective rows and columns. The pixel values represent detected intensities. The pixel values per frame can be used by a visualization component VC to effect display of the image on a display device DD during the intervention.
As mentioned, preferably a series of such images is generated to cause a video feed to be displayed on the monitor device DD. Real time imaging can thus be provided, assisting in real time monitoring of the intervention. However, the provision of a series of such images is not necessarily required in all embodiments, and a single such image frame captured in conjunction with a single or more pressure readings may suffice in some circumstances for computing the prediction. However, the processing by the machine learning module MLM of plural frames and/or plural pressure readings may be more reliable and/or more robust, as this may allow the ML module MLM to take into account trends when computing the prediction.
Referring now to
Broadly, and with continued reference to
Optionally, an image pre-processor PP may be used to process the received imagery to extract, by segmentation or other techniques, relevant image structures such as the balloon catheter footprint BCF and/or the vessel tree etc. Whilst the whole feed of images or pressure readings can be processed by the system, in order to boost responsiveness and/or to save computation resources, it may be beneficial to process merely a sub-selection of the pressure readings and/or frames I. To this effect, an optional sampler SP may be used that samples the image feed and/or pressure reading feed to produce by sub-selection a sub-set of the originally received feeds I,P at input port(s) IN, and it is this subset that is then processed by the system SYS to compute the prediction result R.
Various time- or event-driven sampling schemes may be implemented by the sampler SP. For example, the sampler SP may sample more densely the more recent images and/or pressure readings, and decrease the sampling frequency over time. Preferably, it is ensured that an initial image from the beginning of the balloon inflation phase is always included. A number of different events could be used for indicating the beginning of the balloon inflation phase. For example, the said beginning could be taken as the time (T) at which inflation volume is less than a certain volume threshold, such as less than 5 mL. Alternatively, the said time T is one where pressure is at a certain threshold, such as at 0 atm. Alternatively still, the said time T may be image-defined, for example as the time T at which no balloon footprint is visible in the image feed.
With continued reference to variable sampling frequency schemes, the sampling may proceed evenly over a certain period of time. More dense sampling (higher sampling frequency) may be used to sample most recent time points and then decrease the sampling frequency over time. As a further variant, the sampling is done only when there is a minimum difference between subsequent images and/or corresponding pressure readings. The various sampling schemes and definition of start time T may be used for sampling the imagery and/or the pressure readings. Preferably the same sampling schemes may be use for both, the image channel and the pressure reading channel, although the schemes may differ. For example, if there are no changes in channel, sampling may pause whilst sampling in the other channel may be ongoing. To maintain pairing or synchrony between the two channels, copies of a constant image or pressure reading may be used for pairing up with values in the channel that is being sampled.
In the following, unless specific reference is made to the sampler SP, we shall not distinguish whether the whole or a sampled portion of the imagery or pressure readings are processed.
The images I and the pressure readings P are processed by the machine learning module MLM. As briefly mentioned above, the machine learning module MLM may include in embodiments a pre-trained machine learning model M. The machine learning model M is used to process the input data, that is, the image(s) and pressure reading(s) I,P. We shall refer in the following to the images and the pressure reading feeds as the main input or simply the input data and denote this as “(I,P)”.
The machine learning model M includes a set of parameters which can be adjusted by a machine learning algorithm based on the training data. The machine learning model is hence operable in two modes in training mode and in deployment mode, as will be explained in more detail below at
Whilst model-based ML techniques are mainly envisaged herein for machine learning from the input data (I,P), such model-based approaches are not a necessity herein, and non-model based machine learning techniques/algorithms are specifically envisaged herein in alternative embodiments.
In deployment, the pre-trained machine learning model M processes the input data (I,P) and produces the desired result R. The result R may includes data E indicative of whether an adverse effect such as a balloon rupture is imminent. In addition or instead, the indication provided by the result R may be one of a medical outcome O to be expected in relation to the patient and/or the performed intervention at the given site ST.
A communication network CN may be used to forward in particular the results R to a storage MEM-R or to the (or other) display device DD for display. The results may also be forwarded through the communication network to any other processing component as desired.
The computed result R may be suitably processed by a reporting module RM. Such processing may include graphical or textual rendering, or a combination thereof. To this end, the reporting module RM may include the visualizer component VC which allows displaying the predicted data R on the display device DD. The original input data (I,P) may be displayed in full or in parts alongside the result R. The reporting module RM may be operable to store all or parts of the computed results R in the result storage MEM-R. The reporting module RM may be operable in an alert mode where, once the predicted results indicate that an adverse event is imminent, the acquired data (I,P) is auto-stored in a data storage such as the result storage MEM-R or other data memory.
An annotation tool AT may be used to annotate at least a portion of currently acquired imagery. Whether or not the annotation is applied is a function of the predicted result R.
In general, the input data (I,P), sampled or not, comprises corresponding pairs of images It, and pressure readings Pt. Data components It and Pt can be said to be paired because they represent imagery and pressure readings, both at the same respective, given time t. The above mentioned time stamping may be used to establish sets of pairs. The described pairing via time-stamping or otherwise is optional and the proposed machine learning MLM may also be able to process unpaired readings (I,P), such as when there is too much noise in the data. However, the time-based pairing of the input data feeds is preferred as this may accelerate the learning.
Reference is now made to
Reference is now made to the flow chart in
In the following, when referring to the method steps in
Turning now in more detail to the method, at step S710 the input data (I,P) is received, not necessarily simultaneously. The input data comprises one or more images acquired by the imaging apparatus during the intervention using a medical device such as balloon catheter, and one or more pressure readings acquired by a pressure sensor. The sensor is advantageously in-situ, such as integrated into the interventional device, such as in the balloon catheter. The sensor may be integrated in the balloon portion of the balloon catheter, to measure the pressure inside the balloon as this is inflated or deflated and/or whilst the balloon interacts with the vessel walls. The balloon may be deformed such as twisted and kinked during the inflation or deflation. The balloon portion may fold-over. Such interactions may thus cause higher than expected pressure readings or larger pressure changes such as pressure spikes or drops. The sensor may supply the readings through a wired or wireless connection to one or more data processing units PU, such as computing devices with one or more suitable processors that are configured to implement the method at least in parts.
Preferably, the input data are paired so that the balloon portion FP configuration as captured in a given frame at a given time instant corresponds to the pressure reading obtained at substantially the same time instant by the pressure sensor S. The frames I and pressure readings P are either natively paired or may be processed, for example by time-stamping matching, in an optional step to arrange the input data into paired form. However, having the input data paired is not a necessity herein, but may be beneficial as mentioned earlier.
In an optional step S720 the feed of frames I and/or the pressure reading feed P in the input data (I,P) are sampled according to a sample scheme as described above.
The so sampled input data (I,P) or the whole of the input data (I,P) is then processed by the machine learning module at step S730 to predict the result R.
The result R may be a classification or regression result. In particular, the input data at step S730 is classified as indicative of an adverse event in relation to the patient and/or the interventional device, such as the balloon catheter. In particular, the event may describe the likelihood/probability for a balloon rupture to occur. In addition or instead, the result R is classified or regressed into an indication of a medical outcome in relation to the patient undergoing the procedure as captured by the pressure readings and the images received in the input data (I,P) at step S710.
At step S740 the prediction result(s) R is/are output.
The outputting step S740 may include a reporting step in which the predicted result is transformed into suitable format such as textual, number or graphical, or into a hybrid format including any two or more of text, numbers and graphics. The raw results or the result in reportable form may be forwarded through a communication channel to a recipient, such as a communication endpoint, a user device such as a mobile phone, mobile computer, a desktop computer or any other computing device.
At an optional step S750 the input data (I,P) is stored at least in parts in a data storage as a function of the prediction result R. For example, and mainly envisaged herein in embodiments, if the result R indicates that an adverse event, such as the balloon rupture event, is imminent, an auto-save or storing function is initiated that causes the currently received imagery and/or pressure readings to be stored. Such storing operation continues for a pre-defined and/or user defined period of time after the result indicative for such an adverse event has been computed. This allows capturing data that may then be used by manufacturers or other interested parties to precisely understand and analyze the malfunction situation, if indeed a rupture did occur. Specifically, the triggering of the auto-saving of interventional x-ray and pressure data may optionally depend on the computed likelihoods. The auto-save operation may be triggered when an adverse event likelihood exceeds a threshold such as p=10%, or any other such threshold as may be defined by the user.
The reporting step in particular may also include annotating the currently received imagery and/or pressure readings, as function of the computed result, as has been described above in relation to
At step S760 it is then checked whether new input data is received at one or more input ports, and, if yes, the method then repeats at step S710 and proceeds as described above. If no more data is received the procedure terminates. The method may operate at suitable processing frequency per single frame and/or single pressure reading datum. Alternatively and preferably, as mentioned above, in particular in order to account for trends or for saving computational resources, the method may operate in blocks. Once a block of a predefined number or plural frames and/or plural pressure readings are received, the machine learning based computing of the result at step S730 is based on processing the blocks of frames and/or pressure readings at once, jointly. The block size for the processable frames or pressure readings may be the same (such as in paired data) or may differ. In some embodiments, a single frame is processed with a block of plural pressure readings, or a single pressure reading may be processed with a block of plural frames. Preferably however, a plural block of frames is processed together with a block of pressure readings of the same or different size.
As indicate above, the following will provide more details on the machine learning implementation of the above described method and/or computerized system SYS. Broadly,
Turning now first to
The model M may be trained by a computerized training systems TS to be described more fully below at
The machine learning model M may be stored in one (or more) computer memories MEM. The pre-trained model M may be deployed as a machine learning component MLM that may be run on a computing device PU, such as a desktop computer, a workstation, a laptop, etc or plural such devices in a distributed computing architecture. Preferably, to achieve good throughput, the computing device PU includes one or more processors (CPU) that support parallel computing, such as those of multi-core design. In embodiments, GPU(s) (graphical processing units) or one or more TPUs (tensor processing unit) are used.
Referring now in more detail to
In training, the input data x=(I′,P′) is applied to input layer IL. The input data x then propagates through a sequence of hidden layers Li, RLj(there may be more or less than shown in
The model network M may be said to have a deep architecture because it has more than one hidden layer. In a feed-forward network, the “depth” is the number of hidden layers between input layer IL and output layer OL, whilst in recurrent networks the depth is the number of hidden layers, times the number of passes. In recurrent layers, at least part of their output is fed back to earlier layers, such as the immediately preceding layer. No such feeding back occurs in feedforward type layers.
The layers of the network, and indeed the input (I′,P′) and output R′, and the intermediate input and output between hidden layers (referred to herein as feature maps), can be represented as vectors, or as two or higher dimensional matrices (“tensors”) for computational and memory allocation efficiency. The dimension and/or the number of entries represent the above mentioned size.
The input layer IL of the neural network model M is suitably dimensioned to receive, in training phase, the training input data, referred to herein as (I′, P′). The processing may proceed frame by per frame and/or per individual pressure reading datum P′t. Alternatively, the processing is in blocks of frames or pressure readings, each blocks comprising plural frames or pressure readings. A reference to block-wise processing as described above is understood to include the degenerated case of blocks of length of length 1. If required, we will refer to blocks of length 1 or greater than 1, if required and to emphasize that more than one frame or pressure reading datum is processed at once.
The frames of pressure reading blocks are preferably applied simultaneously to the input layer OL, but sequential processing is not excluded herein. The block of pressure readings may be represented as an image volume, with one dimension indicating time, each time coordinate referring to a specific frame. The block of pressure readings may be represented as a vector of suitable length defining the pressure reading block length, as shown in
The processing of the training input data (I′,P′) may be implemented by processing the pressure readings together with the imagery, or each block may be separately processed in different sequences of layers. Intermediate results of the pressure reading processing layers and the image processing layers may then be merged and processed together in one or more subsequent layers.
The embodiment in
As mentioned earlier, the model M may include one or more hidden layers. As shown in
Upstream or downstream the sequence of feedforward convolutional layers, there are one or more optional recurrent layers RLj, example, j=1 . . . N. The recurrent layers may be convolutional or fully connected, or may include a mix of convolutional and fully connected layers.
Recurrent type layers have been found to be beneficial when processing time dependent data, such as the pressure readings and/or the image frames in the video feed. Convolutional layers have been found to be beneficial in processing image type data. If the data is not image-like, that is, there are no spatial correlations, such as in the vector of pressure readings, one or more fully connected layers of the recurrent or feedforward type may be used. Preferably, the layers are of the recurrent type.
With more detailed reference to the embodiment shown in
An output layer OL processes the received feature maps from the previous one or more layers (such as recurrent layers RLj) into a training estimate for the output result, denoted herein as M(I,P)=R′. That is, the ML model M with parameters θ are applied to training data (I′,P′), imagery and pressure readings from the training data, to yield training result R′. The output layer OL may implement a regression or classification as desired. In embodiments, the output result R′ represents the probability P(E) of the adverse event in relation to the balloon catheter for the observed input data I′,P′. Alternatively, or in addition, the result R′ may represent a probability P(O) of a medical outcome in relation to the treated patient.
Once trained, the parameters θ have been suitably adjusted and are now fixed as θ′. In the described neural network model, the adjusted parameters θ′ include in particular the filter elements of layers RLi,Lj. The processing by the trained network Mθ′ of “real”, unseen-before input data (I,P) during deployment in clinical practice, is exactly as has been described above in relation to
In some embodiments, output layer OL of model M is configured for classification, such as multi-label or binary classification. For example, the result R,R′ of interest is whether or not a rupture event or other adverse effect is to occur. In this embodiment, the output R,R′ is classified into a vector with two entries indicating whether or not there is such an event. As a further variant instead of such a binary output, a probability distribution over more than two entries may be computed where the entries indicate whether or not there is a probability for a particular event. In an extreme case only a single number is provided at the output result, the output being an indication for an event that attracted the highest probability. It is in particular in such classification setups that the model M may include, in addition to the one or more convolutional layers IL, L1-LN, one or more fully connected layers, of the feedforward or recurrent type. In case of a classification result, the output layer OL may be configured as a softmax-function layer or as similar computational nodes where feature maps from previous layer(s) are combined into normalized counts to represent the classification probability per class.
Whilst classification is mainly envisaged herein, regression is not excluded and indeed envisaged in embodiments, in particular in cases where the pressure sensor S is arranged to measure the pressure distribution across the balloon. For example, a pressure sensor patch may be used to infer a pressure distribution. A FORS (Fiber optic real shape) fiber or any other type of fiber optical shape sensing may be used. In such embodiments the pressure reading Pt is a 2D or 3D array rather than a scalar. The output layer OL may thus operate to regress the pressure distribution into probability maps rather than a single output value. The probability maps may be displayed by visualizer VC on display device DD. The results R on adverse events may hence be localized to indicate where the balloon portion FP may rupture, twist or unfold, rather than merely providing a single probability value of the event as described above for a pure classification task. Similarly, a medical outcome may be localized to indicate where the restenosis may originate.
As mentioned earlier, the preprocessor PP may segment the imagery for regions of interest. This option may be used to limit the dimensionality of the input space. The segmentations for the balloon or vessel tree may be provided as additional input to the machine learning component. For example, the segmentations may represent the earlier mentioned additional contextual parametric descriptions for both training and inference, instead of (or in addition to) providing the raw images I. If the segmentations or other image pre-processing data is used as additional input, these may be processed by the model M as additional image channels, the true image data and the additional input forming image volumes, with one or more dimensions used to define the channels, similar to the manner which the pressure readings may be processed as described above after transformation into pseudo-imagery. Rather than using the data by the preprocessor as additional information, the preprocessed image data may be processed by the machine learning component instead of the raw image data.
It will be understood that the above described model M in
Reference is now made to
The training system TS may be used for adjusting training the parameters, for example the weights of a neural network-type model as discussed in
In general, the training is based on training data. Training data comprises k pairs of data items (xk, yk). index k may run into the 100s or 1000s. The training data comprises for each pair k, training input data xk and an associated target yk. For present purposes, xk=(Ik′,Pk′) and yk=Rk′, with Ik′ indicating test imagery (one or more images) for the k-th pair, and P′k indicates the associated pressure reading block. The training result or label Rk′ indicates an adverse event or medical outcome, as desired. The training data may thus be organized in k pairs in particular for supervised learning schemes as mainly envisaged herein. However, it should be noted that non-supervised learning schemes are not excluded herein.
Some or each training data pair k may be stored as a 3-tuple:
d
k=(xk,yk)=(I′tk,P′tk,R′tk) (1)
Given such pair, random batches of training instances may defined.
In (1) and for some angioplasty embodiments, I′ti represents the sampled time-series 2D fluoroscopy projection acquired during the interventional procedure during balloon inflation. P′ti represents the sampled time-series pressure measurement acquired during the interventional procedure during balloon inflation. R′i represents the trial or registry data that is recorded for that patient including. The trial or registry data may encode the adverse event, such as balloon rupture, balloon folding. The result R′i may represent instead or addition, patency at m months post-procedure (eg, m=6, 12, 18 or other) or freedom from device and procedure-related death, or any other. If the model M is to be trained for localized results in form or probability maps etc the target yi=R′i may need to be preprocessed using other machine learning techniques for more advantageous formatting, for example to reduce sparsity. One-hot encoding may be used which may be optionally transformed into another representation (sometimes referred to as “code”) by an auto-encoder for example.
The training input data xk may be obtained from historical image data acquired in the cath lab and held in image repositories, such as the PACS of a HIS (hospital information system) for instance. The pressure readings may likewise be found in medical databases. The targets yk or “ground truth” represents labels of interest for classification type models M, and is an indication for the mentioned balloon rupture or other adverse events. Medical outcomes may be sourced from medical health records of former cath lab patients. The respective labels may be found in header data or could be inferred from inspecting the medical records and notes. Textual reports on historical balloon catheter interventions may be required by a search engine for keyword tell-tales such as “rupture”, “compromised balloon” etc. a. There are large datasets available for clinical trials and registries that track outcomes in relation to new therapy devices, in particular for drug-coated balloons. In such registries, time-series of x-ray and corresponding pressure measurements are stored, as well as trial/registry reporting values. Preferably the training data set is sourced from a large and variable patient population (as is normally reflected in registries). The patient population from which the training data is sourced preferably includes patients with varying anatomical features, disease severity, disease location, gender, age, and medical history.
In the training phase, an architecture of a machine learning model M, such as the shown NN-type of
Assuming for now the paradigm of a cost function F, this measures the aggregated residue(s), that is, the error incurred between data estimated by the model M and the targets as per some or all of the training data pairs k in a batch:
argminθF=Σk∥Mθ(xk),yk∥ (2)
In eq. (2), function MO denotes the result of the model M applied to training input x for a given training pair k. The cost function may be a (squared) Euclidean-distance type cost function (such as least squares or similar) preferably used for regression tasks. However, for classifier tasks as envisaged herein, the summation in (2) is formulated instead as one of cross-entropy or Kullback-Leiber divergence or similar.
In training, the training input data xk of a training pair is propagated through the initialized network M. Specifically, the training input xk for a k-th pair is received at input IL, passed through the model and is then output at output OL as output training data Mθ (x). As indicated, a suitable measure ∥•∥ is used such as p-norm, squared differences, or cross-entropy, to measure the difference, also referred to herein as residue, between the actual training output Mθ (xk) produced by the model M, and the desired target yk.
The output training data M(xk) is an estimate for target yk associated with the applied input training image data xk. In general, there is an error between this output M(xk) and the associated target yk of the presently considered k-th pair. An optimization scheme such as backward/forward propagation or other gradient based methods may then be used to adapt the parameters θ of the model M so as to decrease the residue for the considered pair (xk, yk) or a subset of training pairs from the full training data set.
After one or more iterations in a first, inner, loop in which the parameters θ of the model are updated by updater UP for the current pair (xk,yk), the training system TS enters a second, an outer, loop where a next training data pair xk+1, yk+1 is processed accordingly. The structure of updater UP depends on the optimization scheme used. For example, the inner loop as administered by updater UP may be implemented by one or more forward and backward passes in a forward/backpropagation algorithm. While adjusting the parameters, the aggregated, for example summed, residues of all the training pairs are considered up to the current pair, to improve the objective function F. The aggregated residue can be formed by configuring the objective function F as a sum of residues such as in eq. (2) of some or all considered residues for each pair. In the inner loop, more than one pair may be provided to minimize (2) at once, the number of so provided pairs often referred to as batches of training data. The optimization in the outer loop than progressed from batch to batch until all training data, or a pre-defined minimum number, has been exhausted. It may also be possible to process all training data at once although in most cases this will be inefficient and progression in subsets of training data (that is, the batches) may be preferable.
The training system as shown in
The fully trained machine learning module M may be stored in one or more memories MEM or databases, and can be made available as pre-trained machine learning models for use in system SYS. The trained model M may be made available in a cloud service. Access can either be offered free of charge or their use can be granted via license-pay or pay-per-use scheme.
Reference is now made to
Suitable training data needs to be collated from historic intervention, such as balloon catheterizations, and as documented in medical databases mentioned above. Preferably, supervised learning schemes are envisaged herein although this is not a necessity as unsupervised learning setups are also envisaged herein.
In supervised learning, the training data includes suitable pairs of data items, each pair including training input data and associated therewith a target training output data. Specifically, the pairs comprise imagery and pressure readings suitably labeled in relation to adverse events and/or medical outcome. The imagery and pressure readings can be paired up with their result label by retrieving the same from historic balloon catheterization treatment data records, health records or other medical data repositories, as described above.
With continued reference to
At step S1020, the training input xk is applied to an initialized machine learning model NN to produce a training output yk.
At step S1030 a deviation, or residue, of the training output M(xk) from the associated target yk is quantified by a cost function F. One or more parameters of the model are adapted at step S1040 in one or more iterations in an inner loop to improve the cost function. For instance, the model parameters are adapted to decrease residues as measured by the cost function. The parameters may include in particular weights W of each node in each layer.
The training method then returns in an outer loop to step S1010 where the next one or more pairs of training data (depending on batch size) is fed in. In step S1020, the parameters of the model are adapted so that the aggregated residues of all pairs considered are decreased, in particular minimized. The cost function quantifies the aggregated residues. Forward-backward propagation or similar gradient-based techniques may be used in the inner loop.
More generally, the parameters of the model M are adjusted to improve objective function F which is either a cost function (see eq (2) above) or a utility function. In embodiments, the cost function is configured to the measure the aggregated residues. In embodiments the aggregation of residues is implemented by summation over all or some residues for all pairs considered, depending on batch size. The method may be implemented on one or more general-purpose processing units TS, preferably having processors capable for parallel processing to speed up the training.
Whilst the system SYS for intervention support and the training system have been explained above with main reference to x-ray imaging, other imaging modalities such as 4D CT/MRI, ultrasound, OCT (optical coherence tomography) or any other times series based imaging are also envisaged herein. Some imaging modalities such as OCT could be integrated in the balloon catheter itself. Endoscopic imaging is also envisaged herein.
The components of system SYS,TS may be implemented as one or more software modules, run on one or more general-purpose processing units PU such as a workstation associated with the imager IA, or on a server computer associated with a group of imagers.
Alternatively, some or all components of the system TS, SYS may be arranged in hardware such as a suitably programmed microcontroller or microprocessor, such an FPGA (field-programmable-gate-array) or as a hardwired IC chip, an application specific integrated circuitry (ASIC), integrated into the imaging system IA. In a further embodiment still, any one of systems TS, SYS may be implemented in both, partly in software and partly in hardware.
The different components of any one or system TS, SYS may be implemented on a single data processing unit PU. Alternatively, some or more components are implemented on different processing units PU, possibly remotely arranged in a distributed architecture and connectable in a suitable communication network such as in a cloud setting or client-server setup, etc.
One or more features described herein can be configured or implemented as or with circuitry encoded within a computer-readable medium, and/or combinations thereof. Circuitry may include discrete and/or integrated circuitry, a system-on-a-chip (SOC), and combinations thereof, a machine, a computer system, a processor and memory, a computer program.
In another exemplary embodiment of the present invention, a computer program or a computer program element is provided that is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments, on an appropriate system.
The computer program element might therefore be stored on a computer unit, which might also be part of an embodiment of the present invention. This computing unit may be adapted to perform or induce a performing of the steps of the method described above. Moreover, it may be adapted to operate the components of the above-described apparatus. The computing unit can be adapted to operate automatically and/or to execute the orders of a user. A computer program may be loaded into a working memory of a data processor. The data processor may thus be equipped to carry out the method of the invention. This exemplary embodiment of the invention covers both, a computer program that right from the beginning uses the invention and a computer program that by means of an up-date turns an existing program into a program that uses the invention.
Further on, the computer program element might be able to provide all necessary steps to fulfill the procedure of an exemplary embodiment of the method as described above.
According to a further exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is presented wherein the computer readable medium has a computer program element stored on it which computer program element is described by the preceding section.
A computer program may be stored and/or distributed on a suitable medium (in particular, but not necessarily, a non-transitory medium), such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.
However, the computer program may also be presented over a network like the World Wide Web and can be downloaded into the working memory of a data processor from such a network. According to a further exemplary embodiment of the present invention, a medium for making a computer program element available for downloading is provided, which computer program element is arranged to perform a method according to one of the previously described embodiments of the invention.
It has to be noted that embodiments of the invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims whereas other embodiments are described with reference to the device type claims. However, a person skilled in the art will gather from the above and the following description that, unless otherwise notified, in addition to any combination of features belonging to one type of subject matter also any combination between features relating to different subject matters is considered to be disclosed with this application. However, all features can be combined providing synergetic effects that are more than the simple summation of the features.
While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing a claimed invention, from a study of the drawings, the disclosure, and the dependent claims.
In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items re-cited in the claims. The mere fact that certain measures are re-cited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/083759 | 12/1/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63121289 | Dec 2020 | US |
