Patent Application
20240412368
The present disclosure relates to determining image perspective score values for X-ray projection images. A computer-implemented method, a system, and a computer program product, are disclosed.
Medical investigations frequently use X-ray projection images to provide information on a region of interest. As compared to 3D X-ray images, projection images, i.e. 2D images, may be generated with lower X-ray dose. However, projection images represent the integrated X-ray attenuation along X-ray paths, and it can be challenging to find a perspective of the X-ray imaging system respective the region of interest, i.e. a viewing angle, that provides the desired information in the projection images.
By way of an example, aneurism coiling procedures typically involve the insertion of wire coils into a sack of the aneurism in order to reduce blood flow and thereby enable the blood inside the aneurysm to coagulate. A goal in aneurism coiling is to fill the aneurysm sac with sufficient coils in order to effectively treat the aneurysm, whilst avoiding overfilling. Overfilling the aneurism can result in the aneurysm rupturing, or coil material spilling into the parent vessel that is connected to the aneurysm sac. Aneurism coiling procedures are typically performed using X-ray projection images. In order to detect coil material spilling into the parent vessel, a radiographer typically tries to find a perspective of the X-ray imaging system that minimizes the overlap between the sac and the parent vessel in the projection image. The radiographer may also try to find a perspective of the X-ray imaging system that provides minimal overlap between the parent vessel and other anatomical structures, and which also provides minimal foreshortening of the parent vessel.
The need for finding an optimal perspective of an X-ray imaging system for generating projection images also exists beyond vascular procedures. By way of another example, spinal fusion procedures that are performed to restore stability in severe arthritis or after removal of a herniated disk, often involve the placement of pedicle screws into vertebrae. In order to confirm the successful placement of the screws, a perspective of the X-ray imaging system is desired that not only minimizes foreshortening of the screw, but also minimizes overlap between the screw and the spinal cord in order to ensure that no injury to the spinal cord may occur. Yet another example may be found in the field of orthopaedics, and wherein a perspective of an X-ray imaging system may be desired that minimizes foreshortening of target bony structures whilst also minimizing their overlap with other anatomical structures. Thus, it may be useful to assess the perspective of X-ray projection images in order to provide an optimal perspective for an X-ray imaging system.
Owing to the multitude of factors that affect the choice of a perspective of an X-ray imaging system for generating X-ray projection images, a trial-and-error approach is often used, and wherein an initial promising perspective of the X-ray imaging system is adjusted until an image with an acceptable perspective is obtained. This, however, increases the amount of X-ray dose to a subject, and may ultimately provide an image with a perspective that is only marginally improved.
An improvement to the trial-and-error approach for finding an optimal perspective of an X-ray imaging system is the analytical approach described in a document by D. L. Wilson, et al., entitled “Determining X-ray projections for coil treatments of intracranial aneurysms” in IEEE Transactions on Medical Imaging, vol. 18, no. 10, pp. 973-980, October 1999, doi: 10.1109/42.811309. In this document, a method is presented for automatically finding the angles that will produce desired X-ray projections.
Aside from finding an optimal perspective for an X-ray imaging system, it may be useful to assess the perspective of X-ray projection images in other situations as well. By way of an example, medical investigations often involve the acquisition of multiple X-ray projection images of a region of interest. After the medical investigation, many of the acquired images serve little purpose since the information they provide is duplicated in other images. In order to limit data storage requirements, it would be beneficial to archive only the image(s) having a perspective that meet predetermined criteria. However, manually analyzing such images is time-consuming.
Consequently, there remains room for improvements in assessing the perspective of X-ray projection images.
According to one aspect of the present disclosure, a computer-implemented method of determining image perspective score values for X-ray projection images representing a region of interest in a subject, is provided. The method includes:
The predicted image perspective score values provide an indication of the quality of the perspectives of the X-ray images. The predicted image perspective score values may be used for various purposes. For example, they may be used to determine which of the inputted X-ray projection images provide an acceptable perspective, and thus which perspective of the X-ray imaging system to use to acquire further X-ray images. The predicted image perspective score values may also be used to determine which of the inputted X-ray images to archive.
Further aspects, features, and advantages of the present disclosure will become apparent from the following description of examples, which is made with reference to the accompanying drawings.
Examples of the present disclosure are provided with reference to the following description and figures. In this description, for the purposes of explanation, numerous specific details of certain examples are set forth. Reference in the specification to “an example”, “an implementation” or similar language means that a feature, structure, or characteristic described in connection with the example is included in at least that one example. It is also to be appreciated that features described in relation to one example may also be used in another example, and that all features are not necessarily duplicated in each example for the sake of brevity. For instance, features described in relation to a computer implemented method, may be implemented in a computer program product, and in a system, in a corresponding manner.
In the following description, reference is made to examples of computer-implemented methods that involve determining image perspective score values for X-ray images representing a region of interest in a subject. Reference is made to examples of X-ray projection images in the form of angiographic images wherein the region of interest is an aneurism in the brain. The angiographic images may be generated using angiographic techniques such as digital subtraction angiography “DSA”, for example. Angiographic images, i.e. images that are generated using a contrast agent, are typically used to visualize regions of interest in the vasculature, such as aneurisms. However, it is to be appreciated that these angiographic images serve only as examples, and that the computer-implemented methods disclosed herein may also be used with other types of X-ray projection images, such as for example fluoroscopic images, and also images which represent other regions of interest other than the vasculature. It is therefore to be appreciated that the computer-implemented methods may be used to determine image perspective score values for X-ray projection images in general, and that the use of the methods is not limited to X-ray images that include aneurisms, or to images of the brain.
It is noted that the computer-implemented methods disclosed herein may be provided as a non-transitory computer-readable storage medium including computer-readable instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to perform the method. In other words, the computer-implemented methods may be implemented in a computer program product. The computer program product can be provided by dedicated hardware, or hardware capable of running the software in association with appropriate software. When provided by a processor, the functions of the method features can be provided by a single dedicated processor, or by a single shared processor, or by a plurality of individual processors, some of which can be shared. The functions of one or more of the method features may for instance be provided by processors that are shared within a networked processing architecture such as a client/server architecture, a peer-to-peer architecture, the Internet, or the Cloud.
The explicit use of the terms “processor” or “controller” should not be interpreted as exclusively referring to hardware capable of running software, and can implicitly include, but is not limited to, digital signal processor “DSP” hardware, read only memory “ROM” for storing software, random access memory “RAM”, a non-volatile storage device, and the like. Furthermore, examples of the present disclosure can take the form of a computer program product accessible from a computer-usable storage medium, or a computer-readable storage medium, the computer program product providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable storage medium or a computer readable storage medium can be any apparatus that can comprise, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or a semiconductor system or device or propagation medium. Examples of computer-readable media include semiconductor or solid state memories, magnetic tape, removable computer disks, random access memory “RAM”, read-only memory “ROM”, rigid magnetic disks and optical disks. Current examples of optical disks include compact disk-read only memory “CD-ROM”, compact disk-read/write “CD-R/W”, Blu-Ray™ and DVD.
As mentioned above, it may be useful to assess the perspective of X-ray projection images in various situations.
One example of such a situation relates to an aneurism coiling procedure. A goal in aneurism coiling is to fill the aneurysm sac with sufficient coils in order to effectively treat the aneurysm, whilst avoiding overfilling. In order to detect coil material spilling into the parent vessel, a radiographer typically tries to find a perspective of the X-ray imaging system that minimizes the overlap between the sac and the parent vessel in the projection image. The radiographer may also try to find a perspective of the X-ray imaging system that provides minimal overlap between the parent vessel and other anatomical structures, and which also provides minimal foreshortening of the parent vessel. Overlap between the parent vessel and other anatomical structures is desirably minimized in order to provide a clear view of the parent vessel. Foreshortening is desirably minimized because a significant portion of coiling wire in the vessel could be overlooked if the longitudinal axis of the wire is aligned with the viewing angle.
Another situation in which it is useful to assess the perspective of X-ray projection images, relates to image archiving. Medical investigations often involve the acquisition of multiple X-ray projection images of a region of interest. After the medical investigation, many of the acquired images serve little purpose since the information they provide is duplicated in other images. In order to limit data storage requirements, it would be beneficial to archive only the images that provide a desired perspective of the region of interest. However, the manual analysis of such images is time-consuming.
It may also be useful to assess the perspective of X-ray projection images in other situations as well.
With reference to
The above method is also described with reference to
With reference to
The X-ray projection images 110 that are received in the operation S110 may be angiographic images, or they may be fluoroscopic, i.e. live images, for example. Angiographic projection images may be generated using a digital subtraction angiographic “DSA” technique, and wherein each image is generated by subtracting from the image the corresponding pixel intensities of a background image. The X-ray projection images 110 that are received in the operation S110 may alternatively be synthetic projection images that are generated by projecting, at different perspectives, a 3D X-ray image, such as a 3D angiogram, that is generated by a volumetric X-ray imaging system. In contrast to projection X-ray imaging systems, volumetric X-ray imaging systems typically generate image data whilst rotating, or stepping, an X-ray source-detector arrangement around an imaging region, and subsequently reconstruct the image data obtained from multiple rotational angles into a 3D, or volumetric image. Examples of volumetric X-ray imaging systems include computed tomography “CT” imaging systems, cone beam CT “CBCT” imaging systems, and spectral CT imaging systems. A 3D angiogram may be generated from a volumetric imaging system by performing an imaging operation using a contrast agent. The synthetic projection images may be generated from the 3D X-ray image by projecting the 3D X-ray image onto a virtual detector based on the perspective of an X-ray source-detector arrangement respective the 3D X-ray image.
The X-ray projection images that are received in the operation S110 may be received from an X-ray imaging system, such as the projection X-ray imaging system 130 illustrated in
The X-ray projection images that are received in the operation S110 represent a region of interest 120 from a plurality of different perspectives, i.e. orientations, of the X-ray imaging system respective the region of interest. In this respect, the X-ray projection images may be generated by adjusting a perspective, or orientation, of the projection X-ray imaging system 130 illustrated in
In the operation S120, the X-ray projection images are inputted into a neural network NN1. The neural network NN1 is trained to generate predicted image perspective score values s1 for the X-ray projection images. The predicted image perspective score values s1 provide a subjective assessment of the quality of the perspectives of the X-ray projection images. The subjective assessment depends on the ground truth training data that is used to train the neural network NN1. A high quality perspective might for example be associated with an X-ray projection image in which the region of interest is separated from other features in the image. A high quality perspective might also be associated with an X-ray projection image in which a portion of the region of interest has a low amount of foreshortening. A high quality perspective might also be associated with an X-ray projection image in which there are few artifacts from confounding features. The degree to which such factors affect the predicted image perspective score values s1 depends on the ground truth training data that is used to train the neural network NN1, and thus the predicted image perspective score values s1 provide a subjective assessment of the quality of the perspectives of the X-ray projection images. Further detail on training the neural network NN1 is provided below.
In the operation S130, in response to inputting the X-ray projection images into the neural network NN1, a predicted image perspective score value s1 is generated for each of the X-ray projection images. The predicted image perspective score value s1 may be outputted in various forms. In one example, the image perspective score value s1 is outputted graphically for each of the X-ray projection images. An example of such a graphical output is the graph is illustrated in the right-hand portion of
The predicted image perspective score values s1 thus provide a subjective assessment of the quality of the perspectives of the X-ray projection images. The predicted image perspective score values s1 may be used for various purposes. For example, they may be used to determine which of the inputted images have an acceptable perspective, and thus which perspective of the X-ray imaging system to use to generate further X-ray projection images. The predicted image perspective score values s1 may also be used to select which of the inputted X-ray projection images to archive. In one example, the method described with reference to
The neural network NN1 may also generate a confidence value, u1, for each of the predicted image perspective score values s1. In one example, the method described above with reference to
An example of a technique for generating confidence values associated with a neural network's predictions is disclosed in a document by Ramalho, T. et al., entitled “Density estimation in representation space to predict model uncertainty”, https://arxiv.org/pdf/1908.07235.pdf. The neural network NN1 may be trained in accordance with this technique to generate confidence values such that when the neural network NN1 is presented with an image that is very different from its training dataset, the neural network NN1 it is able to recognize this and the neural network NN1 outputs a low confidence value. The technique described in this document generates confidence values by estimating the training data density in representation space, and determining whether the trained network is expected to make a correct prediction for the input by measuring the distance in representation space between the input and its closest neighbors in the training set. Alternative techniques may also be used to generate confidence values associated with the predictions of the neural network NN1. The dropout technique may be used, for example. The dropout technique involves iteratively inputting the same data into a neural network and determining the neural network's output whilst randomly excluding a proportion of the neurons from the neural network in each iteration. The outputs of the neural network are then analyzed to provide mean and variance values. The mean value represents the final output, and the magnitude of the variance indicates whether the neural network is consistent in its predictions, in which case the variance is small, or whether the neural network was inconsistent in its predictions, in which case the variance is larger.
In one example, an analytical image perspective score value s2 is also generated for each of the received X-ray projection images. The analytical image perspective score value s2 provides an objective assessment of the quality of the perspective of each X-ray projection image. In this example, the analytical image perspective score value s2 is combined with the predicted image perspective score value s1 to provide a combined image perspective score value for each X-ray projection image. This example is described with reference to
With reference to
In contrast to the predicted image perspective score value s1, and which provides a subjective assessment of the quality of the perspective of the X-ray projection image, the analytical image perspective score value s2 provides an objective assessment of the quality of the perspective of the X-ray projection image. By providing a combined image perspective score value s′ that depends on both the analytical image perspective score value s2, as well as the predicted image perspective score value s1, the combined image perspective score value s′ provides an assessment in which a subjective bias of the predicted image perspective score value s1 that is introduced via the training of the neural network NN1, may be compensated-for by the objective analytical image perspective score values s2.
In this example, the X-ray projection images represent the region of interest from a corresponding perspective a, b of the X-ray imaging system respective the region of interest in the subject. The perspective may for example be defined by a rotational angle a of a central ray of the X-ray imaging system around a longitudinal axis of the subject, and also by a tilt angle b of a central ray of the X-ray imaging system with respect to a cranial-caudal axis of the subject 220. An example of such a perspective is illustrated in
In this example, a 3D X-ray image 140 representing the region of interest 120 is received. The 3D X-ray image may be generated using a volumetric imaging system. As described above for the X-ray projection images, the 3D X-ray image may be a 3D angiographic image. The 3D angiographic image may be generated using a digital subtraction angiography technique, for example.
In this example, the 3D X-ray image may be received by the one or more processors 210 illustrated in
In this example, the region of interest in the 3D X-ray image 140 is registered to the region of interest in each of the X-ray projection images in order to provide a perspective a, b of the X-ray imaging system respective the region of interest in the 3D X-ray image 140. This registration may be performed based on the known perspectives of each of the volumetric imaging system that generates the 3D X-ray image 140, and the projection X-ray imaging system that generates the X-ray projection images 110, respective the subject. Alternatively, this registration may be performed using an image matching technique wherein synthetic projections of the 3D X-ray image 140 at different perspectives of the X-ray imaging system respective the 3D X-ray image, are generated, and compared to the X-ray projection images, until a perspective is found that provides matching images, as is known from the image registration field. In the former case, a perspective of the volumetric imaging system that generates the 3D X-ray image 140, respective the subject, is typically known because when a 3D X-ray image 140 is reconstructed, it is reconstructed with respect to the orientation of the volumetric imaging system. The orientation of the subject with respect to the volumetric imaging system is also known because during generation of the 3D X-ray image the subject lies on a patient bed, and the orientation of the patient bed is known with respect to the volumetric imaging system. In a similar manner, when X-ray projection images are generated, the perspective of the projection X-ray imaging system that generates the X-ray projection images 110, is known respective the subject because the patient also lies on a patient bed, and the perspective of the X-ray imaging system is typically recorded with respect to the patient bed for each X-ray projection image in terms of the perspective parameters a and b described above. Thus, the registration that is performed in this operation may be carried out by matching the orientation of the subject in the 3D X-ray image, to the orientation of the subject in each of the X-ray projection images.
In this example, an analytical image perspective score value s2 is computed for each X-ray projection image, from the 3D X-ray image 140, and based on the perspective a, b of the X-ray imaging system respective the region of interest in the 3D X-ray image 140. The perspective a, b of the X-ray imaging system respective the region of interest in the 3D X-ray image 140 is determined using the registration described above. This is illustrated in
The analytical image perspective score value s2 is computed from the 3D X-ray image based on the values of one or more of the following metrics C1..n: a degree of overlap between a plurality of features in the region of interest, a foreshortening of one or more features in the region of interest, and a presence of one or more artifacts in the region of interest.
A technique for computing such analytical image perspective score values is described in a document by D. L. Wilson, et al., entitled “Determining X-ray projections for coil treatments of intracranial aneurysms” in IEEE Transactions on Medical Imaging, vol. 18, no. 10, pp. 973-980, October 1999, doi: 10.1109/42.811309. In this document, a method is presented for automatically finding the angles that will produce desired X-ray projections. An algorithm is presented for computing factors such as the aforementioned overlaps, and foreshortening. The algorithm uses as inputs i) a discretely defined 3D segmentation of a region of interest and surrounding anatomy, and ii) a model of the geometric and imaging parameters of an X-ray imaging system.
An example of the calculation of an analytical image perspective score value s2 for a 3D X-ray image that includes a region of interest 120 in the form of a brain aneurism, is provided as follows. Firstly, the 3D X-ray image of the aneurism is segmented in order to assign voxels to the aneurism sac, and to a parent vessel feeding the aneurism sac. Simulated 2D projections of the 3D X-ray image are then generated at multiple different perspectives of an X-ray imaging system with respect to the 3D X-ray image. At each perspective, the value of an image perspective metric is calculated. The image perspective metric may for example include an overlap metric that is determined based on a number of aneurism sac voxels that overlap with the parent vessel. A value of the overlap metric may be calculated by projecting virtual rays from the X-ray source of the X-ray imaging system, through the 3D X-ray image and onto the X-ray detector of the X-ray imaging system, and counting the number of aneurism sac voxels that overlap parent vessel voxels. Since it is typically desired to provide non-overlapping views of the aneurism sac, and the parent vessel, an overlap criterion can be to minimize the value of the overlap metric.
Similarly, a foreshortening metric may be calculated for a portion of a vessel by projecting virtual rays from the X-ray source of the X-ray imaging system, through the 3D X-ray image and onto the X-ray detector, and calculating value of the average projected intensity of the portion of the vessel on the X-ray detector. When an axis of the vessel is parallel to the X-ray detector plane, the value of the foreshortening metric is lowest because the apparent cross sectional area of the vessel on the X-ray detector is highest. As the axis of the vessel is tilted away from the X-ray detector plane, the value of the foreshortening metric increases due to the reduction in apparent cross sectional area of the vessel on the X-ray detector and due to the increase in integrated amount of contrast agent along the path of the virtual X-rays, until an axis of the portion of the vessel is aligned with the paths of the virtual rays, and at which perspective the value of the foreshortening metric is highest. Since it is typically desired to view the parent vessel such that it is parallel to the detector plane, i.e. to minimize foreshortening, a foreshortening criterion can be to minimize the value of the foreshortening metric for the parent vessel of the aneurism.
Similarly, a presence of one or more artifacts in the region of interest may be detected and a value of an artifact metric may be calculated. By way of an example, the artifact metric may represent an overlap between the region of interest and a confounding feature in the projection image. In the example of the region of interest being an aneurism, confounding features that may also be present in the X-ray projection image include skull shadow, and metal artifacts, for instance, from dental implants or fillings. Since it is typically desired to optimize the clarity of a region of interest, an artifact criterion can be to minimize the value of the artifact metric.
The overlap metric, the foreshortening metric, the artifact metric, and other metrics, provide objective assessments of the quality of the perspective of the X-ray projection image, and may be used individually, or in combination, to provide the analytical image perspective score value s2. If multiple metrics C1..n are used to compute the analytical image perspective score value s2, the metrics may include corresponding weights l1..n defining an importance of the metrics on the analytical image perspective score value s2.
For example, if the region of interest is a side wall aneurysm, and assuming for the sake of simplicity that the analytical image perspective score value s2 is based on only two metrics, C1 and C2, the metrics may include an overlap of the aneurism sac and the parent vessel, C1, and foreshortening of the parent vessel, C2, The analytical image perspective score value s2 may be determined from the metrics C1 and C2 using the equation:
The overlap criterion, the foreshortening criterion, the artifact criterion, and other criteria, may also be used individually, or in combination, to calculate an optimized analytical image perspective score value s2 in order to provide an optimal view of the region of interest. For example, in Equation 1 it may be desirable to minimize the metric C1; this being to provide optimal detection of coil migration into the parent vessel, and also to minimize the metric C2; this being to provide an optimal view of coil insertion from the connecting artery into the aneurism sac.
In this example, the analytical image perspective score value s2, and the predicted image perspective score value s1 that is generated by the neural network NN1, are combined in order to provide a combined image perspective score value s′ for the X-ray projection image. In so doing, a subjective bias of the predicted image perspective score value s1 that is introduced via the training of the neural network NN1, may be compensated-for by the objective analytical image perspective score values s2.
The analytical image perspective score value s2, and the predicted image perspective score value s1, may be combined in various ways. For example, the image perspective score values s1 and s2 may be weighted. The image perspective score values may be weighted with weightings that depend on the X-ray projection images, or the weightings may be fixed. The weightings may for example be fixed such that the combined image perspective score value s′ is an average of the image perspective score values s1 and s2.
In one example, the confidence values u1 of the predicted image perspective score values s1 are used to weight both the predicted image perspective score values s1 and the analytical image perspective score value s2. In this example, the combined image perspective score value s′ may be calculated using the equation:
and wherein u1 lies in the range from 0 to 1. Thus, if the confidence values u1 of the predicted image perspective score values s1 are high, these may be used to place a higher weighting on the predicted image perspective score values s1 than on the analytical image perspective score value s2.
By way of some examples, in one situation, an X-ray projection image 110 may have a very similar appearance or features to the training data that is used to train the neural network NN1. In this situation, the neural network NN1 would be expected to predict an image perspective score value s1 for the X-ray projection image 110 with a high confidence value, u1, e.g. u1=0.9, indicating that it can be relied-upon. In this situation, the combined image perspective score value is calculated as s′=0.9·s1+0.1·s2 In another situation, an X-ray projection image 110 may have fewer features in common with the training data that is used to train the neural network NN1. In this situation, the confidence value of neural network NN1 might be e.g. u1=0.1, and thus the combined image perspective score value would be calculated as s′=0.1·s1+0.9·s2. In another situation, an X-ray projection image 110 may have some similarity to the training data that is used to train the neural network NN1. In this situation, the confidence value of neural network NN1 might be e.g. u1=0.5, and the combined image perspective score value would be calculated as s′=0.5·s1+0.5·s2, i.e. as an average of the predicted image perspective score value s1 and the analytical image perspective score value s2. Using the confidence values u1 of the predicted image perspective score values s1 to weight both the predicted image perspective score value s1 and the analytical image perspective score value s2 in this manner avoids over-reliance on an unreliable image perspective score value s1.
In one example, the predicted image perspective score values s1 may be omitted from the calculation of the combined image perspective score value s′. For example, if the confidence values of the predicted image perspective score values s1 fail to exceed a predetermined threshold value, the predicted image perspective score values s1 may be omitted from the calculation of the combined image perspective score value. This may be achieved by setting the value of s1 to zero in Equation 2, for example. In so doing, over-reliance on an unreliable predicted image perspective score value s1, is also avoided.
As mentioned above, the analytical image perspective score value s2 may be computed based on multiple metrics C1..n. For example, the analytical image perspective score value s2 may be calculated based on an overlap metric, a foreshortening metric, and also based on other metrics. In such cases, the metrics may include corresponding weights l1..n defining an importance of the metrics on the analytical image perspective score value s2. The use of various techniques is contemplated for setting the values of these weights l1..n. The values of the weights l1..n may for example be set based on user input, or based on reference values obtained from a lookup table. However, the task of setting the values of the weights l1..n in is not trivial. This is because, in practical situations their values may depend on factors such as a clinical procedure, the specific region of interest in the X-ray images, e.g. a type of the aneurysm, as well as subjective factors such as a user's preference. Thus it would be advantageous to set the values of the weights l1..n for individual X-ray projection images in an efficient and reliable manner.
In one example, the values of the weights l1..n are set for each X-ray projection image using a neural network. In this example, the analytical image perspective score value s2 is computed based on a plurality of metrics, and the metrics include corresponding weights l1..n defining an importance of the metrics on the analytical image perspective score value s2; and the method described with reference to
Setting the values of the weights l1..n in this manner may be considered to efficiently provide reliable analytical image perspective score values, s2. Moreover, it overcomes the burden of manually setting the weights for individual images, as well as the limitations arising from setting the weights to fixed values.
This example operates in the same manner as described above in relation to
In one example, the second neural network NN2 may include an additional input for receiving input data representing a physician's preference of one or more of the multiple metrics C1..n that are used to calculate the analytical image perspective score values s2. This is illustrated in
In another example, a user's preference may be incorporated into the combined image perspective score value s′ by performing inference with a second neural network NN2 that is selected based on a type of training data used to train the second neural network NN2. In this example, the values of the weights l1..n used to compute the analytical image perspective score value s2, are set based on the predicted values of the weights l1..n; and the method described with reference to
In so doing, the combined image perspective score value s′ that are generated may be tailored to reflect a user's preferences for particular views of a region of interest.
In the case that the same training data is used to train neural network NN1 and neural network NN2, the confidence values u1 that are generated by the neural network NN1 may also be used to decide whether to use the second neural network NN2 to generate the values of the weights l1..n for the metrics C1..n, or to instead use values for these weights l1..n from a lookup table. In situations where the neural network NN1 is presented with an image that is different from its training data, the confidence u1 of the neural network NN1 is expected to be low, and it may be decided to use lookup table values to provide the values of the weights rather than to use NN2 to do this. If the neural network NN1 is presented with an image that is different from its training data, it is likely that this will also yield an unreliable output from the second neural network NN2 as well, and so it may be better to rely on the lookup table values for the weights, rather than the potentially unreliable values provided by the second neural network NN2.
Thus, in one example, the method described with reference to
In one example, the second neural network NN2 may also generate confidence values for its predicted values of the weights l1..n. In one example, the confidence values u2 of the predicted values of the weights l1..n are used to weight both the analytical image perspective score values s2 and the predicted image perspective score values s1. In another example, the analytical image perspective score value s2 are omitted from the provision of the combined image perspective score value if the confidence values u2 of the predicted values of the weights l1..n fail to exceed a predetermined threshold. In so doing, it is avoided that an unreliable analytical image perspective score value s2 affects the combined image perspective score value s′ that is provided for an image.
In another example, the analytical image perspective score values s2 are set for each X-ray projection image based on values for similar X-ray projection images. This example may be used in the absence of both the second neural network NN2 and the analytical scoring module ASM illustrated in
In this example, the reference analytical image perspective score values s2 that are stored in the database may be set by experts. The reference analytical image perspective score values s2 are thus, to some extent, tailored to the images.
When used in combination with the neural network NN2 and the analytical scoring module ASM, the reference analytical image perspective score values s2 may be selectively provided contingent on the confidence values u2 of the predicted values of the weights l1..n. For example, if the confidence values u2 of the predicted values of the weights l1..n fail to exceed a predetermined threshold, the reference analytical image perspective score values s2 may be provided instead. In this way, it is avoided that unreliable weighting values generated by the second neural network NN2, are used to generate the combined image perspective score values s′.
In any of the examples described above, a subsequent perspective of the X-ray imaging system, may also be determined. The subsequent perspective may be outputted in order to inform an operator of the optimal perspective to use to generate further X-ray projection images. In this example, the method described with reference to
In this example, the X-ray imaging system perspective corresponding to the highest predicted image perspective score values s1, may be determined and used as the subsequent perspective. Alternatively, the X-ray imaging system perspective corresponding to the highest combined image perspective score values s′ may be used. This example finds use in determining an optimal viewing angle for imaging a region of interest. The user may for example generate the X-ray projection images 110 from different perspectives of the X-ray imaging system respective the region of interest by stepping the perspective of the X-ray imaging system in angles a and/or b as described above, in order to generate a range of different image perspective score values for the X-ray projection images 110. This procedure may be used to identify a promising perspective for viewing the region of interest. A more optimal position may be determined by iterating on this procedure with different step sizes. In so doing, the method provides an efficient technique for finding an optimal viewing angle with a low X-ray dose.
The training of the neural network NN1, and the second neural network NN2, is now detailed below.
In general, the training of a neural network involves inputting a training dataset into the neural network, and iteratively adjusting the neural network's parameters until the trained neural network provides an accurate output. Training is often performed using a Graphics Processing Unit “GPU” or a dedicated neural processor such as a Neural Processing Unit “NPU” or a Tensor Processing Unit “TPU”. Training often employs a centralized approach wherein cloud-based or mainframe-based neural processors are used to train a neural network. Following its training with the training dataset, the trained neural network may be deployed to a device for analyzing new input data during inference. The processing requirements during inference are significantly less than those required during training, allowing the neural network to be deployed to a variety of systems such as laptop computers, tablets, mobile phones and so forth. Inference may for example be performed by a Central Processing Unit “CPU”, a GPU, an NPU, a TPU, on a server, or in the cloud.
The process of training the neural networks NN1 and NN2 described above therefore includes adjusting its parameters. The parameters, or more particularly the weights and biases, control the operation of activation functions in the neural network. In supervised learning, the training process automatically adjusts the weights and the biases, such that when presented with the input data, the neural network accurately provides the corresponding expected output data. In order to do this, the value of the loss functions, or errors, are computed based on a difference between predicted output data and the expected output data. The value of the loss function may be computed using functions such as the negative log-likelihood loss, the mean squared error, or the Huber loss, or the cross entropy loss. During training, the value of the loss function is typically minimized, and training is terminated when the value of the loss function satisfies a stopping criterion. Sometimes, training is terminated when the value of the loss function satisfies one or more of multiple criteria.
Various methods are known for solving the loss minimization problem such as gradient descent, Quasi-Newton methods, and so forth. Various algorithms have been developed to implement these methods and their variants including but not limited to Stochastic Gradient Descent “SGD”, batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg Marquardt, Momentum, Adam, Nadam, Adagrad, Adadelta, RMSProp, and Adamax “optimizers” These algorithms compute the derivative of the loss function with respect to the model parameters using the chain rule. This process is called backpropagation since derivatives are computed starting at the last layer or output layer, moving toward the first layer or input layer. These derivatives inform the algorithm how the model parameters must be adjusted in order to minimize the error function. That is, adjustments to model parameters are made starting from the output layer and working backwards in the network until the input layer is reached. In a first training iteration, the initial weights and biases are often randomized. The neural network then predicts the output data, which is likewise, random. Backpropagation is then used to adjust the weights and the biases. The training process is performed iteratively by making adjustments to the weights and biases in each iteration. Training is terminated when the error, or difference between the predicted output data and the expected output data, is within an acceptable range for the training data, or for some validation data. Subsequently the neural network may be deployed, and the trained neural network makes predictions on new input data using the trained values of its parameters. If the training process was successful, the trained neural network accurately predicts the expected output data from the new input data.
In one example, the second neural network NN2 is trained to generate the predicted values of the weights l1..n for the X-ray projection images by:
This training method is described with reference to
With reference to
In the operation S220, virtual projection image training data is generated. This is generated by projecting the one or more 3D X-ray images 140′ onto a virtual detector plane of the X-ray imaging system at a plurality of different perspectives of the X-ray imaging system with respect to each 3D X-ray image 140′ to provide a plurality of synthetic projection images. The perspectives may be selected at random, and cover a wide range of all possible perspectives of the X-ray imaging system. Generating synthetic projection images, and using these to train the second neural network, addresses the challenge of obtaining a variety of X-ray projection images for a region of interest from different perspectives in order to train the second neural network NN2.
In the operation S230 an analytical image perspective score value s2 is calculated for each synthetic projection image for the corresponding perspective of the X-ray imaging system respective the 3D X-ray image. The analytical image perspective score value s2 is calculated from the 3D X-ray image using the corresponding perspective, and may be calculated using the technique described above. The analytical image perspective score value s2 may therefore be based on one or more of the following metrics C1..n: a degree of overlap between a plurality of features in the region of interest, a foreshortening of one or more features in the region of interest, and a presence of one or more artifacts in the region of interest. In an initial iteration, the values of the weights l1..n used to compute the analytical image perspective score value s2 are set to initial values. For example, assuming there are n weights, i.e. l1 to ln, the initial values may each be set to 1/n.
In the operation S240, a subset 150 of the synthetic projection images having analytical image perspective score values s2 meeting a predetermined selection criterion, are selected for use in training the second neural network NN2. The selection criterion may for example be that the analytical image perspective score values s2 exceed a predetermined threshold. In so doing, only the best views are selected for use in training the second neural network NN2. Reducing the amount of training data in this manner improves the efficiency of training the second neural network NN2 to predict the analytical image perspective score values s2 with high confidence. It also increases the feasibility of accurately labelling the training data without prohibitively increasing the workload for experts who can provide ground truth subjective scoring.
Alternative selection criteria may also be used for the synthetic projection images. For example, the selection criteria may be set so as to include some synthetic projection images that have high analytical image perspective score values s2, as well as some synthetic projection images that have low analytical image perspective score values s2. Such a selection allows the second neural network NN2 to generate low analytical image perspective score values s2 with high confidence, as well as high analytical image perspective score values s2 with high confidence, which is useful in identifying when the predicted the analytical image perspective score values s2 should, and should not, be used to calculate the combined image perspective score values s′.
In the operation S250, ground truth image perspective score values are received for the selected subset 150 of the synthetic projection images. The ground truth image perspective score values s2 may be provided by an expert.
In the operation S260, the subset 150 of the synthetic projection images is inputted into the second neural network NN2, and the neural network NN2 generates updated values of the weights l1..n for each synthetic projection image. The parameters of the second neural network NN2 are then adjusted in the operation S270 until a difference between the analytical image perspective score values s2 computed for the synthetic projection images with the updated values of the weights l1..n, and the corresponding ground truth image perspective score values, meet a stopping criterion. The operations S260 and S270 may be performed iteratively.
In so doing, the second neural network NN2 is trained to set the values of the weights l1..n in order to obtain the objective analytical image perspective score values s2 provided by the expert. Training the second neural network NN2 in this manner, in particular by using the synthetic projection images, overcomes the challenge of obtaining real projection images from multiple perspectives, in order to train the second neural network NN2.
An example of training the neural network NN1 described above with reference to
In this example, the neural network NN1 is trained to generate the predicted image perspective score values s1 for the X-ray projection images by:
In the operation S310, X-ray projection image training data is received. The X-ray projection image training data may be received as described above in relation to the training data for the second neural network. The X-ray projection image training data may be received from a database DB, as illustrated in
As illustrated in
In the operations S320, the training projection images, and the corresponding ground truth image perspective score values, are inputted into the neural network NN1. In the operation S330, its parameters are adjusted until a difference between the image perspective score values s1 predicted by the neural network NN1, and the corresponding inputted ground truth image perspective score values, meet a stopping criterion. The adjustment of the parameters may be performed by calculating the value of an error function that represents this difference, and using this to perform backpropagation in the neural network NN1.
In so doing, the neural network NN1 is trained to emulate the image perspective score values provided by the expert. The expert's bias is thus built-in to the image perspective score values predicted by the trained neural network NN1.
A potential limitation of the method described with reference to
In this example, the neural network NN1 is trained to generate the predicted image perspective score values s1 for the X-ray projection images by:
With reference to
In the operation S420, virtual projection image training data is generated. The virtual projection image training data includes a plurality of synthetic projection images. The synthetic projection images may be generated from the one or more 3D X-ray images 140′ in the manner described for the operation S220 above.
In the operation S430, an analytical image perspective score value s2 is calculated for each synthetic projection image for the corresponding perspective of the X-ray imaging system respective the 3D X-ray image. This operation may be performed in the manner described above for the operation S230.
In the operation S440, a subset 150 of the synthetic projection images is selected. This selection may be performed in the manner described above for the operation S240, with the difference that in the operation S440, the subset is now used to train the neural network NN1. The selection may be performed so as to include at least some projection images having ground truth image perspective score values that exceed a first threshold value. Alternatively, the selection may be performed so as to include at least some projection images having ground truth image perspective score values that exceed a first threshold value, and at least some ground truth image perspective score values that are below a second threshold value.
In the operation S450, ground truth image perspective score values are received for the selected subset of the synthetic projection images. The ground truth image perspective score values may be provided by an expert.
In the operation S460, the subset of the synthetic projection images, and the corresponding ground truth image perspective score values, are inputted into the neural network NN1. In the operation S470, the parameters of the neural network NN1 are adjusted until a difference between the image perspective score values s1 predicted by the neural network NN1, and the corresponding inputted ground truth image perspective score values, meet a stopping criterion. The adjustment of the parameters may be performed by calculating the value of an error function that represents this difference, and using this to perform backpropagation in the neural network NN1. The operations S460 and S470 may be performed iteratively.
In so doing, the neural network NN1 is trained to emulate the subjective image perspective score values provided by the expert. The expert's bias is thus built-in to the image perspective score values predicted by the trained neural network NN1. Moreover, since synthetic projection images are used, this is achieved without the need for an extensive dataset of real projection training images from multiple different perspectives.
In this example, the values of the weights l1..n that are used to compute the analytical image perspective score value s2, may be generated by the second neural network NN2. Thus, the second neural network NN2 may be trained before the neural network NN1, and then used to train the neural network NN1. Alternatively, a trained neural network NN1 may be re-trained or fine-tuned using the l1..n weighted analytical image perspective score value s2 generated by the neural network NN2. Alternatively, the values of the weights l1..n used to compute the analytical image perspective score value s2 may be set to reference values.
In accordance with another example, a computer program product is provided. The computer program product includes instructions which when executed by one or more processors, cause the one or more processors to carry out a method of determining image perspective score values s1 for X-ray projection images 110 representing a region of interest 120 in a subject. The method comprises:
In accordance with another example, a system for determining image perspective score values s1 for X-ray projection images 110 representing a region of interest 120 in a subject, is provided. The system 100 comprises one or more processors 210 configured to:
The example system 100 is illustrated in
The above examples are to be understood as illustrative of the present disclosure, and not restrictive. Further examples are also contemplated. For instance, the examples described in relation to computer-implemented methods, may also be provided by the computer program product, or by the computer-readable storage medium, or by the system 100, in a corresponding manner. It is to be understood that a feature described in relation to any one example may be used alone, or in combination with other described features, and may be used in combination with one or more features of another of the examples, or a combination of other examples. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims. In the claims, the word “comprising” does not exclude other elements or operations, and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage. Any reference signs in the claims should not be construed as limiting their scope.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2022/079371 | 10/21/2022 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63272318 | Oct 2021 | US |
