Patent Application
20220270270
Any and all applications for which a foreign or domestic priority claim is identified in the Application Data Sheet as filed with the present application are hereby incorporated by reference under 37 CFR 1.57.
The present disclosure generally relates to the field of intraoperative imaging, and more specifically to registration of multiple medical image data sets.
Registration is a method of image processing in which a coordinate transformation or association is determined between at least two images, such that a mutual association of the image contents can be established. Generally, this association may be calculated between two images which were recorded at different times, by means of different imaging methods, and/or under different recording conditions.
It proves to be of particular complexity when registering between two images if portions of one of the two images differ considerably from corresponding portions of the respective other image, for example by the presence of foreign objects which can cover parts of image areas and/or by the occurrence of movement of a medical area to be examined. The presence of such a case, referred to herein as a scene change, can considerably influence the value of a similarity measure between the images. The result can be that a correct coordinate transformation cannot be calculated between the images, or that it is unusable. Particularly in the field of medical imaging, it is desirable to avoid such cases from a risk point of view.
For the mutual registration of two images, there are various aspects which can be taken into account, for example if the images to be registered originate from different image recording devices or if they were produced using different imaging methods. In this case, image contents can already differ due to different exposure methods. For example, in X-ray imaging, these can be, for example, an X-ray C-arm (C-arm), for example also a mobile X-ray C-arm and an X-ray computer tomograph (CT). The noise and contrast conditions occurring in images can likewise impair the registration method, in particular to the extent that associations between the images can be determined with insufficient required accuracy. Furthermore, the degrees of freedom available during a registration (translation, rotation, scaling, shearing, elastic/inelastic transformation) can play a role. If several degrees of freedom are available for a registration, the probability of a successful registration between two images is greater, but the number of possible degrees of freedom available for a registration scales, even with a longer (computing) time for the registration.
Document U.S. Pat. No. 7,596,283 discloses a method for registration between two image data sets, in which features extracted from an image are used, which may be suitable for elastic registrations.
An example problem addressed by certain embodiments of the present technology is that of providing an improved method and an improved device for the registration of two or more medical image data sets. Without limiting the scope of the present disclosure, certain advantageous features are recited in the claims of the present application.
In a first aspect, a method of registering two sets of medical image data taking into account scene changes comprises, under control of one or more processors, a) receiving a first medical image data set and a second medical image data set from one or more medical devices; b) subdividing each of the first and second medical image data sets into an equal number of sub-images; c) performing a plurality of individual registrations between the first and second medical image data sets with respective optimization of a similarity measure, wherein, for each individual registration, at least one sub-image of the first and/or second medical image data set is masked out by means of a random process when determining the respective similarity measure; d) identifying the sub-images that have a scene change; and e) performing a final registration between the first medical image data set and the second medical image data set by masking out the sub-images identified in step d or a masked-out sub-image combination used in step c.
In some embodiments, both medical image data sets are three-dimensional volume images.
In some embodiments, the first medical image data set and the second medical image data set are each subdivided into n×m×p sub-images.
In some embodiments, both medical image data sets are two-dimensional images, preferably at least one of projection images or sectional images.
In some embodiments, the first medical image data set and the second medical image data set are each subdivided into n×m sub-images.
In some embodiments, at least two sub-images are masked out for steps d and e, and wherein completely different or only partially different sub-images are masked out for each registration of step c.
In some embodiments, when determining the respective similarity measure, the sub-images are weighted with an information content determined within the corresponding sub-images.
In some embodiments, identifying the sub-images that have a scene change comprises creating a ranking of the determined similarity measures; determining a set of best similarity measures based on the created ranking and determining the masked-out sub-images in the calculation thereof; and identifying the specific sub-images whose frequency exceeds a threshold value as the sub-images which have a scene change.
In some embodiments, the method further comprises subdividing the sub-images into sub-sub-images.
In some embodiments, the method further comprises determining a sub-similarity measure for each non-masked sub-image, wherein the determined similarity measures are aggregated from the sub-similarity measures.
In some embodiments, the at least one sub-image to be masked out is selected pseudo-randomly for each individual registration.
In a second aspect, an X-ray device comprises an image processing unit having a memory unit storing instructions that, when executed, cause one or more processors of the image processing unit to execute any of the foregoing methods.
In a third aspect, a tangible, non-transitory computer-readable storage medium has stored thereon a computer program which can be loaded directly into a memory unit of a control unit of an X-ray device, in particular a C-arm X-ray device, having program sections in order to execute any of the foregoing methods.
In a fourth aspect, a tangible, non-transitory computer-readable storage medium has stored thereon program sections which can be read and executed by a computer unit to execute any of the foregoing methods.
The present technology provides systems and methods suitable for registration of multiple medical image data sets. Mutual assignment of image contents between two images can be based on the use of a similarity measure. The similarity measure quantitatively describes the correspondence between two images or image regions. The similarity measure can be determined and/or maximized by means of mathematical methods, particularly by means of methods of mathematical optimization, in order to achieve the highest possible accuracy of the registration. Such a maximization of a similarity can also be understood mathematically as the minimization of a dissimilarity and can be converted to a dissimilarity by multiplication by −1, for example. Both cases will be referred to below with the term optimization and will be treated as equivalent. Such an optimization may be based on the calculation of many individual similarity measures in individual optimization steps, for example iteratively or in parallel.
A further possibility for registration is the determination of similarity measures in several small image regions, or sub-images, instead of a single similarity measure for an entire image. These similarity measures may be combined to form an overall or aggregated similarity measure according to the calculation, for example by summation. The individual similarity measures can be weighted with an information content determined within the corresponding sub-images. In this case, the information content can be determined quantitatively, for example, by the standard deviation, the variance or the entropy of the intensity or pixel values of a sub-image. This method may be particularly suitable if there are many comparatively homogeneous image regions which cannot improve or might even impair the quality of the desired registration. However, such an approach may be disadvantageous on its own if there is a scene change characterized by a particularly high information content between the two images. In the case of medical image data sets such as X-ray images, this can arise, for example, if a foreign object, such as an operating instrument or an implant, appears in only one of the two image data sets. A foreign object of this kind usually has a comparatively high contrast and thus a high information content, which leads to a concomitant high weighting in the calculation of the aggregated degree of similarity. However, if the location of the foreign object differs in the two images, or if the foreign object is present in only one of the image data sets, the result of the registration will be significantly and negatively affected.
Depending on the type of image data sets, the image data described herein may be recorded by means of a corresponding medical device. In the case of X-ray images, such medical imaging devices can be, for example, mobile or stationary X-ray C-arms, computer tomographs (CT) or other modalities from the group of X-ray-imaging medical devices. Imaging methods which are based on the principles of magnetic resonance can be, for example, magnetic resonance tomographs (MRI). The methods according to the present technology further provide for processing medical image data records which were generated by means of an ultrasonic apparatus.
First and second medical image data sets can be provided by different or the same medical imaging methods; for example, one medical image data set can be provided by a magnetic resonance tomograph, while the other image data set can be provided by an X-ray device. The methods according to the present technology also include the possibility that the medical image data sets can be recorded by means of the same recording technology, for example by means of X-rays, but the recording modalities can differ; for example, one medical image data set can be recorded by means of a CT device, while the other can be recorded by means of a C-arm. In particular, one of the image data sets to be registered can also be synthesized from an existing image data set, for example by forward projection of a synthetic, two-dimensional X-ray image from a previously obtained three-dimensional computer tomography (Digitally Reconstructed Radiograph, DRR). In this case, the other image data set can correspond to a true X-ray image which can be registered or brought into coincidence with the synthesized image as part of a 3D/2D registration. The method according to the invention can thus also be used as part of such a 3D/2D registration.
In the methods of the present technology, the two image data sets can be identical or the same for successful registration between the first and the second image data sets, so that the contents of the image data sets do not differ from one another, and consequently it is not necessary for a scene change to be present. In accordance with the methods of the present technology, a registration can also be successful if the two medical image data records differ, e.g., when a scene change is present. A scene change can be present, for example, if foreign objects are present in one of the two medical image data sets.
In the methods according to the present technology, foreign objects that may be present in one or both of the two medical image data sets may include, but are not limited to, screws, Kirschner wires, implants, clamps, tubes, instruments, scissors, scalpels or combinations thereof. According to the present technology, extraneous anatomical structures located in at least one of the two medical image data sets can also be identified as foreign objects, for example, tumors and/or hands of the surgeon during the preparation of the recording of a medical image data set. Furthermore, differences between the two medical image data sets which differ in at least one property, for example contrast, illumination, radiation dose or other image recording parameters, are also considered scene changes.
In some embodiments, the first and the second medical image data sets can each be subdivided into an equal number of sub-images (e.g., components or portions), which may in some cases be of equal size. The medical image data sets can be subdivided into any number of sub-images, but the number of sub-images of the first and second medical image data sets can be identical and of equal size. Alternatively, it is possible that individual sub-images can overlap. Alternatively, such a subdivision of the medical image data set can also be understood as a classification.
In some embodiments, a number of individual registrations, e.g., at least one registration, between the first and the second medical image data sets are performed, with a similarity measure being determined and optimized in each case. The individual registrations of the methods according to the present technology can be rigid (e.g., inelastic) registrations, but it is also possible to carry out an elastic registration. A similarity measure can be understood as a numerical value, for example, in a value range from 0 to 1, which contains information about the extent to which the images to be registered correspond to one another. A value of 0 for the similarity measure can mean that the images do not match each other in any area, for example in any sub-image. On the other hand, a value of “1” may mean that the two medical image data sets completely match, for example a match in all the sub-images.
For each individual registration, at least one sub-image of the first medical image data set, and preferably a corresponding sub-image of the second medical image data set, may be masked out by means of a random process, e.g., a stochastic or nondeterministic process, when determining the respective measure of similarity. The masking can be a marking out, an omission, an extraction, a non-consideration or a weighting of zero applied to the corresponding at least one sub-image.
A random selection process, as provided in accordance with the present technology, may be advantageous in comparison to a combinatorially complete selection process for masking out the sub-images, since such a combinatorially complete masking out of sub-images, especially if a very large number of sub-images are present, would entail a considerable additional expenditure in terms of time and computation.
It may be irrelevant within the random selection process how high the degree of randomness is or whether, for technical reasons, there is only a so-called pseudo-randomness. The methods according to the present technology can be performed suitably if a random number generator necessary for the selection process is always initialized in the same way or with the same number, the so-called “seed,” with which a random number generator is initialized.
The present technology provides that the number of individual registrations can run in series and/or parallel to one another. Running the individual registrations in parallel may generate a time advantage due to a simultaneous execution of multiple individual registrations. Preferably, the individual registrations are thus independent of one another and can take place without knowledge of the optimization steps running in the respective other individual registrations or their resulting similarity measures.
The sub-images which have a scene change during the individual registration may be identified and/or masked out. For example, those sub-images which do not have a corresponding sub-image in the respective other medical image data set can be masked out.
After identifying the sub-images that have a scene change, a final registration between the first and the second medical image data set may be performed, for example, by masking out the identified sub-images having scene changes, and/or by masking out a sub-image combination used in the individual registrations. By masking out the identified scene changes, this final registration will generally have a higher degree of similarity and thus a more accurate registration than would be the case without such masking.
In embodiments of the methods according to the present technology, the medical image data sets to be registered with one another can also be three-dimensional (3D) or volume images, so that even such image data sets can be registered with one another.
In some embodiments, the first and the second medical image data set for volume images can each be subdivided into n×m×p sub-images, this subdivision also including the condition that at least two of the three parameters can be equated.
Some embodiments can also provide that both medical image data sets represent two-dimensional images, such as projection and/or sectional images, wherein a sectional image can also be a two-dimensional image from a volume image. If two-dimensional medical image data sets are present, the first and second medical image data sets can each be subdivided into n×m sub-images, this also including the possibility that n=m.
In some embodiments, completely different or only partially different sub-images can be masked out randomly for the individual registrations. In the case of a randomly completely different masking out between, for example, two individual registrations, a set of sub-images can be masked out in a respective individual registration and, in subsequent or parallel individual registrations, a set of sub-images which were not contained in the set of masked-out sub-images of the preceding or parallel individual registrations can be masked out at random. In the case of a randomly only partially different masking of sub-images between two individual registrations, a set of sub-images can be masked out in an individual registration, and a set of sub-images can be masked out in subsequent or parallel individual registrations which contains at least one sub-image from the set of the previous or parallel masked-out sub-images of the previous individual registrations.
In some embodiments of the methods according to the present technology, the sub-images can be weighted in the determination of the similarity measures by using an information content, such as variance, standard deviation, or entropy of the intensity or pixel values, that is determined within the corresponding sub-images. This makes it possible to take little or no account of sub-images with little information content, for example largely homogeneous sub-images, when calculating the individual registrations, (e.g., to mask out such sub-images).
Some embodiments of the methods according to the present technology make it possible that, in the process of identifying sub-images having a scene change, a ranking of the similarity measures optimized during the individual registrations can first be produced. A set of the N best similarity measures, and thus a set of the N best individual registrations, can then be determined by means of the ranking. To create a frequency distribution, it is subsequently possible to calculate the number of times the individual sub-images have been masked out across the N best registrations. It is highly probable that sub-images which have been masked out particularly frequently contain a scene change, in particular since their masking out has improved the respective individual registrations. On the basis of this frequency distribution, it is thus possible to identify those sub-images whose frequency exceeds a threshold value, which can be defined in an organ program, for example. Sub-images whose frequency exceeds the threshold value can be identified as those sub-images which have a scene change, since they particularly frequently lead to an improved individual registration.
Further embodiments of the methods according to the present technology make it possible for the sub-images to be further subdivided into sub-sub-images. In some advantageous embodiments, this can be done by repeating the subdivision, individual registration, scene change identification, and final registration steps, wherein only the sub-images for which a scene change was determined in the previous pass are permitted for the repetition. These sub-images can then be subdivided into more finely resolved sub-sub-images in order to more precisely delimit the scene change, for example an introduced foreign object. Such repetition can also be carried out multiple times until, for example, the sub-sub-images correspond to individual pixels and the scene change has thus been segmented with the highest possible resolution. The methods according to the present technology can do this in particular if the area of the sub-images that exceed a threshold value for the frequency occupies an excessively large image area. An advantage of this sub-subdivision can be that a finer subdivision makes it possible to detect smaller scene changes, for example smaller foreign objects, more accurately in order to achieve a higher accuracy of the final registration.
Further example configurations can also provide that a sub-similarity measure is determined for each sub-image and that the similarity measures for the individual registrations are aggregated from these sub-similarity measures of non-masked sub-images. An advantage of these embodiments can be that the present technology can combine the previously calculated sub-similarity measures and discard a subset of sub-similarity measures much more quickly than performing a complete image re-registration with the omitted sub-images.
The present technology further provides devices such as an X-ray device, in particular a C-arm X-ray device, for example a mobile C-arm X-ray device, a computer tomograph, in particular a cone beam computer tomograph, and gantry-based systems which produce recordings of image data sets by means of X-rays. The devices according to the present technology can include an image processing unit with a memory unit on which instructions are stored which, when executed by a computing unit, bring about a method according to any of the embodiments of present technology.
A largely software-based implementation of the method has the advantage that even previously used registration methods for image recording systems can be retrofitted in a simple manner by a software update in order to operate according to the present technology. The problem disclosed herein may also be solved by a corresponding computer program product having a computer program which can be loaded directly into a memory device of an image recording system, for example a mobile C-arm, said program having program sections for executing all the steps of the method according to the present technology when the computer program is executed in the control device. In addition to the computer program, such a computer program product may optionally comprise additional components such as documentation and/or additional components, including hardware components for using the software.
A computer-readable medium, for example a memory stick, a hard disk or another portable or permanently installed data carrier, on which the program sections of the computer program which can be read in and executed by a computer unit of the control device are stored, can be used for transport to the control device and/or for storage on or in the control device. A connection to a hospital information system connected to a network, to a radiology information system or to a global network, in which systems are stored the program sections of the computer program which can be read in and executed by a computer unit of the control device, can also be used for the transport. The computer unit, and/or any of the other units disclosed herein such as an image output unit, an input unit, and/or an image processing unit, can have, for example, one or more cooperating microprocessors or the like for this purpose, each of which may be in communication with one or more non-transitory computer-readable media having computer-executable instructions stored thereon for execution of the corresponding unit functions.
Certain embodiments of the present technology will now be explained in more detail with reference to the figures.
The variant of the order of individual registrations shown in
The situation illustrated in
The C-arm 11 carries an X-ray generator 13 at one end and an X-ray image detector 12, such as a flat detector or an image amplifier, at the other end and opposite the X-ray generator 13. The C-arm 11 can be adjusted under the control of a motor in a plurality of axes in space, the axes having sensors for detecting the extent of the adjustment.
The device further comprises an image-processing unit 121, a memory unit 122, a control unit 123 and a network interface 124. By means of the network interface 124, data, such as image data sets and results of the methods according to the present technology, can be distributed or made available in a network.
The image processing unit 121 comprises a control unit 122, on which the two- or three-dimensional image data sets used for the methods according to the present technology can be stored or loaded. These image data sets can either be loaded from a server or recorded by means of the C-arm 11 before or during an intervention. The memory unit further comprises instructions which are used for carrying out the methods according to the present technology by means of a computing unit.
Furthermore, the device can contain a GUI, having an image output unit (16, 17) and an input unit 19, with which corresponding settings can be made for the image processing unit 121 in corresponding organ programs.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2021 001 022.1 | Feb 2021 | DE | national |
