Patent Application
20250213207
The invention relates to operation image alignment methods and systems, and particularly related to an operation image alignment method and system for aligning three-dimensional images and two-dimensional images.
The operation navigation system is an extremely important part in medical operation, assisting doctors to accurately operate on the lesion location during the operation. Before performing an operation, a computed tomography (CT) scanner or a magnetic resonance imaging (MRI) scanner is used to obtain three-dimensional images of an operation site, allowing doctors to accurately grasp the image contents of the lesion location. When performing the operation, registration of the operation images is required. The lesion locations in two-dimensional image (obtained during the operation) and the three-dimensional images (obtained before the operation) are aligned for dynamic image tracking by the subsequent navigation system. Therefore, reducing alignment error and alignment processing time of operation images is a concern matter to those skilled in the art.
An operation image alignment method performed by a computer system, comprising: inputting an actual three-dimensional image and an actual two-dimensional image of an actual desired part into the computer system; converting the actual three-dimensional image into multiple sets of two-dimensional projection image according to image parameters of the actual three-dimensional image through a built-in image alignment prediction model of the computer system; comparing each of the sets of two-dimensional projection image to the actual two-dimensional image to calculate an image parameter difference value for each of the sets of two-dimensional projection image, and selecting one of the sets of two-dimensional projection image to obtain a predicted rotation angle and a predicted translation, wherein the one of the sets of two-dimensional projection image has an image parameter difference value matching a preset difference value; the image alignment prediction model is an artificial intelligence model trained by a model algorithm, and multiple sets of historical images including at least one historical three-dimensional image and at least one historical two-dimensional image are used as a training data set of the image alignment prediction model.
In some embodiments, the operation image alignment method further comprising a model building step to build the image alignment prediction model, which comprises: defining historical desired regions of an historical desired part from the at least one historical three-dimensional image and the at least one historical two-dimensional image of each set of historical images, in which each of the historical desired regions includes a historical position information.
In some embodiments, the model building step further comprising: converting the at least one historical three-dimensional image of each set of historical images into at least one historical two-dimensional projection image with a first perspective or a second perspective through an image projection transformation technology; and using the historical position information of the historical desired regions as an initial position, and obtaining a historical rotation angle and a historical translation in the first perspective or the second perspective between the at least one historical three-dimensional image and the at least one historical two-dimensional image in each set of historical images through the at least one historical two-dimensional projection image.
In some embodiments, the first perspective is a side-view, the second perspective is a top-view.
In some embodiments, the image parameters include an image contour and an image gradient value.
In some embodiments, the model algorithm is one of a generative adversarial networks algorithm and a deep iterative 2D/3D registration algorithm.
In some embodiments, the operation image alignment method further comprising: using an imaging equipment to capture the actual two-dimensional image of the actual desired part, in which the actual two-dimensional image includes an actual position information.
An operation image alignment system which is to use a computer system to perform the operation image alignment method as described in any of the foregoing.
In some embodiments, the operation image alignment system further comprises an imaging equipment, which is a C-arm X-ray machine with a transmitter and a receiver.
In some embodiments, the operation image alignment system further comprises a computed tomography equipment, which is used to capture the actual desired part to obtain the actual three-dimensional image.
The aspect of the invention can be better understood from the following detailed description combined with the accompanying drawings. It should be noted that features are not drawn to scale in accordance with standard industry practice. In fact, the dimensions of each feature may be arbitrarily increased or decreased for clarity of discussion.
Embodiments of the present invention are discussed in detail below. However, it should be appreciated that the embodiments provide many applicable concepts that can be embodied in variety of specific contexts. The embodiments discussed and disclosed are for illustration only and are not intended to limit the scope of the invention.
The three-dimensional imaging equipment 120 is used to capture the three-dimensional images before operation. The three-dimensional imaging equipment 120 can be a magnetic resonance imaging (MRI) scanner, a computed tomography (CT) scanner, a positron emission tomography (PET) scanner, a single photon emission CT (SPECT) scanner, or any device that can obtain the three-dimensional images of a target. For example, patients can take computed tomography images of the spine 150 before the operation, and convert the spine 150 into a three-dimensional simulated object through the computer system 140. Therefore, a three-dimensional image with a three-dimensional simulated object can be obtained, and the at least one vertebra 151 is separate therefrom.
The two-dimensional imaging equipment 130 is used to capture the two-dimensional images during operation. The two-dimensional imaging equipment 130 is a C-arm X-ray machine with a C-frame 131, a transmitter 132 and a receiver 133. The C-frame 131 drives the transmitter 132 and the receiver 133 to rotate around the target, so that the C-arm X-ray machine can capture the two-dimensional images of the target at different angles. For example, the transmitter 132 and the receiver 133 are disposed opposite to each other, the transmitter 132 emits X-rays to the vertebrae 151, and the receiver 133 receives the X-rays passing through the vertebrae 151, thereby converting the X-rays into the two-dimensional images. The two-dimensional images include images taken from the front and back (A-P view) (i.e., top-view) and images taken from the side (lateral view) (i.e., side-view).
The computer system 140 is communicatively connected to the three-dimensional imaging equipment 120 and the two-dimensional imaging equipment 130, and data can be transmitted between each other through any wired or wireless method. The computer system 140 includes memory and processor, which can used to store multiple sets of historical images (each set of historical images includes a historical three-dimensional image IH1 and a corresponding historical two-dimensional image IH2), and used to perform image processing on these historical images to define at least one vertebra 151 from these historical images. Then, the historical three-dimensional images IH1 and the historical two-dimensional images IH2 are used as a training data set to train an artificial intelligence model, thereby building the image alignment prediction model of the invention. Among them, the training of the image alignment prediction model is achieved through an image projection conversion technology, which converts the historical three-dimensional image IH1 into historical two-dimensional projection images IPH1. Therefore, the relative position information between each the historical two-dimensional projection image IPH1 and the historical two-dimensional image IH2 with different perspectives can be obtained. In particular, each of the desired vertebrae 151 in the actual three-dimensional image IT1 and the actual two-dimensional image IT2 can be aligned together through the image alignment prediction model while the computer system 140 receives the actual three-dimensional image IT1 (captured through the three-dimensional imaging equipment 120 before operation) and the actual two-dimensional image IT2 (captured through the two-dimensional imaging equipment 130 during the operation), allowing doctors to perform subsequent operational image tracking. The computer system 140 can be a smartphone, a tablet, a personal computer, a notebook computer, a server, an industrial computer or other electronic devices with computing capabilities, and the invention is not limited thereto.
In the model building step 210, step 211 is performed first to obtain multiple sets of historical images of historical spine 150. Each set of historical images includes at least one historical three-dimensional image IH1 of the historical spine 150 (captured through the three-dimensional imaging equipment 120 in the past) and at least one historical two-dimensional image IH2 of the historical spine 150 (captured through the two-dimensional imaging equipment 130 in the past). The sources of the historical spines 150 can be taken from different people, and the historical two-dimensional image IH2 and the historical three-dimensional image IH1 captured in the past are used as the training data set for training the artificial intelligence model. In some embodiments, one historical three-dimensional image IH1 corresponds to one or more historical two-dimensional images IH2 with different perspective. For example, but not limited to, the historical two-dimensional image IH2 is an image captured from a side-view (a first perspective), or an image captured from a top-view (a second perspective).
In step 212, multiple historical desired regions (corresponding to multiple historical desired vertebrae 151) are defined from the historical two-dimensional image IH2 and the historical three-dimensional image IH1 of the historical spine 150 in each set of historical images. Please referring to
Please return to
Referring to
Returning to
Referring to
Then in step 223, the image alignment prediction model compares each of the sets of two-dimensional projection images IPT1 to the actual two-dimensional image IT2, and calculates the image parameter difference value between each of the sets of two-dimensional projection images IPT1 and the actual two-dimensional image IT2. Afterwards, the image alignment prediction model selects one of the sets of two-dimensional projection images IPT1 whose image parameter difference value matching the preset difference value as a prediction result, thereby obtaining the predicted rotation angle and the predicted translation between the desired vertebra 151 in the actual three-dimensional image IT1 and the actual two-dimensional image IT2. In this way, the operation navigation system can complete the registration of the operation images based on the alignment between the actual three-dimensional image IT1 and the actual two-dimensional image IT2.
In some embodiments, the model building step 210 further includes a discrimination step of the image alignment prediction model to train the accuracy of the image alignment prediction model. Specifically, the computer system 140 inputs an experimental three-dimensional image IE1 and an experimental two-dimensional image IE2 of the historical spine 150 into the image alignment prediction model, thereby obtaining an experimental predicted rotation angle and an experimental predicted translation between the experimental three-dimensional image IE1 and the experimental two-dimensional image IE2. Then the image alignment prediction model generates the experimental predicted projection image PE based on the experimental predicted rotation angle and the experimental predicted translation. Next, the generated experimental prediction projection image PE is used to build an image alignment discriminator model to determine whether the accuracy of the image alignment prediction model is qualified. Specifically, each of the experimental three-dimensional images IE1 and the experimental two-dimensional images IE2 are input into the image alignment prediction model as an experimental set in the training data set; comparing the experimental predicted projection image PE (which includes the experimental predicted rotation angle and experimental predicted translation between the experimental three-dimensional image IE1 and the experimental two-dimensional image IE2) predicted by the image alignment prediction model with the correct experimental two-dimensional image IE2; and training the artificial intelligence model to build the image alignment discriminator model. Therefore, when the image alignment discriminator model cannot determine the authenticity between the experimental predicted projection image PE and the correct experimental two-dimensional image IE2, the prediction of the image alignment prediction model is deemed to be accurate enough. Otherwise, the image alignment prediction model is re-predicted until the accuracy of the prediction result is qualified.
During the entire alignment processes, the image alignment prediction model uses the actual position information of the actual desired vertebra 151 in the actual three-dimensional image IT1 and the actual two-dimensional image IT2 as the initial position, and uses the image alignment discriminator model to continuously determine whether the prediction results of the image alignment model are accurate enough until the correct rotation angle and the correct translation between the actual three-dimensional image IT1 and the actual two-dimensional image IT2 are predicted. In this way, the actual three-dimensional image and the actual two-dimensional image of the actual desired vertebra 151 can finally be aligned with each other. It should be understood that establishing the image alignment discriminator model is only to improve the prediction accuracy of the image alignment prediction model. In fact, in some embodiments of the invention, even if the establishment of the image alignment discriminator model is omitted, it will not affect the image alignment between the three-dimensional images and the two-dimensional images of the invention.
In addition, through continuous prediction and adjustment, the image alignment model can perform a series of alignment processes, so that the actual three-dimensional image IT1 of the desired vertebra 151 can finally be aligned with the actual two-dimensional images IT2 of the top-view and the side-view.
The features of several embodiments are described above, so that those skilled in the art can better understand the aspects of the invention. Those skilled in the art should understand that they can easily use the invention as a basis to design or modify other processes and structures to achieve the same goals and/or achieve the same advantages as the embodiments. Those skilled in the art should also understand that these equivalent structures do not deviate from the spirit and scope of the invention, and they can make various changes, substitutions and variations without departing from the spirit and the scope of the invention.
