Patent Application
20230386101
The present invention relates generally to diagnostic imaging. More specifically, it relates to techniques for magnetic resonance (MR) image reconstruction using deep learning.
In recent years, deep learning (DL) methods for multi-coil under-sampled MR reconstruction have garnered considerable attention, with the goal of reconstructing high-quality images from a set of under-sampled k-space measurements. Unrolled neural networks (UNNs), which are a DL extension of iterative reconstructions, have gained popularity and demonstrated superior performance to conventional reconstructions based on parallel imaging and compressed sensing (PI-CS). Variational Networks (VN) and Model-based Deep Learning (MoDL) are two representative UNNs.
While UNNs significantly improved reconstructed image quality when compared to standard compressed sensing (CS) methods, one of their major drawbacks is over-smoothing, which results in the loss of high-frequency details. UNNs are typically trained using loss functions such as mean squared error (MSE) or mean absolute error (MAE). They are known, however, to be insufficiently representative of the human visual system and to be blind to critical local features such as minute details, resulting in blurry images. As a result, many researchers are currently interested in improving high-frequency visual details in DL reconstruction. The recovery of high-frequency details has primarily been studied in the context of generative adversarial networks (GAN). However, when used for MR reconstruction these approaches have several drawbacks: they are sensitive to parameters difficult to train and can produce unexpected hallucinations of artificial structures.
The present inventors have created a new technique for refining DL-based image reconstructions to improve the high-frequency details in reconstructed images. After DL-based reconstruction, this technique can be conveniently inserted in as a stand-alone post-processing stage.
There have been several studies that explored feature losses with generative adversarial networks (GANs), which are notable for generating high-quality realistic images. However, GAN-based methods are shown to have the possibility of creating hallucinations of artificial structures and increase ambiguity in the reconstructed results. Compared with GAN-based methods, the present technique does not require additional training or retraining, and the refinement quality does not depend on training data and does not create hallucination or artificial structure.
The method first estimates a kernel from the auto-calibration lines of the input k-space using a calibration method, then uses the kernel to refine the Fourier domain projection (k-space domain) of the DL reconstructed image. Optionally, the number of coils can be increased using virtual coil augmentations to further improve refinement, along with coil compression to reduce the number coil substantially to maintain computational efficacy.
This technique can be used to refine the image quality and texture of DL-based undersampled MR reconstruction. The technique visually and quantitatively increases the high-frequency features of DL reconstructions for the three datasets. It generates clearer images and better maintains minute features such as microvascular structures and tissue borders. The technique is successful in improving high-frequency image details (SSIM, GMSD) without sacrificing overall image error (PSNR), as shown by the IQMs. Due to the more defined tissue boundaries, this approach may have benefits in clinical situations, such as the identification of detailed ligaments and microscopic structures inside the joint.
This technique may be applied to various types of MR imaging including 2D imaging, 3D imaging, multi-contrast imaging, and cine imaging. Generally, this technique can be applied as a post-processing step after any type of DL-based multi-coil MRI reconstruction method to refine textures in the reconstructed image.
In one aspect, the invention provides a method for magnetic resonance imaging (MRI) comprising: acquiring under-sampled k-space measurements from an MRI apparatus using multiple receiver coils; reconstructing an MRI image from the under-sampled k-space measurements and coil sensitivity maps using an unrolled neural network; generating reconstructed multi-coil k-space data from the MRI image by multiplying the MRI image by the coil sensitivity maps followed by performing a Fourier transform; estimating a k-space null-space convolutional kernel from fully-sampled k-space measurements in autocalibration signal lines of the under-sampled k-space measurements; solving a convex optimization problem to produce refined k-space data from the k-space null-space kernel, the under-sampled k-space measurements, and the reconstructed multi-coil k-space data; producing a refined MRI image from the refined k-space data by performing an inverse Fourier transform followed by a coil combination using the coil sensitivity maps.
Preferably, estimating the k-space null-space kernel comprises performing a virtual coil augmentation, coil compression, and solving a constrained optimization problem. Preferably, solving the convex optimization problem to produce the refined k-space data from the k-space null-space kernel comprises performing a conjugate gradient (CG) algorithm.
Deep Learning based MRI image reconstruction from under-sampled measurements using unrolled neural networks has enabled accelerated MRI. These DL-based reconstructions, however, often have a loss of high-frequency details and textures. To address this problem, an image refinement method is described that uses null-space kernel to refine k-space and improve image details and textures. The method constrains the output of the DL to comply to the linear neighborhood relationship calibrated in the auto-calibration lines. The technique visually reduces the structural error in the k-space domain, and enhances the homogeneity of the k-space intensity. Consequently, the reconstructed image is sharper and has enhanced details and textures. The method is also successful in improving high-frequency image details (SSIM, GMSD) without sacrificing overall image error (PSNR).
The present technique provides a strategy for refining DL-based reconstructions that differs from prior approaches. Rather than focusing on a neural network-based training technique, it focuses on the Fourier domain projection (k-space domain) of the DL reconstruction and uses null-space kernel based techniques to improve this k-space. The method focuses on rectifying the k-space, which we discovered is effective for recovering high frequency features and textures.
Deep Learning Methods for MRI Reconstruction
Embodiments of the present invention provide a post-processing image refinement technique that follows conventional DL-based MRI reconstruction approaches.
k
i=(Six)+ϵ, (1)
where is a binary mask operator for selecting acquired k-space positions,
is the Fourier transform operator, Si is the i-th coil's complex-valued sensitivity map 104. The k-space measurements 102 and coil sensitivity maps 104 are input to multiple stages of the UNN 106 which outputs a reconstructed image 108.
By incorporating an additional regularization function, classical compressed sensing (CS) enables the reconstruction of under-sampled measurements. This can be denoted by the following minimization problem:
where Ψ is a predefined regularization function, A is a linear operator composed of sensitivity map projection, Fourier transform, and discrete sampling. {tilde over (k)} is a vector representing the obtained under-sampled k-space. This minimization seeks to reconstruct an image x that is the closest match to the measured data {tilde over (k)} and minimizes the regularization function Ψ.
A UNN 106 goes beyond CS by extending this regularization function to be data-adaptive and highly nonlinear. In UNN, the regularization function Ψ is designed as a trainable convolutional neural network (CNN), and the iterative reconstruction algorithms to solve the optimization problem are unrolled into a deep neural network.
During the training stage, the CNN's parameters are updated using gradient descent with respect to the computed loss using the fully sampled reference.
Calibration-Based Parallel MRI Recovery in K-Space Domain
To further understand the basic idea of the present refinement method, it is helpful to revisit some well-established parallel-MRI reconstruction methods applied in the k-space domain. These methods can reconstruct the image from under-sampled k-space by ensuring the self-neighborhood-consistency of k-space leveraging correlated information from multiple coils and neighboring points. A point in k-space data can be effectively expressed as a linear combination of multi-coil signals and neighboring points. Several well-known approaches for reconstructing MRI using this information include GRAPPA, SPIRiT, ESPIRiT, LORAKS, and ALOHA.
From a tiny portion of the completely sampled k-space center, which is also called the autocalibration signal (ACS), calibration-based parallel MRI approaches derive a linear relationship between adjacent and inter-coil k-space data. This relationship then is extended to reconstruct missing lines.
In an operator form, this can be expressed as:
k=
k, (3)
where k is a vector of multi-coil k-space and denotes a linear convolutional kernel operator with the linear relationship mentioned previously. Then, the reconstruction problem can be performed by solving an optimization problem given by,
minimize∥(−I)k∥2
s.t. ∥
k−y∥
2≤ϵ, (4)
where the minimization seeks for reconstructed k-space k that matches this optimization problem. Here, y is the acquired k-space data, and is a mask operator that picks the acquired locations. The kernel (
−I) is often referred to as the k-space null-space kernel since it nullifies the k-space.
One optimization approach for solving this minimization problem is to reformulate and solve it in the unconstrained Lagrangian form:
Additionally, an extra penalty term (·) can be used to further regularize the problem as:
For example, in the 1-SPIRiT model, the extra penalty term is the
1 wavelet operator, which prefers solutions that are sparse in the wavelet domain.
Refinement Method
As shown in the lower portion of
To begin, by multiplying 112 the DL reconstructed image 108 by the sensitivity maps 110 and performing the Fourier transform 114, the DL estimated multi-coil k-space 118 is obtained. Second, using calibration methods 120 like as SPIRiT, ESPIRiT, and others, the null-space kernel 122 can be obtained from the fully-sampled ACS lines. Additionally, the calibration can be improved by using virtual coil conjugate (VCC) augmentation to further constrain conjugate symmetry. This augmentation is equivalent to phase constrained reconstruction in image domain.
The image refinement is performed by performing a convex optimization 124, specifically, finding the k that minimizes an expression involving three terms:
where {tilde over (k)} is the DL estimated multi-coil k-space 118, is an operator that selects acquired locations, y is the acquired under-sampled k-space 102,
−I is the null-space kernel 122. The coefficients λ1, λ2 are configurable weighting parameters that allow us to choose the desired image quality. The appropriate selection of parameters will depend on three main factors: the uncertainty of DL reconstruction, accuracy of the null-space kernel, and the signal to noise ratio or accuracy of the acquired k-space. The convex optimization 124 can be solved by performing conjugate gradient (CG) algorithm. The solution k that minimizes this expression is the refined multi-coil k-space 126 and is then transformed to a refined MR image 132 by performing inverse Fourier Transform 128 and coil combination 130 using the sensitivity maps 110.
The goal of the refinement is to further constrain the DL reconstruction to satisfy the previously discussed null-space constraint in k-space. We observed that when values of λ2 that are too small the image converges to the original DL result, and values of λ2 that are too large produce noisy or aliased output as it converges to a parallel imaging reconstruction. Here for simplicity, we apply identical regularization settings for λ1 and λ2, resulting in λ=λ1=λ2. Likewise for R=4 cases we set λ=5, and λ=10 for R=6, where R is the parallel imaging acceleration factor.
The following experimental tests were designed to validate the refinement technique under a variety of conditions (i.e., dataset, unrolled neural networks, and under-sampling scheme):
As shown above, three subsets from publicly available datasets—two knee datasets and one brain dataset—from the multicoil track of the fast MRI dataset were used (27,28). Each dataset was separately trained and tested with Ntrain and Ntest denoting the number of subjects used for training and testing, respectively.
The DL reconstruction was performed using two unrolled neural networks—MoDL and VN. To ensure a fair comparison, we replaced convolutional networks with U-Nets in both MoDL and VN. In both networks, ESPIRiT-computed sensitivity maps were used. Due to memory constraints on the GPU, 12 unrolled blocks were used for VN, 10 unrolled blocks for MoDL, and per-pixel MAE was used for the loss function.
We under-sampled the dataset retrospectively using an equi-spaced sampling pattern with net acceleration factors of 4 and 6. (with 21 fully sampled ACS lines). We show resultant images from the MoDL network for the sake of readability. Results from VN are analogous to MoDL.
The refinement process is performed after DL reconstruction. Firstly, image reconstructed with DL was projected onto the multi-coil k-space to produce DL-estimated k-space. Secondly, the null-space kernel is derived in the ACS lines of the under-sampled k-space with the methods with following parameters:
Finally, with the three inputs—DL-estimated k-space, under-sampled k-space, and the calibrated null-space kernel—minimization of Eq. 7 outputs refined k-space, which are then reconstructed to an image with inverse Fourier Transform and coil combination using the sensitivity maps.
Evaluation Metrics
To evaluate the overall quality of the reconstructed images, widely used general image quality metrics (IQMs) such as PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index) were assessed. Results of DL and the refinement technique for each slice in the test-set were evaluated and compared. Specifically, magnitude images were normalized between 0 and 1 and IQSMs were computed for each slice. PSNR reflects the overall pixel-wise errors of the image, and SSIM shows the local image similarity with respect to the fully sampled reference.
One metrics was additionally assessed to assess the effect of the refinement technique on high-frequency image details: Gradient Magnitude Standard Deviation (GMSD). The GMSD metric represents the standard deviation of the two images' gradient magnitude similarity maps.
Results
Effect of Refinement on k-Space
Effect of Refinement on Reconstructed Images
Comparison of Refinement Using Different Null-Space Calibration Methods
The results of various refining methods using null-space kernels from the three calibration methods—SPIRiT, VCC-SPIRiT, and ESPIRiT—are shown in
Image Quality Assessment
Measures for each of the three IQMs—i.e., SSIM, PSNR, and GMSD—are shown in Table 1 for the test dataset. For PSNR, the refined result achieves results that are comparable to those with DL without refinement. On the other hand, our method outperforms the metrics for SSIM and GMSD. This suggests that the refinement can effectively recover the details without sacrificing on the overall error.
We have described a null-space-based refinement technique that may be easily added after DL reconstruction to enhance the quality and texture of a reconstructed image. Our findings imply that refining DL output using the present method may generally improve DL reconstruction as tested with various large-scale dataset and networks. Our method makes use of the constraint of k-space neighborhood consistency, which is may be easily extracted from the auto-calibration lines collected during an MRI scan.
The present technique visually and quantitatively increases the high-frequency features of DL reconstructions for the three datasets. As can be seen in
Here, we examined various null-space calibration methods for the study. By ensuring conjugate symmetry in k-space, VCC augmentation to SPIRiT (VCC-SPIRiT) can confine reconstruction to have slow varying phase and has demonstrated superior refinement compared to other approaches.
This application claims priority from U.S. Provisional Patent Application 63/346,280 filed May 26, 2022, which is incorporated herein by reference.
This invention was made with Government support under contract EB019241 awarded by the National Institutes of Health, and under contract EB009690 awarded by the National Institutes of Health. The Government has certain rights in the invention.
| Number | Date | Country | |
|---|---|---|---|
| 63346280 | May 2022 | US |
