Patent Application
20240177832
This application claims priority from a Chinese patent application number 202211502755.2 filed Nov. 28, 2022, which disclosure is incorporated herein by reference in its entirety.
Part of the present invention was disclosed in (1) a paper entitled “Virtual Contrast-Enhanced Magnetic Resonance Images Synthesis for Patients With Nasopharyngeal Carcinoma Using Multimodality-Guided Synergistic Neural Network” published in the International Journal of Radiation Oncology*Biology*Physics, Volume 112, Issue 4, Pages 1033-1044 (DOI: 10.1016/j.ijrobp.2021.11.007) on Mar. 15, 2022; and (2) a conference abstract entitled “Gadolinium-free Contrast-enhanced MRI (GFCE-MRI) Synthesis via Generalizable MHDgN-Net for Patients with Nasopharyngeal Carcinoma” for Joint Annual Meeting ISMRM-ESMRMB ISMRT 31st Annual Meeting held on May 11, 2022, and the disclosures of which are incorporated herein by reference. The paper and conference abstract are grace period inventor-originated disclosures disclosed within one year before the effective filing date of this application.
The present invention relates to a system and methods for precision tumor delineation in cancer treatment and diagnosis based on contrast agent-free, virtual contrast-enhanced magnetic resonance images (VCE-MRI) generated by a multi-hospital data-guided neural network (MHDgN-Net), in particular, using the MHDgN-Net to improve diversity of training samples and model generalizability for generation of the VCE-MRI.
Nasopharyngeal carcinoma (NPC) is a highly aggressive malignancy that is particularly popular in the population of East and Southeast Asia. Currently, the mainstay treatment for NPC is radiotherapy. Precision tumor delineation is the most critical prerequisite of a successful radiotherapeutic treatment for NPC. Contract agents such as gadolinium-based contrast agents (GBCAs) have been widely used in the current precision tumor delineation to enhance the contrast of brain magnetic resonance images (MRI) due to their excellent tumor-to-normal tissue contrast. Nonetheless, gadolinium exposure has been reported to strongly associate with allergic reactions and fatal nephrogenic systemic fibrosis (Gong et al., “Deep learning enables reduced gadolinium dose for contrast-enhanced brain MRI”. Journal of magnetic resonance imaging. 2018; 48(2):330-40).
Recently, a deep learning-assisted gadolinium-free contrast-enhanced MRI (GFCE-MRI) has been proposed as an alternative to reduce or eliminate the use of GBCAs (Kleesiek et al., “Can virtual contrast enhancement in brain MRI replace gadolinium? a feasibility study”. Investigative radiology. 2019; 54(10): 653-60; Luo et al., “Deep learning-based methods may minimize GBCA dosage in brain MRI”. European Radiology. 2021; 31: 6419-28; Chen et al., “Synthesizing MR Image Contrast Enhancement Using 3D High-resolution ConvNets”. arXiv:2104.01592. 2021 Apr. 4). However, these existing GFCE-MRI models suffer from a bench-to-bedside deficiency in low or unknown model generalizability. The effect of failing to leverage complementary information between input imaging modalities can be more prominent in the case of deeply infiltrative NPC because of intricated relationship of pixel intensity among different imaging modalities.
A need therefore exists for an improved neural network and model that at least diminishes or eliminates the disadvantages and problems described above.
Accordingly, a first aspect of the present invention provides a system for precision tumor delineation in cancer treatment or diagnosis for subjects with carcinoma built on a multimodality-guided synergistic neural network (MMgSN-Net) provided in Li et al. (2022), and incorporates a mixture model of multiple hospital training data and an external data distribution matching method to increase diversity of training samples and minimize intensity variation of external data, respectively, in order to increase the model generalizability of a multi-hospital data-guided neural network.
In the first aspect, the present system includes:
In certain embodiments, the acquired MRI data from the at least two medical institutions include MRI data acquired under different scanning conditions. Said scanning conditions may include one or more of the followings: magnetic field intensity, configuration of RF coil(s), spatial resolution of images, phase encoding levels, use of fast scanning sequence, scanning time, repetition time (TR), echo time (TE), and number of excitation (or NSA), etc.
In certain embodiments, the acquired MRI data from the at least two medical institutions include MRI data obtained from MRI scanners of different manufacturers.
In certain embodiments, the acquired MRI data include 1.5T MRI and 3T MRI data.
In certain embodiments, the T1w-MRI, T2w-MRI and CE-MRI data of different subjects are acquired on the same day in sequence.
In certain embodiments, the virtual contrast-enhanced MRI data is generated by using MRI imaging and scanning parameters of the external medical institutions to scan the subjects.
In certain embodiments, the MRI data from different subjects are randomly selected for modifying the MRI imaging and scanning parameters in order to enhance the diversity of the training dataset.
In certain embodiments, the normalization or standardization is performed by z-score normalization.
In certain embodiments, the z-score normalization is based on the image data obtained from a single subject.
In certain embodiments, the neural network is built based on an architecture of MMgSN-Net.
In certain embodiments, the MMgSN-Net is configured to become a two-input neural network including sub-components of multimodality learning modules, synthesis network, self-attention module, multi-level module and discriminator.
In certain embodiments, the T1w-MRI and T2w-MRI data are separately inputted into two different multimodality learning modules.
In certain embodiments, the two different multimodality learning modules extract features from the T1w-MRI and T2w-MRI data, respectively.
In certain embodiments, the extracted features from the T1w-MRI and T2w-MRI data by the respective multimodality learning modules are fed to the synthesis network for complementary feature selection and fusion.
In certain embodiments, the selected and fused features after the synthesis network are further subjected to self-attention module and multi-level module for focusing on an enhanced exploded structure and increasing the network insight, thereby emphasizing regions of relatively larger tumors and tissues adjacent thereto.
In certain embodiments, the features after being subjected to the self-attention module and multi-level module are used to generate virtual contrast-enhanced MRI (VCE-MRI) data.
In certain embodiments, the as-generated virtual contrast-enhanced MRI data and contrast-enhanced MRI (CE-MRI) data obtained from GCBAs-injected subjects are fed into the discriminator to train the discriminator how to differentiate images of VCE-MRI from CE-MRI, and feedback the training result to the synthesis network, thereby increasing synthesis accuracy of VCE-MRI and the decision making accuracy by the discriminator.
Detailed architecture and model development of MMgSN-Net can be referred to the Detailed Description of the Invention section hereinafter and the article entitled “Virtual Contrast-Enhanced Magnetic Resonance Images Synthesis for Patients With Nasopharyngeal Carcinoma Using Multimodality-Guided Synergistic Neural Network” by the present inventors, which disclosure is incorporated herein by reference.
By using MMgSN-Net as a basis, the generalizability of the model with respect to the MRI data from the external medical institution is increased through inputting the training dataset of MRI data obtained under different scanning conditions and after said pre-processing into the MMgSN-Net for model training and confinement, in order to establish a more generalizable model.
In certain embodiments, the present system further includes a data matching module for distribution matching of a mean value of an external dataset formed from image data of the at least one external medical institution with a mean value of the target training dataset formed from the closest training dataset to the external dataset in order to minimize intensity variation of the external dataset.
In certain embodiments, the distribution matching is performed according to the following equation:
y=(μ1/μ2)*x,
where x represents pixel value of MRI data of the external dataset; μ1 represents mean pixel value of the target training dataset; μ2 represents mean pixel value of the external dataset; y represents the matched pixel value.
In certain embodiments, the external dataset after the distribution matching has the same mean pixel value as that of the target training dataset which minimizes data distribution variation of the external dataset, thereby improving generalizability of the model after training and also reserving original scale of the image data.
In certain embodiments, the neural network can be adjusted by increasing the volume of the training dataset.
In certain embodiments, the present system further includes a classification network to screen out unfit data from the model in order to increase the model reliability.
A second aspect of the present invention provides a method for improving generalizability of a neural network for precision tumor delineation in cancer treatment or diagnosis for subjects with carcinoma based on contrast agent-free, virtual contrast-enhanced MRI data generated by the neural network, where the method includes forming a training dataset containing some of MRI data obtained from different subjects and/or acquired under different imaging and scanning conditions from multiple medical institutions to increase diversity of the training dataset followed by pre-processing to normalize or standardize image size of multiple MRI data prior to inputting the pre-processed MRI data into a multimodality-guided synergistic neural network (MMgSN-Net) for training to obtain a multi-hospital data-guided neural network (MHDgN-Net), matching an external dataset obtained from at least one external medical institution with the training dataset to obtain a matched pixel value such that both the external dataset and the training dataset have the same mean pixel value to minimize intensity variation of the external dataset, and inputting the external dataset after said matching into the MHDgN-Net for external evaluation, prior to generating virtual contrast-enhanced MRI (VCE-MRI).
In certain embodiments, only T1w-MRI and T2w-MRI data from the training dataset are matched with the corresponding T1w-MRI and T2w-MRI data from the external dataset in order to avoid using contrast-enhanced MRI (CE-MRI) data obtained from GBCAs.
In certain embodiments, the generalizability of the neural network can be validated by the external dataset after said matching.
In certain embodiments, the matching of the external dataset with the training dataset is performed by the following equation:
y=(μ1/μ2)*x,
where x represents pixel value of MRI data of the external dataset; μ1 represents mean value of the target training dataset; μ2 represents mean value of the external dataset; y represents the matched pixel value.
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. Other aspects of the present invention are disclosed as illustrated by the embodiments hereinafter.
The appended drawings, where like reference numerals refer to identical or functionally similar elements, contain figures of certain embodiments to further illustrate and clarify the above and other aspects, advantages and features of the present invention. It will be appreciated that these drawings depict embodiments of the invention and are not intended to limit its scope. The invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been depicted to scale.
It will be apparent to those skilled in the art that modifications, including additions and/or substitutions, may be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, the disclosure is written to enable one skilled in the art to practice the teachings herein without undue experimentation.
The present invention provides a multi-hospital data-guided neural network (MHDgN-Net) including at least four modules: data acquisition module, data pre-processing module, model training module, and virtual image generation module. The model training module of the present invention is primarily constructed based on the MMgSN-Net disclosed in Li et al. (2022). A more detailed architecture of MMgSN-Net is illustrated in
The proposed MHDgN-Net can be mainly divided into two stages: a mixture modeling (MM) and an external distribution matching (EDM). At the stage of MM, MRI data obtained from different medical institutions, e.g., different hospitals, are integrated to construct a mixture dataset. To keep the patient number of the mixture dataset consistent with that of each individual hospital dataset, a certain number of patients are randomly sampled from each hospital, e.g., 18 patients from each hospital. Following that, one patient is randomly excluded from the mixture dataset, i.e., only 53 patient samples remain in the mixture dataset from three different hospitals when each randomly sampled with 18 patients. A mixture model is thereby generated based on the mixture dataset.
Since different hospitals may use different imaging and scanning protocols/parameters, it will result in various intensity distribution in the mixture dataset, reducing generalizability of the mixture model. To improve the generalizability, EDM is performed which is an indirect approach by minimizing intensity variation of an external data obtained from one or more external medical institutions which is/are different from those at where the dataset for training the deep learning model (or mixture model) is generated. Such external dataset matches the mixture dataset according to the following equation:
y=(μ1/μ2)*x (1),
where x represents pixel value of the external dataset; μ1 represents slice-based overall mean pixel value of the mixture dataset; μ2 represents mean pixel value of the external dataset; y represents the matched pixel value.
After EDM, the mean pixel value of the external dataset matches the mean pixel value of the mixture dataset such that the data distribution variation is minimized, thereby indirectly reducing generalizability of the mixture model. A comparison of the synthetic VCE-MRI by MMgSN-Net with and without EDM is illustrated in
Turning to
Among 71 enrolled patients from each of Hospitals 1 to 3, each group was randomly split into 53 and 18 patients for model training; Hospital 4 was used for external evaluation. All enrolled patients were scanned with contrast agent-free T1w and T2w MRI, and GBCA-based CE-MRI. The collected T1w and T2w MRI data will be used as input for model learning; CE-MRI will be used as learning target of the model training.
To increase the diversity of the training dataset, it is possible to vary the MRI imaging/scanning parameters when acquiring some of the patients' samples, for instance, about 50% of the patients' samples are varied in terms of the imaging/scanning parameters: change T/R coil into body coil; reduce NSA; change into a faster imaging sequence; vary TR, TE value within 5-10%, etc.
After the acquisition of MRI data by the data acquisition module 110, the data will be fed to a data pre-processing module 120. In the context of deep learning model, there are usually two main methods for data normalization/standardization: 0-1 normalization and z-score normalization. Before normalization/standardization, the MRI data fed from the data acquisition module 110 are resampled. MRI data collected from different hospitals or acquired by different MRI scanners will vary in size/resolution. Bilinear interpolation can be adopted to resample the MR images with different sizes/pixels into a 256*224 configuration, for example, which can be enabled according to the following pseudocode:
cv2.resize(input_image,(224,256),interpolation=cv2.INTER_LINEAR)
After the resampling, the mean pixel value and standard deviation of the whole image volume of each patient are computed, followed by standardization of the MRI data of that patient using z-score normalization method.
Another method of normalizing the MRI data of a patient is 0-1 normalization, i.e., to map the data within the range of 0-1 for processing. The following equation is employed:
x*=(x−min)/(max−min) (2),
where x denotes the sample value; min denotes the smallest value of sample range; max denotes the largest value of sample range; x*denotes the value after normalization.
As compared to 0-1 normalization method, z-score normalization method makes the data after pre-processing in compliance with standard normal distribution, i.e., the normal value (μ) is 0; standard deviation (σ) is 1. In other words, the data is normalized as: data distribution μ=0; σ=1, where its conversion coefficient becomes x+=(x−μ)/σ. Normalization or standardization can eliminate detrimental effect arising from abnormal data, making all indicative values at the same quantitative level such that they are more comparable to each other and the accuracy of the discriminator can be greatly enhanced.
Since the image data of T1w-MRI, T2w-MRI and CE-MRI from different patients are scanned in sequence on the same day, the extracted features from the three types of sequence images have been aligned without an additional image mapping. The model training module 130 randomly takes 70% of patients' MRI data (training dataset) for performing model training, whereas the remaining 30% of patients' MRI data (internal evaluation dataset) is used for internal evaluation. In these embodiments, external dataset obtained from NPC patients of two external medical institutions are used for external (generalizability) evaluation 142. After the external evaluation, the external dataset will be fed into the trained MMgSN-Net (i.e., MHgDN-Net) by the virtual image generation module 140 for generation of the VCE-MRI.
As described herein, MMgSN-Net is used for model training according to certain embodiments. In other embodiments, other neural network or models can be used for model training. To evaluate the model training by MMgSN-Net, MRI data from NPC patients are used to carry out a series of evaluations. The synthesized images by MMgSN-Net are quantitatively evaluated with respect to GBCA enhanced T1w-MRI based on a series of evaluating metrics including, but not limited to, mean absolute error (MAE), mean squared error (MSE), structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR), which results reveal that MMgSN-Net is capable of synthesizing a substantially real VCE-MRI.
The MMgSN-Net is also subjected to qualitative evaluation by visual inspection of the ground truth (CE-MRI) and synthesized (VCE-MRI) images. The results reveal that MMgSN-Net is better in terms of model performance over the existing neural networks such as CycleGAN, U-Net, and Hi-Net.
The SGS of the synthesis network 132 according to certain embodiments is configured for leveraging complementary information between two imaging modalities in a synergistic manner. One of the common strategies to fuse learned information from the multimodality learning modules is to directly concatenate the information to different channels as input. Alternatively, in certain embodiments of the present invention, different fused features are generated by pixel-wise summation, pixel-wise product, and pixel-wise maximization separately, concatenated thereof as different channels, followed by subjecting the fused features back to the corresponding convolution layer to adaptively select useful complementary information for final VCE-MRI synthesis.
Different from a previous work by Zhou et al. (Hi-Net: Hybrid-fusion network for multimodal MR image synthesis. IEEE Trans Med Imaging 2020; 39:2772-2781), the SGS of the synthesis network 132 can communicate with each other. In certain embodiments, after each round of fuse operation, the corresponding features output from the SGS containing both T1w- and T2w-MRI information are fed back to a subsequent convolution block of the multimodality learning module. In this way, one modality of the multimodality learning module can learn information from the other modality, while the power of each individual channel of the multimodality learning module can be reduced by communication and cooperation among two different modalities. There are only two pooling layers in the multimodality learning module for optimizing the size of the features output from each of the first and second SGS, respectively. In addition, to avoid removal of critical information before feature fusion, extracted features from multimodality learning module 131 are preferably fed into the SGS of the synthesis network 132 without any pooling operation. In certain embodiments, the number of filters for the first and second SGS are 128 and 128, and 128 and 256, respectively, where the size of each filter is 3×3.
Since the nature of NPC tumors is highly aggressive, which is possible to invade adjacent healthy tissues such as neural structures and even bones of the corresponding skull base, synthesis of virtual CE-MRI data involves capturing large structure information of infiltrative tumor and/or different image regions. To maintain high computational efficiency without varying the number of convolution layers or the size of the convolution kernels, the self-attention module 133 is introduced between the second and third convolution blocks of the synthesis network 132 according to certain embodiments to capture the large size information across multiple image regions. The self-attention module 133 can be any conventional self-attention generative adversarial network such as the one proposed by Zhang et al. (Proceedings of the 36th International Conference on Machine Learning, 97:7354-7363, 2019).
To improve the image segmentation and edge detection performances of the deep learning model which are critical in distinguishing tumor from surrounding normal tissues, the multi-level module 134 is incorporated into the model learning module 130 according to certain embodiments to aggregate multi-level features by upsampling features output from the convolution layers of the synthesis network 132 to the size of the final synthetic image, followed by fusing the upsampled features through concatenation operation. A 1×1 convolution layer is applied for final VCE-MRI generation.
To minimize the difference between the “real” or ground truth (CE-MRI) and “fake” or virtual (VCE-MRI) images, the discriminator 135 is incorporated into the model learning module 130 to improve the VCE-MRI synthesis performance through adversarial learning. The discriminator 135 can be any discriminator that classifies images on the basis of whether the input image patches are real or fake such as the one disclosed by Li et al. (2022). For example, the PatchGAN-based discriminator maps an input image P to a M×N size of an output image Q. All pixels in Q can be traced back to its receptive field, where the receptive field means the “patch” in P that needs to be classified. The final image authenticity is determined by averaging the M×N results in Q). In certain embodiments, the discriminator 135 comprises a plurality of convolution layers each convolution filter being in 3×3 size, batch normalization and Leaky Rectified Linear Unit (Leaky ReLu) layers each being arranged between two convolution layers. The input image P (two-dimensional axial slices) having a matrix size of 256×224 is used and linearly normalized to a range of [−1,1]. The output image Q will have a size of 16×14 matching the size of the final synthetic VCE-MRI. Mean square error (MSE) loss is used as the loss function of the PatchGAN-based discriminator for distinguishing real patches from fake patches and Adam algorithm is used to optimize the generated model according to certain embodiments. More details of the structure of PatchGAN-based discriminator are illustrated in
Turning to
51.69 ± 8.59
0.87 ± 0.03
33.42 ± 1.84
Hospital 4
95.98 ± 22.12
23869.40 ± 9309.05
0.72 ± 0.05
28.93 ± 1.56
20.52 ± 7.36
0.96 ± 0.02
44.10 ± 3.13
Hospital 4
104.18 ± 39.18
0.76 ± 0.08
31.61 ± 3.51
Hospital 4
72.14 ± 23.76
0.81 ± 0.04
29.25 ± 2.72
17.45 ± 6.16
10408.60 ± 4817.63
68.60 ± 19.68
As seen in Table 2, the MHDgN-Net achieved comparable internal quantitative results on all internal (affiliated hospitals) hospital dataset with respect to the models trained by single hospital dataset. The lower the MAE and MSE are, or the higher the SSIM and PSNR are, the higher is the generalizability of the neural network/model. The results show that models trained by single-hospital dataset can perform well on training internal hospital data, but with poor generalizability to other hospital images. On the other hand, the MHDgN-Net has higher generalizability against different single-hospital dataset trained models. MHDgN-Net also achieved the best result on external hospital dataset (Hospital 4) compared to those trained by single-hospital dataset (Hospitals 1-3). The model trained by Hospital 2 dataset is better than the other two models trained by single-hospital dataset (Hospitals 1 and 3) because Hospital 2 dataset has relatively smaller intensity value than the other two single-hospital datasets.
To increase the accuracy of the trained model, the training dataset and external dataset can be varied to include more clinically relevant data such as different tumor stages, morphologies, and sizes of different patients and/or different types of tumors.
In certain embodiments, an additional classification network may be incorporated into the present system to decide whether a patient or a population of patients is suitable for being assessed by contrast agent-free, virtual contrast-enhanced MRI generated by the present system. For instance, the performance of deep learning neural network in the clinical application may be affected by the size and/or morphology of some rarely seen tumors, or the initial training dataset of the present invention does not include the data of the relevant cases to these rarely seen tumors, or the volume of available clinical data of these rarely seen cases is limited. If this kind of deep learning neural network is directly applied in clinical evaluation of these patients, the corresponding MRI data from them could not be considered and effectively enhanced by such neural network. Thus, the introduction of the classification network can screen out these rarely seen cases such that the reliability of these VCE-MRI data can be increased.
From the perspective of data acquisition, the generalizability of the present neural network/model can be improved by increasing the diversity of the training data such as increasing the number of medical institutions, including more types of MRI scanner, utilizing a number of clinically applicable imaging and scanning parameters, etc.
From the perspective of data matching, the generalizability of the present neural network/model can be improved by matching the data distribution of the external dataset with that of the training dataset, in turn decreasing the distribution variation between the external and training datasets.
A better performance on the external dataset by the present neural network/model trained with an expanded training dataset and a decreased variation between the external and training datasets is observed, suggesting that the present invention is more generalizable and suitable for MRI data of external medical institutions compared to the models trained by dataset from single medical institution.
It should be understood that the above embodiments and examples are intended to assist the illustration of the present invention, but should not be considered to limit the scope of the present invention. Any neural network or model within the knowledge of an ordinary skilled artisan in the relevant field that are able to generate contrast agent-free, virtual contrast-enhanced MRI based on T1w- and T2w-MRI data as training dataset and CE-MRI as learning target with a data distribution mapping between external and training datasets without departing the objectives and spirit of the present invention should also fall within the scope of the present invention and be used to enable the present invention.
The following descriptions further illustrate how the present system is established, trained and modified according to certain embodiments and its performance in tumor delineation as various examples and be supported by corresponding experiments and comparative results, which should not be considered to limit the scope of the present invention thereto.
In the following examples, the architecture of multimodality-guided synergistic neural network (MMgSN-Net) capable of optimizing complementary features between multiparametric MR modalities, including contrast-free T1w and T2w images, for VCE-MRI synthesis, and its performance in terms of effectiveness with respect to some state-of-art deep learning (DL) models via a series of evaluating metrics will be illustrated. The authenticity of the synthesized VCE-MRI images was assessed by seven board-certified radiation oncologists from four hospitals via Turing tests. Corresponding MRI data from NPC patients of four hospitals were used.
Multi-parametric MR images, including T1w, T2w and CE-MRI, were retrospectively retrieved from 64 biopsy-proven (stages I-IVb) NPC patients who received RT at Hong Kong Queen Mary Elizabeth Hospital between 2012 and 2016. Patient consent was waived due to the retrospective nature of this study. All MR images were acquired under a 1.5 Tesla MRI scanner (Avanto, Siemens, Germany). Acquisition parameters for the T1w and CE-MRI include: repetition time (TR): 562-739 ms; echo time (TE): 13-17 ms; matrix: 256-320; slice thickness: 3.3-4.0 mm; voxel size 0.75-0.94 mm. In particular, the CE-T1w MR images were acquired less than 30 seconds post GBCA injection (Gd-DOTA, 0.2 ml/kg). The T2w MR images were acquired using the short tau inversion recovery (STIR) sequence with the following acquisition parameters: TR: 7640 ms; TE: 97 ms; inversion time: 165 ms; matrix: 320; slice thickness: 4.0 mm; voxel size 0.75 mm.
The multimodality-guided synergistic neural network (MMgSN-Net) was configured for VCE-MRI synthesis. The MMgSN-Net consists of five key modules: (i) multimodality learning module, (ii) synergistic guidance system (SGS), (iii) self-attention module, (iv) multi-level module, and (v) discriminator.
This module was devised to unravel tumor-related imaging features from each of the input MR modalities, overcoming the limitation of single modality-based VCE-MRI synthesis. As indicated in
This component was specifically designed for leveraging complementary information between the two studied imaging modalities in a synergistic manner.
To fuse the learned information from multimodality learning modules, a common strategy is to directly concatenate the information to different channels as input. Alternative combination methods include pixel-wise summation, pixel-wise product, and pixel-wise maximization. Similar to a previous work by Zhou et al. (Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis. IEEE Trans Med Imaging. 2020; 39(9): 2772-2781), pixel-wise summation, pixel-wise product, and pixel-wise maximization were first used separately to generate different fused features. Subsequently, the fused features were concatenated as different channels, followed by subjecting to a convolution layer to adaptively select useful complementary information for final VCE-MRI synthesis.
However, there are several differences between the present SGS and Zhou's. Firstly, in Zhou's work, separate information extractors learn the features from each input modality individually, and the extractors cannot communicate with each other, which may limit complementary information learning. In contrast, as inspired by the knowledge distillation concept where a master network modulates the learning activity of an assistant network, the present SGS was used as a supervisor to fuse the learned information from each modality, and after the fuse operation, the output features from the present SGS contained both T1w- and T2w-MRI data. The fused information was fed back to the next convolution block of the multimodality learning module to guide complementary information selection. In this way, the multimodality learning module can aware the information from the other modality, and the power of each individual multimodality learning module was further harnessed by communication and cooperation among the two modules in learning the complementary information for VCE-MRI synthesis. The fused features were not only fed directly back to the second convolution block of each input channel in the multimodality learning model, but also sent to the third convolution block via the adoption of an additional pooling layer optimizing the size of output features from the first SGS. Secondly, the present MMgSN-Net contained only two SGSs and two pooling layers that fuse and down-sample the extracted features, acting as the encoders of the synthesis network. The size of the SGS filters is 3×3, and the number of filters for the first and second SGS was 128 & 128, and 128 & 256, respectively. Thirdly, the extracted features from the multimodality learning module was fed into the SGS without any pooling operation to avoid removal of critical information prior to feature fusion.
In a convolutional neural network, large-size tissues across intra-slice image regions are captured by the convolution operator. As the field of the convolution operator is merely locally receptive, optimization algorithms may encounter difficulty in searching for the optimum parameter values when capturing the large-size tissues. To this end, there may be two solutions: either using multiple convolution layers or increasing the size of the convolution kernels. However, both solutions would degrade the computational efficiency. A balance between the ability to capture the large size information and the computational efficiency can be achieved by a self-attention mechanism, which calculates the response at a position as a weighted sum of the features at all positions.
Since NPC tumors can be highly aggressive, which presents a high tendency to invade nearby healthy tissues like neural structures and bony skull base, and the size of tumor can be large and exists across different image regions, if the MMgSN-Net is provided with limited convolutional kernel size only for VCE-MRI synthesis from these NPC tumor images, the algorithms will encounter difficulty in capturing this large structural information, for example, the shape of infiltrative tumor. Therefore, a self-attention module was introduced into the MMgSN-Net to capture the large size information across image regions, enabling MMgSN-Net to faithfully preserve the shape of large anatomic structures. The self-attention module was inserted between the second and third convolution block of the synthesis network decoder, and it may be in the same configuration as that in Zhang et al. (Self-Attention Generative Adversarial Networks. In: International Conference on Machine Learning. PMLR; 2019; 7354-7363).
Multi-level feature integration has been widely applied in areas of image segmentation and edge detection. Several studies have shown that integrating features from multiple deep layers can improve the performance in image segmentation and, more remarkably, in edge detection. In VCE-MRI synthesis, edge information is critical for discriminating the tumor from surrounding normal tissues. Thus, a multi-level module was utilized in this model to aggregate the multi-level features. In MMgSN-Net, upsampling for the output features was performed on each side of the decoders to the size of the output image. Subsequently, the upsampled features were fused through a concatenation operation and a 1×1 convolution layer was used for final output generation.
A discriminator was utilized in MMgSN-Net to distinguish synthetic images from real CE-MRI, thus to improve the VCE-MRI synthesis performance through adversarial learning. An overall structure of the discriminator is illustrated in
(C) Implementation Details of MMgSN-Net on VCE-MRI Synthesis from NPC Tumor MR Images:
All the T1w, T2w and CE-MRI images for each NPC patient acquired for radiotherapy purpose were well-aligned. Rigid registration was applied to fine-tune the alignment, when necessary. Triangle thresholding was performed to eliminate background noise from all MR images, which may otherwise be mistakenly learned by the deep learning network and lead to model performance degradation. A total of 35 patients were used for model training, whereas 29 patients were employed for model testing. Two-dimensional axial slices with a matrix size of 256×224 were adopted to acquire knowledge information from the T1w and T2w images for mapping the CE-MRI. Prior to model training, all images were linearly normalized to a range of [−1,1]. The T1w and T2w MR images were used as inputs to the network, and the CE-MRI was used as learning targets.
The L1 loss between the synthetic VCE-MRI and the corresponding real GBCA-enhanced CE-MRI was deployed as a loss function of the synthesis network (SGS). Mean square error (MSE) loss was used as the loss function of the PatchGAN-based discriminator for distinguishing between real and fake patches. The Adam algorithm was utilized to optimize the generated model. The network was trained under a fixed learning rate of 0.0002 with 200 epochs, with the batch size of 1. The code was implemented in the PyTorch library using an NVIDIA RTX 3090 graphic card.
The effectiveness of the MMgSN-Net was assessed quantitatively using four widely adopted evaluating metrics in the areas of medical imaging synthesis including mean absolute error (MAE), mean squared error (MSE), structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR) and compared against three state-of-the-art image synthesis networks: CycleGAN (Zhu et al., Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition; 2017:2223-2232), Hi-Net (Zhou et al., 2020), and U-Net (Ronneberger and Fischer, U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N, Hornegger J, Wells W M, et al., eds. Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. Lecture Notes in Computer Science. Cham: Springer International Publishing; 2015: 234-241). These evaluating metrics can be expressed by the following equations:
where N is the number of pixels in each image slice; y(x) and g(x) denote the ground truth image and synthetic VCE-MRI, respectively. μy(x), μg(x) and σy(x), σg(x) are the means and variances of the ground truth image and the synthetic image, while σy(x)g(x) is the covariance of y(x) and g(x). c1=(k1L)2 and c2=(k2L)2 are two variables used to stabilize the division by the weak denominator, and L is the dynamic range of the pixel values. Here, L=4095, k1=0.01, and k2=0.03 were set by default.
In this comparative study, the CycleGAN network which only supports single input channel was trained with a paired dataset by using T1w and T2w images separately (referred to as “CycleGAN_T1w” and “CycleGAN_T2w” in
Qualitative evaluation of the synthetic VCE-MRI by the MMgSN-Net (
Lastly, the Turing test, a long-established test in areas of artificial intelligence for determining the capability of a machine to exhibit intelligent human behavior, was used to assess authenticity of the synthetic VCE-MRI by the MMgSN-Net. In that test, seven board-certified radiation oncologists from four hospitals participated in discriminating the synthetic VCE-MRI from the real CE-MRI. 5 patients were randomly chosen from the test set for the Turing test. For each patient, 10 tumor-bearing image slices were randomly selected (5 ground truth CE-MRI plus 5 paired synthetic VCE-MRI) and presented to the participating oncologists in a random order. The oncologists were blinded with respect to the relative proportions of ground truth and synthetic images. Additionally, the participants were asked to provide justifications when determining a synthetic case, such that potential limitations of the MMgSN-Net can be realized.
Table 3 summarizes the results of quantitative comparisons between the MMgSN-Net and the comparative state-of-the-art DL networks for both whole image and tumor regions, in aspects of MAE, MSE, PSNR, and SSIM. For MMgSN-Net, the mean (±standard deviation [SD]) of the MAE, MSE, PSNR, and SSIM for the synthesized VCEMRI relative to the ground truth CE-MRI were calculated to be 44.50±13.01, 9193.22±5405.00, 0.887±0.042, and 33.17±2.14 for whole image and 110.31±20.69, 25924.77±10385.70, 0.706±0.073,
28.74±1.52 for tumor regions, respectively. The MMgSN-Net significantly outperformed all the comparative networks in all studied aspects (p<0.05). Among the comparative state-of-the-art networks, on the other hand, U-Net obtained the best performance in all four evaluating aspects, while the CycleGAN models (both CycleGAN_T1w and CycleGAN_T2w) underperformed the others.
Overall, in comparison with the state-of-the-art networks, the MMgSN-Net achieved outstandingly, with mean MAE improvements of 13.07% versus the Hi-Net, 3.47% versus the multi-channel U-Net, 31.32% versus the CycleGAN_T1w, and 30.40% versus the CycleGAN_T2w.
As seen in
In respect of the synthetic images generated from the U-Net models, they are relatively blurry throughout the images (
In respect of the Hi-Net predicted VCE-MRI (
For the two CycleGAN models (insets (i) and (j) in
In summary, the results of the quantitative evaluations demonstrate that the MMgSN-Net outperformed all the comparing networks for both whole image and tumor regions (Table 3), yielding the top-ranked scores in averaged MAE (44.50±13.01, 110.31±20.69), MSE (9193.22±5405.00, 25924.77±10385.70), SSIM (0.887±0.042, 0.706±0.073), and PSNR (33.17±2.14, 28.74±1.52) for whole image and local tumor regions, respectively. This is in line with findings of our qualitative evaluation, where the synthetic images predicted by the MMgSN-Net (
Intriguingly, it was observed that inputting single T2w MR images yielded better performance in both U-Net and CycleGAN networks than when using single T1w MR images (Table 3). A possible explanation would be related to the superiority of T2w MR images in revealing hyperintensity or inhomogeneity information on various pathologies, such as in peripheral edema and tumor necrosis, which makes T2w MR images contribute to more valuable information on pathology-related contrast enhancement for VCE-MRI synthesis, compared to contrast-free T1w MRI. This finding is also consistent with a brain tumor study conducted by Kleesiek et al. (2019), who reported that T2w MR images provided more useful information for VCE-MRI synthesis. Kleesiek et al. (2019) trained a 3D BayesUNet using multi-parametric MR modalities of 47 contrast-enhanced samples and obtained a mean SSIM of 0.862±0.029. Another work by Gong et al. (2018) reported a mean SSIM value of 0.85 □ 0.07 using a U-Net model that was trained on 10% GBCA-dose CE-MRI and contrast-free T1w MR images of 10 patients with brain diseases. In these models, information in different input modalities was simply concatenated into different channels without emphasis on potential interaction of features between the modalities. While satisfying in brain imaging, their models were deficient in leveraging complementary information between input imaging modalities, which could be more detrimental in the case of deeply infiltrative NPC. In comparison, the MMgSN-Net achieved a higher mean SSIM of 0.887±0.042 after training with 35 samples using both T1w and T2w MR images. To a large extent, this improvement in SSIM was mainly attributable to the capacity of the MMgSN-Net in unraveling complementary information from individual unique imaging modalities for VCE-MRI synthesis.
Turning to
Table 4 below summarizes quantitative results of the Turing tests from the seven participating oncologists. In Hospital 1, the two oncologists failed to differentiate between the real and virtual contrast-enhanced MR images in approximately half of the cases, with an accuracy of 52% and 42% for Oncologists 1 and 2, respectively. They reported that their decisions were mostly based on the clarity of the alveoli and blood vessels, as well as the texture of the muscles and cerebellum. In Hospital 2, the two oncologists raised the difficulties in discriminating the real and fake images based on the irregularly shaped tumor structures. For this reason, they made their decisions according to the anatomical structures and image signal intensities during the Turing test, resulting in an accuracy of 58% and 52% for Oncologists 3 and 4 from Hospital 2, respectively. In Hospital 3, discussion sessions were held between the Oncologist 5 and 6, in view of the heavy clinical workload. An overall accuracy of 58% was reported based on their judgements. They reported that their decisions were made based on the differences between the parotid gland and non-vascular tissues. In Hospital 4, Oncologist 7 correctly identified only 13, leading to an accuracy of 26%, and was unable to make decisions for another 13 images. Overall, the average accuracy of the seven oncologists was 49.43%, which is in close approximation to a random guess accuracy (i.e., 50%).
To identify the importance of the key components in the MMgSN-Net, three ablation studies were conducted. First, to evaluate the importance of the SGS, it was replaced by a concatenation operation. The learned features from individual multimodality learning modules were directly concatenated without performing feature selection. Second, to validate the importance of the multi-level module, the synthesis performance of full MMgSN-Net was compared with that absent the multi-level module. Third, to verify the importance of the self-attention module, it was removed and compared the modified version with the full version of the MMgSN-Net. Evaluating metrics in each comparative pair of the ablation studies were analyzed.
However, for simplicity, only the results of MAE were provided. First, after replacing the SGS with the concatenation operation, the MAE increased from 44.50±13.01 to 45.43±12.97 (p<0.05), implying that the SGS contributed to accuracy improvement. Second, after excluding the multi-level module, the MAE increased from 44.50±13.01 to 45.22±13.04 (p<0.05), suggesting that the multi-level module enhanced the synthesis performance of MMgSN-Net. Third, after removing the self-attention module, the MAE increased from 44.50±13.01 to 45.89±13.02 (p<0.05), indicating that the self-attention module was helpful in capturing long-term dependencies.
However, the results of the Turing test may underscore the reliability of the MMgSN-Net. In a study conducted by Kleesiek et al. (2019), two resident radiologists were invited to distinguish 10 synthetic MR images from another 10 real CE--MRI, chosen in a random manner. The radiologists correctly discriminated between the real and synthetic images in 80% and 90% of cases, respectively. In contrast, in the Turing test of the present disclosure, seven experienced oncologists from multiple hospitals were merely able to correctly classify 49.43% of the presented images, suggesting a higher authenticity of the synthesized VCE-MRI by the MMgSN-Net. It is noteworthy that the high authenticity of the present MMgSN-Net can be observed in both tumor-bearing and tumor-free MR slices. In tumor-bearing slices, the MMgSN-Net model provided comparable tumor visualization with respect to the ground-truth (
However, the MMgSN-Net used in these examples was only trained and validated using a small-sized NPC data from the same MRI scanner at a single medical institution, leading to synthesis failure arising from limited training samples for specific patients. An example of this kind of failure is illustrated in
The MMgSN-Net only trained with T1w- and T2w-MRI data may also limit the performance of synthesis network since T1w- and T2w-MRI may not provide complete information for VCE-MRI synthesis. One solution may be to include more MR modalities, e.g., diffusion-weighted MRI, as input to the network. A more homogeneous dataset for model training or a larger dataset from different MRI scanners and/or from different medical institutions likely minimizes “data bias”, in turn increasing generalizability of the trained model.
Although the invention has been described in terms of certain embodiments, other embodiments apparent to those of ordinary skill in the art are also within the scope of this invention. Accordingly, the scope of the invention is intended to be defined only by the claims which follow.
The VCE-MRI generated by the present invention does not only apply to precision tumor delineation in radiotherapy for cancer patients, but also to tumor staging based on the generated VCE-MRI.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202211502755.2 | Nov 2022 | CN | national |
