SELF-SUPERVISED LEARNING FOR ARTIFICIAL INTELLIGENCE-BASED SYSTEMS FOR MEDICAL IMAGING ANALYSIS

Abstract

Systems and methods for training an artificial intelligence-based system using self-supervised learning are provided. For each respective training medical image of a set of unannotated training medical images, the following steps are performed. A first augmented image is generated by applying a first augmentation operation to the respective training medical image. A second augmented image is generated by applying a second augmentation operation to the respective training medical image. A first representation vector is created from the first augmented image using an encoder network. A second representation vector is created from the second augmented image using the encoder network. The first representation vector is mapped to first cluster codes. The second representation vector is mapped to second cluster codes. The encoder network is optimized using the first and second representation vectors and the first and second cluster codes.

Description

TECHNICAL FIELD

The present invention relates generally to training of artificial intelligence-based systems, and in particular to self-supervised learning for artificial intelligence-based systems for medical imaging analysis.

BACKGROUND

Often times, artificial intelligence-based systems are trained for medical imaging analysis (e.g., detection, segmentation, or classification of anatomies or abnormalities) on relatively small sets of training data. This often results in limited performance and generalizability on unseen data during the inference stage, as the appearance of certain anatomies and abnormalities is highly variable and may not be fully captured in such small datasets. While such artificial intelligence-based systems may be trained using relatively large sets of training data, the availability of such training data is limited due to strong privacy regulations and the high cost of annotating the training data.

BRIEF SUMMARY OF THE INVENTION

In accordance with one or more embodiments, systems and methods for training an artificial intelligence-based system using self-supervised learning are provided. For each respective training medical image of a set of unannotated training medical images, the following steps are performed. A first augmented image is generated by applying a first augmentation operation to the respective training medical image. A second augmented image is generated by applying a second augmentation operation to the respective training medical image. A first representation vector is created from the first augmented image using an encoder network. A second representation vector is created from the second augmented image using the encoder network. The first representation vector is mapped to first cluster codes. The second representation vector is mapped to second cluster codes. The encoder network is optimized using the first and second representation vectors and the first and second cluster codes.

In one embodiment, the encoder network is optimized to find cluster codes that maximize a similarity between representation vectors and cluster prototypes. The encoder network may be optimized by calculating a first loss based on the first representation vectors and the second cluster codes, calculating a second loss based on the second representation vectors and the first cluster codes, and combining the first loss and the second loss. In one embodiment, the encoder network is optimized by optimizing a loss function by clustering based on a modality of each of the training medical images.

In one embodiment, one of the first augmentation operation or the second augmentation operation comprises an energy-based augmentation. The energy-based augmentation is performed by decomposing an image into a plurality of energy bands, computing an energy value for each of the plurality of energy bands, and transforming the image based on the energy values. The set of unannotated training medical images may comprise a plurality of subsets of training medical images each associated with different sites. The encoder network may be continuously optimized with the plurality of subsets of training medical images using federated learning and continual learning.

In one embodiment, annotated training data comprising annotated training medical images and corresponding annotations is received. The annotated training medical images are mapped to predicted annotations using a machine learning based network based on features of the encoder network. The encoder network is optimized to minimize a distance of the predicted annotations to the corresponding annotations.

In one embodiment, an input medical image is received. A representation vector is created from the input medical image using the optimized encoder network. A medical imaging analysis task is performed for the input medical image based on the representation vector. Results of the medical imaging analysis task are output. In one embodiment, the medical imaging analysis task is performed by decoding the representation vector using a decoder network to perform the medical imaging analysis task. The medical imaging analysis task may comprise at least one of detection, segmentation, or classification of an anatomy or abnormality in the input medical image.

These and other advantages of the invention will be apparent to those of ordinary skill in the art by reference to the following detailed description and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a method for self-supervised learning for training an artificial intelligence-based system, in accordance with one or more embodiments;

FIG. 2 shows a framework for self-supervised learning for training an artificial intelligence-based system, in accordance with one or more embodiments;

FIG. 3 shows a method for performing a medical imaging analysis task using a trained encoder network, in accordance with one or more embodiments;

FIG. 4 shows a graph comparing performance of conventional artificial intelligence-based systems with an artificial intelligence-based system pretrained according to the self-supervised learning in accordance with embodiments described herein;

FIG. 5 shows a graph illustrating the robustness of an artificial intelligence-based system pretrained according to the self-supervised learning in accordance with embodiments described herein;

FIG. 6 shows graphs comparing the performance of artificial intelligence-based systems trained using only annotations and the performance of artificial intelligence-based systems trained using annotations with self-supervised learning in accordance with embodiments described herein;

FIG. 7 shows a framework for self-supervised learning performed with federated learning, in accordance with one or more embodiments;

FIG. 8 shows an exemplary artificial neural network that may be used to implement one or more embodiments;

FIG. 9 shows a convolutional neural network that may be used to implement one or more embodiments; and

FIG. 10 shows a high-level block diagram of a computer that may be used to implement one or more embodiments.

DETAILED DESCRIPTION

The present invention generally relates to methods and systems for self-supervised learning for artificial intelligence-based systems for medical imaging analysis. Embodiments of the present invention are described herein to give a visual understanding of such methods and systems. A digital image is often composed of digital representations of one or more objects (or shapes). The digital representation of an object is often described herein in terms of identifying and manipulating the objects. Such manipulations are virtual manipulations accomplished in the memory or other circuitry/hardware of a computer system. Accordingly, is to be understood that embodiments of the present invention may be performed within a computer system using data stored within the computer system.

Self-supervised learning is a type of learning using unlabeled training images. In accordance with embodiments described herein, self-supervised learning utilizes a set of arbitrary N-dimensional training medical images, which can scale to the thousands, millions, billions, or more. Certain embodiments described herein are based on the SWaV (swapping assignments between views) framework, but modified to jointly learn from multiple imaging modalities by a contrastive clustering loss function. Further, embodiments described herein provided for a hybrid self-supervised/supervised learning approach to incorporate annotations of annotated training images in the self-supervised learning framework. In some embodiments, self-supervised learning can be deployed in the context of federated learning and continual learning.

Training (or pretraining) of an artificial intelligence-based system, such as, e.g., a machine learning-based system, by self-supervised learning is described herein with respect to FIGS. 1-2 below. Such training of the artificial intelligence-based system is performed during a prior offline or training stage. Once trained, the trained artificial intelligence-based system is applied to perform a medical imaging analysis task (e.g., detection, segmentation, or classification of anatomies or abnormalities) during an online or inference stage, as described with respect to FIG. 3 below.

FIG. 1 shows a method 100 for self-supervised learning for training an artificial intelligence-based system, in accordance with one or more embodiments. The steps of method 100 may be performed by one or more suitable computing devices, such as, e.g., computer 1002 of FIG. 10. Method 100 is performed during a prior offline or training stage. FIG. 2 shows a framework 200 for self-supervised learning for training an artificial intelligence-based system, in accordance with one or more embodiments. FIG. 1 and FIG. 2 will be described together.

At step 102 of FIG. 1, a set of unannotated (also referred to as unlabeled) training medical images is received. FIG. 2 shows an exemplary training medical input image I 202 sampled from a set of training medical images. The set of training medical images may comprise any number of unannotated training medical images. The set of training medical images may be a batch of training medical images from a larger set of training medical images. The training medical images may depict, for example, anatomy and/or abnormalities of a patient.

In one embodiment, the training medical images in the set are of a plurality of different modalities, such as, e.g., MRI (magnetic resonance imaging), CT (computed tomography), ultrasound, x-ray, and/or any other medical imaging modality or combinations of medical imaging modalities. The set of training medical images may comprise 2D (two dimensional) images and/or 3D (three dimensional) volumes, and each training medical image may comprise a single image or a plurality of images (e.g., channels). The set of training medical images may be received directly from an image acquisition device, such as, e.g., a CT scanner, as the training medical images are acquired, or can be received by loading previously acquired training medical images from a storage or memory of a computer system (e.g., a PACS (picture archiving and communication system)) or by receiving medical images that have been transmitted from a remote computer system.

Method 100 then proceeds to step 104 to perform steps 104-114 for a first respective training medical image in the set of unannotated training medical images.

At step 104 of FIG. 1, a first augmented image is generated by applying a first augmentation operation to the respective training medical image and, at step 106 of FIG. 1, a second augmented image is generated by applying a second augmentation operation to the respective training medical image. In one example, as shown in FIG. 2, a first augmented image I₁ 206 is generated by applying a first augmentation operation t₁ to training medical input image 1202 and a second augmented image I₂ 208 is generated by applying a second augmentation operation t₂ to training medical input image 1202. Augmentation operations t₁ and t₂ are selected from a set of augmentation operations T.

In one embodiment, the first augmentation operation and the second augmentation operation are different augmentation operations. By applying both the first augmentation operation and the second augmentation operation to the respective training medical image, different augmented images are generated that correlate with each other. The first augmentation operation and the second augmentation operation may be any suitable operation or operations for transforming the respective training medical image. The first augmentation operation and the second augmentation operation may comprise any other suitable augmentation operation, such as, e.g., 1) crop and resize to original resolution, 2) crop, resize to original resolution, and flip, 3) color distortion (drop), 4) color distortion (jitter), 5) rotation (e.g., 90 degrees, 180 degrees, 270 degrees), 6) cutout, 7) Gaussian noise, 8) Gaussian blur, 9) Sobel filtering, or a combination thereof. In one embodiment, the first augmentation operation and the second augmentation operation are compositional such that each operation comprises or is composed of a plurality of operations. For example, the first augmentation operation may comprise rotation, rescaling, intensity variation, and cropping and the second augmentation operation may comprise rotation and cropping.

In one embodiment, one of the first augmentation operation or the second augmentation operation comprises an energy-based augmentation where, based on different energy bands of the respective training medical image, statistics may be computed and used to vary the image reconstruction weighting factor on each band. In energy-based augmentation, an image I is decomposed into B energy bands B⁽¹⁾, B⁽²⁾, . . . , B^(B) using Gaussian filtering. For each band 1≤i≤B, the energy value e_i(I, Ω) is computed as the brightness dispersion in a predefined image region defined by Ω(e.g., the entire image). Image I is then normalized to transform the image based on the energy values e_i(I, Ω) as follows:

$\begin{array}{cc}\hat{I}\left(\Omega \right)={\sum }_{i=1}^B{\lambda }_i\left(\Omega \right)I^{\left(i\right)}={\sum }_{i=1}^B\frac{e_i^{ref}\left(\Omega \right)}{e_i\left(I,\Omega \right)}{I}^{\left(i\right)}& \left(\mathrm{Equation}1\right)\end{array}$

where

$e_i^{\mathrm{ref}}\left(\Omega \right)=\frac{1}{R}{\sum }_{k=1}^R{e}_i\left(I_k,\Omega \right)$

denotes the reference energy value on band i, with I₁, I₂, . . . , I_R denoting R preselected reference images. In one example, R is set to 1000 and image I is augmented using a variable reference energy e_i^ref (Ω) for any given band 1≤i≤B around the mean value. On each band, the distribution of the band R reference value is modelled using a Gaussian distribution and the value of the reference energy is sampled from the range band [−σ, +σ] around the mean energy value.

At step 108 of FIG. 1, a first representation vector is extracted from the first augmented image using an encoder network and, at step 110 of FIG. 1, a second representation vector is extracted from the second augmented image using the encoder network. In one example, as shown in FIG. 2, a first representation vector Z₁ 214 is extracted from the first augmented image I₁ 206 using encoder network ƒ_θ212 and a second representation vector Z₂ 216 is extracted from the second augmented image I₂ 210 using encoder network ƒ_θ212. While encoder network ƒ_θ212 is separately shown in FIG. 2 to separately illustrate the processing of first augmented image I₁ 206 and second augmented image I₂ 210, it should be understood that the same encoder network ƒ_θ212 is used to extract first representation vector I₁ 214 and second representation vector I₂ 216. This can be formally expressed as Z₁=ƒ_θ(I₁) and Z₂=ƒ_θ(I₂).

The encoder network may be any suitable machine learning-based network for respectively encoding the first and second augmented images to first and second representation vectors. In one example, the encoder network is a neural network, such as, a ResNet (Residual Network). The first and second representation vectors respectively comprise latent features of the first and second augmented images.

At step 112 of FIG. 1, the first representation vector is mapped to first cluster codes and, at step 114 of FIG. 1, the second representation vector is mapped to second cluster codes. In one example, as shown in FIG. 2, first representation vector Z₁ 214 is mapped to first cluster codes Q₁ 220 from a set of cluster prototypes C 218 and second representation vector Z₂ 216 is mapped to second cluster codes Q₂ 222 from the set of cluster prototypes C 218. This can be formally expressed as Q₁=g(Z₁) and Q₂=g(Z₂), where g(·) is the projection network. The first cluster codes Q₁ 220 and second cluster codes Q₂ 222 are features of the first representation vector Z₁ 214 and the second representation vector Z₂ 216 respectively in a feature space where a contrastive clustering loss function is applied (at step 118).

The projection network may be any suitable machine learning-based network for respectively mapping the first and second representation vectors to first and second cluster codes. In one example, the projection network is a neural network, such as, e.g., an MLP (multilayer perceptron).

Method 100 of FIG. 1 then proceeds to decision block 116, where it is determined whether there are any remaining training medical images in the set of training medical images. If there are remaining training medical images in the set of training medical images, then method 100 returns to step 104 and steps 104-114 are performed for a next respective training medical image. Accordingly, steps 104-114 are performed for each respective training medical images in the set of training medical images. If there are no remaining training medical images in the set of training medical images, method 100 proceeds to step 118. The set of training medical images may represent a batch of training medical images for training the encoder network.

At step 118 of FIG. 1, the encoder network is optimized using the first and second representation vectors and the first and second cluster codes. The encoder network is optimized according to a contrastive loss function based on swapped predictions, such that the loss is calculated based on a first loss calculated based on the first representation vector and the second cluster codes, a second loss calculated based on the second representation vector and the first cluster codes, and the overall loss is calculated by combining (e.g., adding) the first loss and the second loss. Formally, this is expressed as L(Z₁,Z₂)=l(Z₁, Q₂)+l(Z₂, Q₁), where Z represents a representation vector and Q represents cluster codes. The encoder network is optimized to find cluster codes Q that maximize the similarity between the representation vectors Z and cluster prototypes C according to the following loss function:

$\begin{array}{cc}\underset{Q\in Q}{\max }\mathrm{Tr}\left(Q^T{C}^{T}Z\right)+ϵH\left(Q\right)& \left(\mathrm{Equation}2\right)\end{array}$

where H is entropy and E is a regularization weighing factor. The solution Q* to Equation 2 can be determined as a normalized exponential matrix using the Sinkhorn-Knopp algorithm. In one example, as shown in FIG. 2, encoder network ƒ_θ212 is optimized based on swapped prediction optimization 224 by optimizing cluster codes Q₁ 220 and Q₂ 222 to maximize the similarity between representation vectors Z₁ 214 and Z₂ 216 and cluster prototypes C 218. By optimizing the encoder network based on swapped predictions, features are obtained that are robust to the first and second augmentation operations.

In local experiments, pretraining on 1.8 million x-ray images or pretraining on 105 million CT/MR/ultrasound/x-ray images yielded a considerable increase in performance in assessing chest x-ray abnormalities based on a downstream task on 12 thousand images (1-3% AUC (area under the curve) increase compared to transfer learning and up to 12% increase in AUC compared to no pretraining), significantly increased training speed, and provided a consistent increase in robustness to various input augmentations (e.g., rotation, scaling, intensity variations, etc.).

Equation 2 may be used to learn from a multi-modality dataset of training medical images using random sampling. However, in one embodiment, the loss function of Equation 2 is modified to a contrastive clustering loss function that conditions clustering based on a modality of the training medical images. Specifically, cluster prototypes C_m are clustered according to different modalities m according to the following contrastive clustering loss function:

$\begin{array}{cc}{\sum }_{m=1}^M\underset{Q_m\in Q}{\max }Tr\left(Q_m^T{C}_m^T{Z}^{\left(m\right)}\right)+ϵH\left(Q_m\right)& \left(\mathrm{Equation}3\right)\end{array}$

where Q_m and C_m are conditioned on modality m and Z^(m) denotes the aggregate of

$\frac{B}{M}$

vectors z associated with modality m.

At step 120 of FIG. 1, the optimized encoder network is output. For example, the optimized encoder network can be output by storing the optimized encoder network on a memory or storage of a computer system, or by transmitting the optimized encoder network to a remote computer system. The optimized encoder network may be applied during an online or inference stage for performing a medical imaging analysis task on an input medical image. In one embodiment, method 100 may be repeated for addition sets of training medical images representing additional batches.

FIG. 3 shows a method 300 for performing a medical imaging analysis task using a trained encoder network, in accordance with one or more embodiments. The steps of method 300 may be performed by one or more suitable computing devices, such as, e.g., computer 1002 of FIG. 10. Method 300 is performed during an online or inference stage.

At step 302, an input medical image is received. The input medical image may depict, for example, anatomies and/or abnormalities of a patient. The input medical image may be of any suitable modality, such as, e.g., MRI, CT, ultrasound, x-ray, and/or any other medical imaging modality or combinations of medical imaging modalities. The input medical image may comprise 2D (two dimensional) images and/or 3D (three dimensional) volumes, and may comprise a single input medical image or a plurality of input medical images. The input medical image may be received directly from an image acquisition device, such as, e.g., a CT scanner, as the input medical image is acquired, or can be received by loading a previously acquired input medical image from a storage or memory of a computer system (e.g., a PACS (picture archiving and communication system)) or by receiving an input medical image that have been transmitted from a remote computer system.

At step 304, a representation vector is extracted from the input medical image using a trained encoder network. In one embodiment, the trained encoder network is the encoder network optimized according to method 100 of FIG. 1. The representation vector comprises latent features of the input medical image. The encoder network may be any suitable machine learning-based network for encoding the input medical image to the representation vector. In one example, the encoder network is a neural network, such as, e.g., a ResNet.

At step 306, a medical imaging analysis task is performed for the input medical image based on the representation vector. The medical imaging analysis task may be any task for analyzing the input medical image, such as, e.g., detection, segmentation, or classification of anatomies and/or abnormalities in the input medical image. In one embodiment, a decoder network is applied to decode the representation vector to perform the medical imaging analysis task. The encoder network may be reused for other medical imaging analysis tasks, for example, using transfer learning (i.e., pretraining) where training is continuously performed in a supervised way on a specific downstream task. For downstream tasks, more complex models may also be utilized and embodiments described herein are not limited to an encoder network. For example, self-supervised learning may be used to pretrain a backbone network (residual network based).

At step 308, results of the medical imaging analysis task are output. The results of the medical imaging analysis task may comprise, for example, a probability map, a segmentation map, a classification, etc. The results of the medical imaging analysis task can be output by displaying the results of the medical imaging analysis task on a display device of a computer system, storing the results of the medical imaging analysis task on a memory or storage of a computer system, or by transmitting the results of the medical imaging analysis task to a remote computer system.

FIG. 4 shows graphs 400 comparing performance for lesion and pneumothorax detection in chest radiographs of conventional artificial intelligence-based systems with an artificial intelligence-based system pretrained according to the self-supervised learning in accordance with embodiments described herein. Graphs 400 compare AUC over time (in Epochs) and average instance-level sensitivity over time. Lines 402 and 404 respectively represent testing and validation performance of conventional artificial intelligence-based systems that are not pretrained. Lines 406 and 408 respectively represent testing and validation performance of conventional pretrained ImageNet. Lines 410 and 412 respectively represent testing and validation performance of an artificial intelligence-based system pretrained according to the self-supervised learning in accordance with embodiments described herein. The artificial intelligence-based system pretrained according to the self-supervised learning in accordance with embodiments described herein outperform both conventional approaches in terms of AUC and average instance detection sensitivity. The difference is significant, ranging between 3-5%. The difference is much larger when compared to using no pretraining, ranging between 20-25% on the lesion test and reaching almost 30% on the very challenging pneumothorax test data.

FIG. 5 shows a graph 500 illustrating the robustness of an artificial intelligence-based system pretrained according to the self-supervised learning in accordance with embodiments described herein. Box plot 502 represents probability variation of a conventional pretrained ImageNet. Box plot 504 represents probability variation of artificial intelligence-based system trained according to the self-supervised learning in accordance. As shown in graph 500, using the self-supervised learning (box plot 504) yields a higher level of robustness relative to standard augmentations.

In one embodiment, where at least some of the training medical images of the set of training medical images are annotated (e.g., with image level labels, annotation masks, bounding boxes, associated text descriptions), the annotations may be exploited and incorporated into the self-supervised learning framework. In one embodiment, annotated training images may be incorporated into the self-supervised learning framework by, for example, adding a new auxiliary loss to the optimization of the encoder network (at step 118 of FIG. 1), by a loss-conditional training, or by using an alternating learning strategy (i.e., alternating between self-supervised learning and supervised learning based on the annotations). In one embodiment, a hybrid self-supervised—supervised learning approach provided for dynamically incorporating annotations in the self-supervised learning framework. Given dataset =[x₁, x₂, . . . , x_N′, (x_N′+1,y_N′+1), . . . , (x_N, y_N)] of N training samples, N-N′ samples are paired with annotations or labels y_k (where k>N′). Let g_ω, be a deep neural network projector parameterized by ω. Annotated samples x_k are mapped to predicted annotations ŷ_k from features of encoder network ƒ_θ (output features and/or intermediate layer features) is as follows:

ŷ _k =g _ω(x_k|ƒ_θ),∀_k>N′

In this case, to optimize the encoder network ƒ_θ, the goal is similar to any supervised learning problem, i.e., minimize the distance of predicted label ŷ_k to label y_k according to a loss function _sup (sup=supervised):

$\left[{\theta }^{*},{\omega }^{*}\right]=\begin{array}{c}\mathrm{argmin}\\ \theta ,\omega \end{array}\frac{1}{N-N^{\prime }}{\sum }_{k=N^{\prime }+1}^N{ℒ}_{\sup }\left[g_{\omega }\left(x_k|f_{\theta }\right),y_k\right].$

Combining equations to a single global optimization criterion and re-balancing the contribution of each using factors α, β∈ gives the following objective for optimizing the encoder network:

$\left[{\theta }^{*},{\omega }^{*},C^{*}\right]=\begin{array}{c}\mathrm{argmin}\\ \theta ,\omega ,C\end{array}\frac{1}{N}{\sum }_{k=1}^N{\mathrm{\alpha ℒ}}_{sss}\left(x_k\right)+1_{k>n^{\prime }}{\mathrm{\beta ℒ}}_{\sup }\left[g_{\omega }\left(x_k|f_{\theta }\right),y_k\right].$

Experiments incorporating annotated training medical images in self-supervised learning have shown improved performance in the downstream medical imaging analysis task. FIG. 6 shows graphs comparing the performance of artificial intelligence-based systems trained using only annotations and the performance of artificial intelligence-based systems trained using annotations with self-supervised learning in accordance with embodiments described herein. Graph 600 shows AUC and ACC (accuracy) for validating and testing of artificial intelligence-based system trained only with training medical images annotated with hemorrhage labels. The AUC for validating was found to be 0.88 and the AUC for testing was found to be 0.87. Graph 602 shows AUC and ACC for validating and testing of artificial intelligence-based system trained with training medical images annotated with hemorrhage labels, types, and slice labels with self-supervised learning. The AUC for validating was found to be 0.95 and the AUC for testing was found to be 0.94. Accordingly, performance of artificial intelligence-based systems increased from 0.87 to 0.94 AUC when trained with annotated training medical images with self-supervised learning.

FIG. 7 shows a framework 700 for self-supervised learning performed with federated learning, in accordance with one or more embodiments. Federated learning is a machine learning technique that trains a machine learning algorithm across multiple decentralized devices holding local training data, without exchanging the local training data. In the context of medical imaging, coupling self-supervised learning with federated learning brings a number of advantages. Federated self-supervised learning naturally scales to large pools of data (ranging up to billions of images), while avoiding stringent data privacy regulations by accessing the data on site and training the machine learning algorithm on site. Federated self-supervised learning may provide a more effective strategy to control the learning strategy based on the quality of local annotations. For example, if a certain site provides low quality annotations, the system learns from the local training data of that site in a purely self-supervised way (i.e., without annotations). Further, in federated self-supervised learning, training from different sites may be weighted based on the downstream medical imaging analysis task.

FIG. 7 shows a plurality of sites comprising Site₁ 702, Site₂ 704, Site₃ 706, and Site_N 708, each offering subsets of training medical images in different modalities, with or without annotations, and with or without non-imaging data (e.g., clinical reports, laboratory data, etc.). In practice, access to each site, and their associated training medical data, is not perpetual. If it would be, training central machine learning model 710 would be possible using a number of different approaches for federated learning. However, access to certain sites can be lost or gained. In accordance with one embodiment, central model 710 is trained with self-supervised learning combined with federated learning and continual learning. Federated learning allows for centralized training of central model 710 from training data from sites 702-708. Continual learning allows for continual learning of central model 710 without forgetting what the model has already learned, even when access to one or more using sites 702-708 is lost. Accordingly, central model 710 may be trained using self-supervised learning with federated learning and continual learning using training data of all sites 702-208 over time, while not losing information learned from sites once access to those sites is lost, and while learning from training medical images from new sites. Since the central model 710 is continually updated by different sites, gaining or losing access to a certain site does not impact the knowledge already gained by the system. Furthermore, the amount of knowledge gained by central model 710 from a specific site can be modulated based on the task for central model (e.g., if central model 710 is tasked to label regions in XR images, Site₁ 702 will be weighed less than the other sites). Continual learning may be implemented via Bayesian inference to address the diversity of data from different datasets. In addition, the so called coreset may be modelled using generative replay to achieve stability in the modeling of the posterior after observing T-1 datasets, while properly capturing the information encoded in dataset T as a likelihood p(S_T|θ), where θ represents the model parameters p(θ|S_{1 . . . T})∝p(θ|S_{1 . . . T-1})p(S_T|θ).

Embodiments described herein are described with respect to the claimed systems as well as with respect to the claimed methods. Features, advantages or alternative embodiments herein can be assigned to the other claimed objects and vice versa. In other words, claims for the systems can be improved with features described or claimed in the context of the methods. In this case, the functional features of the method are embodied by objective units of the providing system.

Furthermore, certain embodiments described herein are described with respect to methods and systems utilizing trained machine learning based networks (or models), as well as with respect to methods and systems for training machine learning based networks. Features, advantages or alternative embodiments herein can be assigned to the other claimed objects and vice versa. In other words, claims for methods and systems for training a machine learning based network can be improved with features described or claimed in context of the methods and systems for utilizing a trained machine learning based network, and vice versa.

In particular, the trained machine learning based networks applied in embodiments described herein can be adapted by the methods and systems for training the machine learning based networks. Furthermore, the input data of the trained machine learning based network can comprise advantageous features and embodiments of the training input data, and vice versa. Furthermore, the output data of the trained machine learning based network can comprise advantageous features and embodiments of the output training data, and vice versa.

In general, a trained machine learning based network mimics cognitive functions that humans associate with other human minds. In particular, by training based on training data, the trained machine learning based network is able to adapt to new circumstances and to detect and extrapolate patterns.

In general, parameters of a machine learning based network can be adapted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and/or active learning can be used. Furthermore, representation learning (an alternative term is “feature learning”) can be used. In particular, the parameters of the trained machine learning based network can be adapted iteratively by several steps of training.

In particular, a trained machine learning based network can comprise a neural network, a support vector machine, a decision tree, and/or a Bayesian network, and/or the trained machine learning based network can be based on k-means clustering, Q-learning, genetic algorithms, and/or association rules. In particular, a neural network can be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, a neural network can be an adversarial network, a deep adversarial network and/or a generative adversarial network.

FIG. 8 shows an embodiment of an artificial neural network 800, in accordance with one or more embodiments. Alternative terms for “artificial neural network” are “neural network”, “artificial neural net” or “neural net”. Machine learning networks described herein, such as, e.g., the encoder network described with respect to method 100 of FIG. 1, the encoder network ƒ_θ212 of framework 200 of FIG. 2, the encoder network described with respect to method 300 of FIG. 3, and central model 710 of framework 700 of FIG. 7, may be implemented using artificial neural network 800.

The artificial neural network 800 comprises nodes 802-822 and edges 832, 834, . . . , 836, wherein each edge 832, 834, . . . , 836 is a directed connection from a first node 802-822 to a second node 802-822. In general, the first node 802-822 and the second node 802-822 are different nodes 802-822, it is also possible that the first node 802-822 and the second node 802-822 are identical. For example, in FIG. 8, the edge 832 is a directed connection from the node 802 to the node 806, and the edge 834 is a directed connection from the node 804 to the node 806. An edge 832, 834, . . . , 836 from a first node 802-822 to a second node 802-822 is also denoted as “ingoing edge” for the second node 802-822 and as “outgoing edge” for the first node 802-822.

In this embodiment, the nodes 802-822 of the artificial neural network 800 can be arranged in layers 824-830, wherein the layers can comprise an intrinsic order introduced by the edges 832, 834, . . . , 836 between the nodes 802-822. In particular, edges 832, 834, . . . , 836 can exist only between neighboring layers of nodes. In the embodiment shown in FIG. 8, there is an input layer 824 comprising only nodes 802 and 804 without an incoming edge, an output layer 830 comprising only node 822 without outgoing edges, and hidden layers 826, 828 in-between the input layer 824 and the output layer 830. In general, the number of hidden layers 826, 828 can be chosen arbitrarily. The number of nodes 802 and 804 within the input layer 824 usually relates to the number of input values of the neural network 800, and the number of nodes 822 within the output layer 830 usually relates to the number of output values of the neural network 800.

In particular, a (real) number can be assigned as a value to every node 802-822 of the neural network 800. Here, x⁽ⁿ⁾_i denotes the value of the i-th node 802-822 of the n-th layer 824-830. The values of the nodes 802-822 of the input layer 824 are equivalent to the input values of the neural network 800, the value of the node 822 of the output layer 830 is equivalent to the output value of the neural network 800. Furthermore, each edge 832, 834, . . . , 836 can comprise a weight being a real number, in particular, the weight is a real number within the interval [−1, 1] or within the interval [0, 1]. Here, w^(m,n)_i,j denotes the weight of the edge between the i-th node 802-822 of the m-th layer 824-830 and the j-th node 802-822 of the n-th layer 824-830. Furthermore, the abbreviation w(n)_i,j is defined for the weight w^(n,n+1)_i,j.

In particular, to calculate the output values of the neural network 800, the input values are propagated through the neural network. In particular, the values of the nodes 802-822 of the (n+1)-th layer 824-830 can be calculated based on the values of the nodes 802-822 of the n-th layer 824-830 by

$x_j^{\left(n+1\right)}=f\left({\sum }_i{x}_i^{\left(n\right)}·w_{i,j}^{\left(n\right)}\right).$

Herein, the function f is a transfer function (another term is “activation function”). Known transfer functions are step functions, sigmoid function (e.g. the logistic function, the generalized logistic function, the hyperbolic tangent, the Arctangent function, the error function, the smoothstep function) or rectifier functions. The transfer function is mainly used for normalization purposes.

In particular, the values are propagated layer-wise through the neural network, wherein values of the input layer 824 are given by the input of the neural network 800, wherein values of the first hidden layer 826 can be calculated based on the values of the input layer 824 of the neural network, wherein values of the second hidden layer 828 can be calculated based in the values of the first hidden layer 826, etc.

In order to set the values w^(m,n)_i,j for the edges, the neural network 800 has to be trained using training data. In particular, training data comprises training input data and training output data (denoted as t_i). For a training step, the neural network 800 is applied to the training input data to generate calculated output data. In particular, the training data and the calculated output data comprise a number of values, said number being equal with the number of nodes of the output layer.

In particular, a comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network 800 (backpropagation algorithm). In particular, the weights are changed according to

w _i,j ^{T (n)} =w _i,j ⁽ⁿ⁾−γ·δ_j⁽ⁿ⁾·x_i⁽ⁿ⁾

wherein γ is a learning rate, and the numbers δ⁽ⁿ⁾_j can be recursively calculated as

δ_j⁽ⁿ⁾=(Σ_kδ_k⁽ⁿ⁺¹⁾·w_j,k⁽ⁿ⁺¹⁾)·f^T(Σ_ix_i⁽ⁿ⁾·w_i,j⁽ⁿ⁾)

based on δ⁽ⁿ⁺¹⁾_j, if the (n+1)-th layer is not the output layer, and

δ_j⁽ⁿ⁾=(x_k⁽ⁿ⁺¹⁾−t_j⁽ⁿ⁺¹⁾)·f^T(Σ_ix_i⁽ⁿ⁾·w_i,j⁽ⁿ⁾)

if the (n+1)-th layer is the output layer 830, wherein f′ is the first derivative of the activation function, and y⁽ⁿ⁺¹⁾_j is the comparison training value for the j-th node of the output layer 830.

FIG. 9 shows a convolutional neural network 900, in accordance with one or more embodiments. Machine learning networks described herein, such as, e.g., the encoder network described with respect to method 100 of FIG. 1, the encoder network ƒ_θ212 of framework 200 of FIG. 2, the encoder network described with respect to method 300 of FIG. 3, and central model 710 of framework 700 of FIG. 7, may be implemented using convolutional neural network 900.

In the embodiment shown in FIG. 9, the convolutional neural network comprises 900 an input layer 902, a convolutional layer 904, a pooling layer 906, a fully connected layer 908, and an output layer 910. Alternatively, the convolutional neural network 900 can comprise several convolutional layers 904, several pooling layers 906, and several fully connected layers 908, as well as other types of layers. The order of the layers can be chosen arbitrarily, usually fully connected layers 908 are used as the last layers before the output layer 910.

In particular, within a convolutional neural network 900, the nodes 912-920 of one layer 902-910 can be considered to be arranged as a d-dimensional matrix or as a d-dimensional image. In particular, in the two-dimensional case the value of the node 912-920 indexed with i and j in the n-th layer 902-910 can be denoted as x⁽ⁿ⁾_[i,j]. However, the arrangement of the nodes 912-920 of one layer 902-910 does not have an effect on the calculations executed within the convolutional neural network 900 as such, since these are given solely by the structure and the weights of the edges.

In particular, a convolutional layer 904 is characterized by the structure and the weights of the incoming edges forming a convolution operation based on a certain number of kernels. In particular, the structure and the weights of the incoming edges are chosen such that the values x⁽ⁿ⁾_k of the nodes 914 of the convolutional layer 904 are calculated as a convolution x⁽ⁿ⁾_k=K_k*X^(n-1) based on the values x⁽ⁿ⁻¹⁾ of the nodes 912 of the preceding layer 902, where the convolution * is defined in the two-dimensional case as

x _k ⁽ⁿ⁾[i,j]=(K_k*x^(n-1))[i,j]=Σ_i,Σ_j,K_k[i′,j′]·x^(n-1)[i-i′,j-j′].

Here the k-th kernel K_k is a d-dimensional matrix (in this embodiment a two-dimensional matrix), which is usually small compared to the number of nodes 912-918 (e.g. a 3×3 matrix, or a 5×5 matrix). In particular, this implies that the weights of the incoming edges are not independent, but chosen such that they produce said convolution equation. In particular, for a kernel being a 3×3 matrix, there are only 9 independent weights (each entry of the kernel matrix corresponding to one independent weight), irrespectively of the number of nodes 912-920 in the respective layer 902-910. In particular, for a convolutional layer 904, the number of nodes 914 in the convolutional layer is equivalent to the number of nodes 912 in the preceding layer 902 multiplied with the number of kernels.

If the nodes 912 of the preceding layer 902 are arranged as a d-dimensional matrix, using a plurality of kernels can be interpreted as adding a further dimension (denoted as “depth” dimension), so that the nodes 914 of the convolutional layer 904 are arranged as a (d+1)-dimensional matrix. If the nodes 912 of the preceding layer 902 are already arranged as a (d+1)-dimensional matrix comprising a depth dimension, using a plurality of kernels can be interpreted as expanding along the depth dimension, so that the nodes 914 of the convolutional layer 904 are arranged also as a (d+1)-dimensional matrix, wherein the size of the (d+1)-dimensional matrix with respect to the depth dimension is by a factor of the number of kernels larger than in the preceding layer 902.

The advantage of using convolutional layers 904 is that spatially local correlation of the input data can exploited by enforcing a local connectivity pattern between nodes of adjacent layers, in particular by each node being connected to only a small region of the nodes of the preceding layer.

In embodiment shown in FIG. 9, the input layer 902 comprises 36 nodes 912, arranged as a two-dimensional 6×6 matrix. The convolutional layer 904 comprises 72 nodes 914, arranged as two two-dimensional 6×6 matrices, each of the two matrices being the result of a convolution of the values of the input layer with a kernel. Equivalently, the nodes 914 of the convolutional layer 904 can be interpreted as arranges as a three-dimensional 6×6×2 matrix, wherein the last dimension is the depth dimension.

A pooling layer 906 can be characterized by the structure and the weights of the incoming edges and the activation function of its nodes 916 forming a pooling operation based on a non-linear pooling function f. For example, in the two dimensional case the values x⁽ⁿ⁾ of the nodes 916 of the pooling layer 906 can be calculated based on the values x^(n-1) of the nodes 914 of the preceding layer 904 as

x ⁽ⁿ⁾[i,j]=f(x^(n-1)[id₁,jd₂], . . . , x^(n-1)[id₁+d₁−1,jd₂+d₂−1])

In other words, by using a pooling layer 906, the number of nodes 914, 916 can be reduced, by replacing a number d1·d2 of neighboring nodes 914 in the preceding layer 904 with a single node 916 being calculated as a function of the values of said number of neighboring nodes in the pooling layer. In particular, the pooling function f can be the max-function, the average or the L2-Norm. In particular, for a pooling layer 906 the weights of the incoming edges are fixed and are not modified by training.

The advantage of using a pooling layer 906 is that the number of nodes 914, 916 and the number of parameters is reduced. This leads to the amount of computation in the network being reduced and to a control of overfitting.

In the embodiment shown in FIG. 9, the pooling layer 906 is a max-pooling, replacing four neighboring nodes with only one node, the value being the maximum of the values of the four neighboring nodes. The max-pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, the max-pooling is applied to each of the two two-dimensional matrices, reducing the number of nodes from 72 to 18.

A fully-connected layer 908 can be characterized by the fact that a majority, in particular, all edges between nodes 916 of the previous layer 906 and the nodes 918 of the fully-connected layer 908 are present, and wherein the weight of each of the edges can be adjusted individually.

In this embodiment, the nodes 916 of the preceding layer 906 of the fully-connected layer 908 are displayed both as two-dimensional matrices, and additionally as non-related nodes (indicated as a line of nodes, wherein the number of nodes was reduced for a better presentability). In this embodiment, the number of nodes 918 in the fully connected layer 908 is equal to the number of nodes 916 in the preceding layer 906. Alternatively, the number of nodes 916, 918 can differ.

Furthermore, in this embodiment, the values of the nodes 920 of the output layer 910 are determined by applying the Softmax function onto the values of the nodes 918 of the preceding layer 908. By applying the Softmax function, the sum the values of all nodes 920 of the output layer 910 is 1, and all values of all nodes 920 of the output layer are real numbers between 0 and 1.

A convolutional neural network 900 can also comprise a ReLU (rectified linear units) layer or activation layers with non-linear transfer functions. In particular, the number of nodes and the structure of the nodes contained in a ReLU layer is equivalent to the number of nodes and the structure of the nodes contained in the preceding layer. In particular, the value of each node in the ReLU layer is calculated by applying a rectifying function to the value of the corresponding node of the preceding layer.

The input and output of different convolutional neural network blocks can be wired using summation (residual/dense neural networks), element-wise multiplication (attention) or other differentiable operators. Therefore, the convolutional neural network architecture can be nested rather than being sequential if the whole pipeline is differentiable.

In particular, convolutional neural networks 900 can be trained based on the backpropagation algorithm. For preventing overfitting, methods of regularization can be used, e.g. dropout of nodes 912-920, stochastic pooling, use of artificial data, weight decay based on the L1 or the L2 norm, or max norm constraints. Different loss functions can be combined for training the same neural network to reflect the joint training objectives. A subset of the neural network parameters can be excluded from optimization to retain the weights pretrained on another datasets.

Systems, apparatuses, and methods described herein may be implemented using digital circuitry, or using one or more computers using well-known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include, or be coupled to, one or more mass storage devices, such as one or more magnetic disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.

Systems, apparatus, and methods described herein may be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computers are located remotely from the server computer and interact via a network. The client-server relationship may be defined and controlled by computer programs running on the respective client and server computers.

Systems, apparatus, and methods described herein may be implemented within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor that is connected to a network communicates with one or more client computers via a network. A client computer may communicate with the server via a network browser application residing and operating on the client computer, for example. A client computer may store data on the server and access the data via the network. A client computer may transmit requests for data, or requests for online services, to the server via the network. The server may perform requested services and provide data to the client computer(s). The server may also transmit data adapted to cause a client computer to perform a specified function, e.g., to perform a calculation, to display specified data on a screen, etc. For example, the server may transmit a request adapted to cause a client computer to perform one or more of the steps or functions of the methods and workflows described herein, including one or more of the steps or functions of FIGS. 1-3. Certain steps or functions of the methods and workflows described herein, including one or more of the steps or functions of FIGS. 1-3, may be performed by a server or by another processor in a network-based cloud-computing system. Certain steps or functions of the methods and workflows described herein, including one or more of the steps of FIGS. 1-3, may be performed by a client computer in a network-based cloud computing system. The steps or functions of the methods and workflows described herein, including one or more of the steps of FIGS. 1-3, may be performed by a server and/or by a client computer in a network-based cloud computing system, in any combination.

Systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier, e.g., in a non-transitory machine-readable storage device, for execution by a programmable processor; and the method and workflow steps described herein, including one or more of the steps or functions of FIGS. 1-3, may be implemented using one or more computer programs that are executable by such a processor. A computer program is a set of computer program instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

A high-level block diagram of an example computer 1002 that may be used to implement systems, apparatus, and methods described herein is depicted in FIG. 10. Computer 1002 includes a processor 1004 operatively coupled to a data storage device 1012 and a memory 1010. Processor 1004 controls the overall operation of computer 1002 by executing computer program instructions that define such operations. The computer program instructions may be stored in data storage device 1012, or other computer readable medium, and loaded into memory 1010 when execution of the computer program instructions is desired. Thus, the method and workflow steps or functions of FIGS. 1-3 can be defined by the computer program instructions stored in memory 1010 and/or data storage device 1012 and controlled by processor 1004 executing the computer program instructions. For example, the computer program instructions can be implemented as computer executable code programmed by one skilled in the art to perform the method and workflow steps or functions of FIGS. 1-3. Accordingly, by executing the computer program instructions, the processor 1004 executes the method and workflow steps or functions of FIGS. 1-3. Computer 1002 may also include one or more network interfaces 1006 for communicating with other devices via a network. Computer 1002 may also include one or more input/output devices 1008 that enable user interaction with computer 1002 (e.g., display, keyboard, mouse, speakers, buttons, etc.).

Processor 1004 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of computer 1002. Processor 1004 may include one or more central processing units (CPUs), for example. Processor 1004, data storage device 1012, and/or memory 1010 may include, be supplemented by, or incorporated in, one or more application-specific integrated circuits (ASICs) and/or one or more field programmable gate arrays (FPGAs).

Data storage device 1012 and memory 1010 each include a tangible non-transitory computer readable storage medium. Data storage device 1012, and memory 1010, may each include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM) disks, or other non-volatile solid state storage devices.

Input/output devices 1008 may include peripherals, such as a printer, scanner, display screen, etc. For example, input/output devices 1008 may include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to the user, a keyboard, and a pointing device such as a mouse or a trackball by which the user can provide input to computer 1002.

An image acquisition device 1014 can be connected to the computer 1002 to input image data (e.g., medical images) to the computer 1002. It is possible to implement the image acquisition device 1014 and the computer 1002 as one device. It is also possible that the image acquisition device 1014 and the computer 1002 communicate wirelessly through a network. In a possible embodiment, the computer 1002 can be located remotely with respect to the image acquisition device 1014.

Any or all of the systems and apparatus discussed herein may be implemented using one or more computers such as computer 1002.

One skilled in the art will recognize that an implementation of an actual computer or computer system may have other structures and may contain other components as well, and that FIG. 10 is a high level representation of some of the components of such a computer for illustrative purposes.

The foregoing Detailed Description is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the principles of the present invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention.

Claims

1. A computer-implemented method comprising: for each respective training medical image of a set of unannotated training medical images: generating a first augmented image by applying a first augmentation operation to the respective training medical image,generating a second augmented image by applying a second augmentation operation to the respective training medical image,creating a first representation vector from the first augmented image using an encoder network,creating a second representation vector from the second augmented image using the encoder network,mapping the first representation vector to first cluster codes, andmapping the second representation vector to second cluster codes; andoptimizing the encoder network using the first and second representation vectors and the first and second cluster codes.

2. The computer-implemented method of claim 1, wherein optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: optimizing the encoder network to find cluster codes that maximize a similarity between representation vectors and cluster prototypes.

3. The computer-implemented method of claim 1, wherein optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: calculating a first loss based on the first representation vectors and the second cluster codes;calculating a second loss based on the second representation vectors and the first cluster codes; andcombining the first loss and the second loss.

4. The computer-implemented method of claim 1, wherein optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: optimizing a loss function by clustering based on a modality of each of the training medical images.

5. The computer-implemented method of claim 1, wherein one of the first augmentation operation or the second augmentation operation comprises an energy-based augmentation performed by: decomposing an image into a plurality of energy bands;computing an energy value for each of the plurality of energy bands; andtransforming the image based on the energy values.

6. The computer-implemented method of claim 1, wherein the set of unannotated training medical images comprises a plurality of subsets of training medical images each associated with different sites, and wherein optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: continuously optimizing the encoder network with the plurality of subsets of training medical images using federated learning and continual learning.

7. The computer-implemented method of claim 1, further comprising receiving annotated training data comprising annotated training medical images and corresponding annotations; andmapping the annotated training medical images to predicted annotations using a machine learning based network based on features of the encoder network,wherein optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises optimizing the encoder network to minimize a distance of the predicted annotations to the corresponding annotations.

8. The computer-implemented method of claim 1, further comprising: receiving an input medical image;creating a representation vector from the input medical image using the optimized encoder network;performing a medical imaging analysis task for the input medical image based on the representation vector; andoutputting results of the medical imaging analysis task.

9. The computer-implemented method of claim 8, wherein performing a medical imaging analysis task for the input medical image based on the representation vector comprises: decoding the representation vector using a decoder network to perform the medical imaging analysis task.

10. An apparatus comprising: for each respective training medical image of a set of unannotated training medical images: means for generating a first augmented image by applying a first augmentation operation to the respective training medical image,means for generating a second augmented image by applying a second augmentation operation to the respective training medical image,means for creating a first representation vector from the first augmented image using an encoder network,means for creating a second representation vector from the second augmented image using the encoder network,means for mapping the first representation vector to first cluster codes, andmeans for mapping the second representation vector to second cluster codes; andmeans for optimizing the encoder network using the first and second representation vectors and the first and second cluster codes.

11. The apparatus of claim 10, wherein the means for optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: means for optimizing the encoder network to find cluster codes that maximize a similarity between representation vectors and cluster prototypes.

12. The apparatus of claim 10, wherein the means for optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: means for calculating a first loss based on the first representation vectors and the second cluster codes;means for calculating a second loss based on the second representation vectors and the first cluster codes; andmeans for combining the first loss and the second loss.

13. The apparatus of claim 10, wherein the means for optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: means for optimizing a loss function by clustering based on a modality of each of the training medical images.

14. The apparatus of claim 10, wherein one of the first augmentation operation or the second augmentation operation comprises an energy-based augmentation performed by: means for decomposing an image into a plurality of energy bands;means for computing an energy value for each of the plurality of energy bands; andmeans for transforming the image based on the energy values.

15. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising: for each respective training medical image of a set of unannotated training medical images: generating a first augmented image by applying a first augmentation operation to the respective training medical image,generating a second augmented image by applying a second augmentation operation to the respective training medical image,creating a first representation vector from the first augmented image using an encoder network,creating a second representation vector from the second augmented image using the encoder network,mapping the first representation vector to first cluster codes, and mapping the second representation vector to second cluster codes; andoptimizing the encoder network using the first and second representation vectors and the first and second cluster codes.

16. The non-transitory computer readable medium of claim 15, wherein optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: optimizing the encoder network to find cluster codes that maximize a similarity between representation vectors and cluster prototypes.

17. The non-transitory computer readable medium of claim 15, wherein the set of unannotated training medical images comprises a plurality of subsets of training medical images each associated with different sites, and wherein optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises: continuously optimizing the encoder network with the plurality of subsets of training medical images using federated learning and continual learning.

18. The non-transitory computer readable medium of claim 15, the operations further comprising receiving annotated training data comprising annotated training medical images and corresponding annotations; andmapping the annotated training medical images to predicted annotations using a machine learning based network based on features of the encoder network,wherein optimizing the encoder network using the first and second representation vectors and the first and second cluster codes comprises optimizing the encoder network to minimize a distance of the predicted annotations to the corresponding annotations.

19. The non-transitory computer readable medium of claim 15, the operations further comprising: receiving an input medical image;creating a representation vector from the input medical image using the optimized encoder network;performing a medical imaging analysis task for the input medical image based on the representation vector; andoutputting results of the medical imaging analysis task.

20. The non-transitory computer readable medium of claim 19, wherein performing a medical imaging analysis task for the input medical image based on the representation vector comprises: decoding the representation vector using a decoder network to perform the medical imaging analysis task.