SYSTEMS AND METHODS FOR CONTRASTIVE LEARNING WITH SELF-LABELING REFINEMENT

Abstract

Embodiments described herein provide a contrastive learning mechanism with self-labeling refinement, which iteratively employs the network and data themselves to generate more accurate and informative soft labels for contrastive learning. Specifically, the contrastive learning framework includes a self-labeling refinery module to explicitly generate accurate labels, and a momentum mix-up module to increase similarity between a query and its positive, which in turn implicitly improves label accuracy.

Description

TECHNICAL FIELD

The present disclosure relates generally to machine learning models and neural networks, and more specifically, to contrastive learning with self-labeling refinement.

BACKGROUND

Supervised learning for neural models usually require a large amount of manually annotated training data, which can be time-consuming and expensive. Self-supervised learning (SSL), or unsupervised visual representation learning, provide a training mechanism for the neural model to learn features without manual annotations. Such SSL methods can often be successful in many downstream tasks, e.g. image classification and object detection. Specifically, SSL constructs a pretext task that can obtain data labels via designing the task itself, and then builds a network to learn from these tasks. For instance, by constructing jigsaw puzzle, spatial arrangement identification, orientation, or chromatic channels as a pretext task, SSL learns high-quality features from the pretext task that can be well transferred to downstream tasks.

Contrastive learning is a recently developed SSL method, which constructs an instance discrimination pretext task to train a network so that the representations of different augmentations or crops of the same instance are pulled close to each other, while representations of different instances are pushed away from each other. Specifically, for an image crop query, contrastive learning randomly augments the same image to obtain a positive instance and view other image crops as negatives. Then it constructs a one-hot label for instance discrimination over the positive and negative instances to pull the positive pair closer while pushing away negative instances in the feature space. The one-hot labels used in contrastive learning, however, can often be inaccurate and uninformative. This is because a query can often be semantically similar or even more similar to its negative instances than the corresponding positive instances.

Therefore, there is a need to improve the accuracy of contrastive learning.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram illustrating a contrastive learning system 100 with self-labeling refinery, according to embodiments described herein.

FIG. 2 is a simplified block diagram illustrating the self-label computation module shown in FIG. 1, according to embodiments described herein.

FIG. 3 is a simplified logic flow diagram illustrating an aspect of contrastive learning with self-labeling refinement as described in FIGS. 1-2, according to some embodiments.

FIG. 4 is a simplified logic flow diagram illustrating an aspect of using momentum mix-up to refine the contrastive learning described in FIGS. 1-3, according to some embodiments.

FIG. 5 is a simplified diagram of a computing device for implementing the proposed contrastive learning with self-labeling refinement, according to some embodiments.

FIGS. 6-10 provide example tables illustrating performance of the contrastive learning system and/r method described in FIGS. 1-5, according to some embodiments.

In the figures and appendix, elements having the same designations have the same or similar functions.

DETAILED DESCRIPTION

Contrastive learning is a self-supervised learning method, which usually learn through an augmented positive instance of a training instance, paired with a negative instance to the training instance that are both input to the neural network. During training, the representations of different augmentations or crops of the same instance are pulled close to each other, while representations of different instances are pushed away from each other. Thus, without any pre-annotated training instances, contrastive learning methods often generate “artificially” labeled data by assuming augmentations of the same instance are positives and augmentations of other instances are negative. Such label assignments can be noisy and impairs the generalization performance, because sometimes negative instances can be semantically similar to the original instance, or even share the same semantic class as the original instance.

For example, a query could be semantically similar or even more similar to some negatives than its positives. Indeed, some negatives may even belong to the same semantic class as the query. This is because, to achieve satisfactory performance, one often uses sufficient negatives that are much more than the semantic class number, which unavoidably leads to the issue on negatives. In addition, even for the same image, especially for images with several different objects which occurs in ImageNet, random augmentations, e.g. crop, can provide crops with (slightly) different semantic information, and thus some of the huge negatives could be (more) similar to query. Hence, the one-hot label does not well reveal the semantically similarity between the query and its positives and “negatives”, and thus cannot guarantee the semantically similar samples to close each other, leading to performance degradation of contrastive learning.

In view of the need to improve the accuracy of contrastive learning, embodiments described herein provide a contrastive learning mechanism with self-labeling refinement, which iteratively employs the network and data themselves to generate more accurate and informative soft labels for contrastive learning. Specifically, the contrastive learning framework includes a self-labeling refinery module to explicitly generate accurate labels, and a momentum mix-up module to increase similarity between a query and its positive, which in turn implicitly improves label accuracy.

For example, given a query, the self-labeling refinery module adopts a positive instance of the query to estimate semantic similarity between the query and its keys (i.e. its positive and negatives) by computing their feature similarity. This is because a query and its positive come from the same image and should have similar semantic similarity on the same keys. Then the self-labeling refinery module linearly combines the estimated similarity of a query with its vanilla one-hot label in contrastive learning to iteratively generate more accurate and informative soft labels. Thus, in this way, at early training stage, one-hot labels have greater combination weights to provide relatively accurate labels. As the training process progresses, the estimated similarity becomes more accurate and informative, and thus the combination weight for the similarity becomes larger. This is because the similarity captures useful underlying semantic information between the query and its keys, which can sometimes be missing from the one-hot labels. This strategy is both empirically and theoretically effective.

In this way, even when the semantic labels in the instance discrimination task for contrastive learning are corrupted, the generated self-labeling may recover the true semantic labels of training data. Thus, networks trained with self-labeling may more accurately predict the true semantic labels of test samples.

In one embodiment, the momentum mix-up module for contrastive learning to further reduces the possible label noise and also increases augmentation diversity. For example, a dataset of queries {x_i}_i=1ⁿ and their corresponding positives {{tilde over (x)}_i}_i=1ⁿ may be randomly combined using a random variable as the combination weight: x_i′=θx_i+(1−θ){tilde over (x)}_k. The estimated label corresponding to an input of x_i′ is y_i′=ƒy_i+(1θ)y_k, where index i and k are randomly selected, y_i is the label of both x_i and {tilde over (x)}_i estimated by the label refinery module, and θ∈(0,1) is a random variable. In this way, the component {tilde over (x)}_k in the virtual query x_i′ directly increases the similarity between the query x_i′ and the positive key {tilde over (x)}_k. Therefore, the label weight (1−θ) of label y_i′ on positive key {tilde over (x)}_i to bring x_i′ and {tilde over (x)}_k together is relatively accurate, since x_i′ contains the sematic information of {tilde over (x)}_k. Meanwhile, the possible noise at the remaining positions of label y_i′ is scaled by θ and becomes smaller. This is precisely why the positive {tilde over (x)}_k is selected to play a contrastive key for instance discrimination instead of the original query x_k for mixup. The momentum mix-up module feeds {tilde over (x)}_k into momentum-updated network to construct a positive key.

As used herein, the term “network” may comprise any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system and/or any training or learning models implemented thereon or therewith.

As used herein, the term “module” may comprise hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.

FIG. 1 is a simplified block diagram illustrating a contrastive learning system 100 with self-labeling refinery, according to embodiments described herein. The contrastive learning system 100 includes an online network f( ) 110 and a target network g( ) 120, each of which includes a feature encoder and a projection head (e.g., a 3-layered multi-layer perceptron).

Given a batch of training images {c_i}_i=1^s at each iteration, each original image sample c_i is randomly augmented into two views (x_i, {tilde over (x)}_i) with x_i being referred to as a query sample and {tilde over (x)}_i being referred to as the positive instance of the query sample. A set B={b_i}_i=1^b denotes the negative keys of current query samples {x_i}_i=1^s. For example, a large dictionary size b is often used to achieve satisfactory performance, e.g. 65,536. In one implementation, B may be updated by the minibatch features {g({tilde over (x)}_i)}_i=1^s in the first-in and first-out order.

In one embodiment, the query sample x_i 102 may be input to the online network f( ) 110, while the set of positive instances {{tilde over (x)}_i}_i=1^s 104 and the set of negative instance B={b_i}_i=1^b 106 may be input to the target network g( ) 120.

The online network f( ) 110 may in turn generate an encoded query representation q=f(x_i) 112, while the target network g( ) 120 may generate a set of encoded key representations {g({tilde over (x)}_i)}_i=1^s∪{g(b_i)}b_i=1^b 122. A similarity metric between the encoded query representation 112 and the set of encoded key representations 122 is then computed at similarity module σ(,) 125, which computes a similarity of two representations in the feature space, e.g.,

$\sigma \left(x,y\right)=\exp \left(-\frac{\langle f\left(x\right),g\left(y\right)\rangle }{\tau {f\left(x\right)}_2·{g\left(y\right)|}_2}\right)$

with a temperature parameter τ.

For example, the similarity module 125 may compute a similarity metric σ(x_i, b_k) where b_k denotes the k-th sample in B={{tilde over (x)}_i}_i=1^s∪{b_i}_i=1^b.

The similarity metrics computed from module 125 may be sent to the contrastive loss module 130. Specifically, the contrastive loss module 130 computes a loss 135 based on the self-labels 129 from a self-label computation 128 and the similarity metrics:

${ℒ}_c\left(w,\left\{\left(x_i,{\stackrel{\_}{y}}_i\right)\right\}\right)=-\frac{1}{s}\sum _{i=1}^s\sum _{k=1}^{s+b}{\stackrel{\_}{y}}_{ik}\log \left(\frac{\sigma \left(x_i,{\overline{b}}_k\right)}{\sum _{l=1}^{s+b}\sigma \left(x_i,{\overline{b}}_l\right)}\right),$

where w denotes the parameters of the online network f( ), b_k denotes the k-th sample in B, and y_i is the self-label of query x_i whose i-th entry y_ii is one. Due to the shared instance set B, all queries have unified label definition and their labels can be linearly combined to form the self-label, as further described in FIG. 2.

The online network f( ) 110 is then updated by the computed loss 135, e.g., via backpropagation, by fixing the parameters of the target network g( ) 120. The target network g ( ) 120 is then updated via exponential moving average (EMA), e.g., ξ=(1+ι)ξ+ιω where ξ denotes the parameters of g( ) and ι∈(0,1) is a constant.

FIG. 2 is a simplified block diagram illustrating the self-label computation module 128 shown in FIG. 1, according to embodiments described herein. Traditionally, one-hot labels of query samples have been used in computing the contrastive loss 135, which does not always well reveal the semantic similarity between x_i and the instance keys in the set B, and thus impairs good representation learning.

In view of this issue, a self-labeling refinery module 128 employs network and data themselves to improve the quality of inaccurate labels during training, which generates more accurate and informative labels, and improves the performance of contrastive learning. Specifically, to refine the one-hot label y_i of query x_i, the positive instance {tilde over (x)}_i is input to the online network f( ) 110 and instance in the set B={{tilde over (x)}_i}_i=1^s∪{b_i}_i=1^b are input to the target network g( ) 120, the outputs of which are used by the similarity module 125 to estimate the possible underlying semantic similarity between x_i and instances in B={{tilde over (x)}_i}_i=1^s∪{b_i}_i=1^b. x_i and {tilde over (x)}_i come from the same image and should have similar semantic similarity with instances in B.

To this end, at the t-th iteration, the instance-class probability p_i^t∈^s+b of x_i on the set B whose k-th entry is defined as

$p_{ik}^t={\sigma }^{1/{\tau }^{\prime }}\left({\tilde{x}}_i,{\overline{b}}_k\right)/\sum _{l=1}^{s+b}{\sigma }^{1/{\tau }^{\prime }}\left({\tilde{x}}_i,{\overline{b}}_l\right),$

where b_k is the k-th sample in B. The constant τ′∈(0,1] sharpens p_i^t, and removes some possible small noise. As p_ik^t measures the semantic similarity between x_i and b_k, it can serve as an instance-class probability to measure whether x_i and b_k share the same semantic class.

On the other hand, as {tilde over (x)}_i is highly similar to itself, p_ii^t could be much larger than others and conceals the similarity of other semantically similar instances in B. To alleviate this artificial effect, {tilde over (x)}_i is then removed from the set B. For example, instance in B\{tilde over (x)}_i are input to the target network g( ) 120, and {tilde over (x)}_i is input to the online network f( ) 110 to estimate the similarity between {tilde over (x)}_i and the remaining instances in B by the similarity metric 125. The estimated similarity metrics are then used to compute the instance class probability:

$q_{ik}^t={\sigma }^{1/{\tau }^{\prime }}\left({\tilde{x}}_i,{\overline{b}}_k\right)/\sum _{l=1,l\ne i}^{s+b}{\sigma }^{1/{\tau }^{\prime }}\left({\tilde{x}}_i,{\overline{b}}_l\right),q_{ii}^t=0$

Then, the combination module 138 combines the one-hot label y_i and two label estimations, i.e. p_i and q_i, to obtain the more accurate, robust and informative label y_i^t of x_i at the t-th iteration:

y _i ^t=(1−α_t−β_t)y_i+α_tp_i^t+β_tq_i^t,

where α_t and β_t are two constants. In one implementation, α_t=μmax_kp_ik^t/z and β_t=μmax_kq_ik^t/z, where z=1+μmax_kp_ik^t+μmax_kq_ik^t, the constants 1, max_kp_ik^t and max_kq_ik^t respectively denote the largest confidences of labels y_i, p_i^t and q_i^t on a certain class. Here hyperparameter μ controls the prior confidence of p^t and q^t. So the self-label refinery only has two parameters τ′ and μ to tune.

FIG. 3 is a simplified logic flow diagram illustrating an aspect of contrastive learning with self-labeling refinement as described in FIGS. 1-2, according to some embodiments. One or more of the processes 302-318 of method 300 may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes 302-318. In some embodiments, method 300 may correspond to the method used by the module 530 described in relation to FIG. 5.

At step 302, a training batch of unlabeled queries is received, e.g., a mini-batch of queries {x_i}_i=1^s.

At step 304, for each unlabeled query, a positive instance {tilde over (x)}_i paired with the query sample is generated.

At step 306, a first instance probability distribution is computed based on a first semantic similarity between the first positive instance and a set of positive instances and negative instances generated from the training batch of query samples. For example, at the t-th iteration, the instance-class probability p_i^t∈^s+b of x_i on the set B={{tilde over (x)}_i}_i=1^s∪{b_i}_i=1^b is computed.

At step 308, the first positive instance corresponding to the first unlabeled query sample is removed from the set of positive instances and negative instances.

At step 310, a second instance probability distribution is computed based on a second semantic similarity between the first positive instance and remaining instances in the set of positive instances and negative instances. For example, {tilde over (x)}_i is removed from the set B to estimate the similarity between x_i and the remaining instances in B.

At step 312, a first self-label is generated by combining a one-hot label of the first unlabeled query, the first label estimation and the second label estimation. For example, the one-hot label y_i and two label estimations, i.e. pi and qi, are linearly combined to obtain more accurate, robust and informative label y_i^t of x_i at the t-th iteration.

At step 314, an encoded output based on a contrastive input of the set of positive instances and negative instances is generated by a machine learning model, e.g., networks f( ) and g( ).

At step 316, a contrastive loss objective may be computed based at least in part on the generated self-label, e.g., as described in relation to module 130 in FIG. 1.

At step 318, the machine learning model is updated based on the contrastive loss objective via backpropagation. For example, the online network f( ) 110 is updated by the computed loss 135, e.g., via backpropagation, by fixing the parameters of the target network g( ) 120. The target network g( ) 120 is then updated via exponential moving average (EMA), e.g., ξ=(1−ι)ξ+ιω where ξ denotes the parameters of g( ) and ι∈(0,1) is a constant.

FIG. 4 is a simplified logic flow diagram illustrating an aspect of using momentum mix-up to refine the contrastive learning described in FIGS. 1-3, according to some embodiments. One or more of the processes 402-410 of method 400 may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes 402-410. In some embodiments, method 400 may correspond to the method used by the module 530 described in relation to FIG. 5.

Method 400 uses momentum mix-up to further reduce the possible label noise in realistic data and increase the diversity of data as well. Continuing on from step 304 in FIG. 3, at step 402, a second positive instance (which may be different from the first positive instance described in FIG. 3) is randomly sampled from the set of positive instances. At step 404, a virtual query sample is constructed by a linear combination of the first query sample and a second positive instance corresponding to the second query sample. At step 406, a virtual label is constructed by a linear combination of the first self-label and the second self-label corresponding to the second query sample. For example, the virtual query sample x_i′ and the virtual label y_i′ are computed by:

x _i ′=θx _i+(1−θ){tilde over (x)}_k,y_i′=θy_i+(1−θ)y_k,

where {tilde over (x)}_k is randomly sampled from the key set {{tilde over (x)}_i}_i=1^s, y_i denotes the self-label generated by module 128, θ∈[0,1] obeys the beta distribution Beta(κ,κ). Here x_i and {tilde over (x)}_i share the same label y_i on the set B={{tilde over (x)}_i}_i=1^s∪{b_i}_i=1^b, since they come from the same instance. The virtual label is referred to as “momentum mix-up”, as the sample {tilde over (x)}_k is fed into the momentum-updated network g_ξ, and plays a contrastive key for instance discrimination.

At step 408, a second contrastive loss objective may be computed by using the virtual sample and the virtual label, e.g., to replace x_i and the self-label y_i at loss module 130 in FIG. 1.

Thus, in this way, momentum mix up can further improve the accuracy of the label y_i′ compared with the traditionally used one hot labels. The virtual sample x_i′ has two positive keys x_i and {tilde over (x)}_k. Accordingly, the component {tilde over (x)}_k in x_i′=θx_i+(1−θ){tilde over (x)}_k directly increases the similarity between the query x_i′ and its positive key {tilde over (x)}_k in B. So the label weight (1−θ) of label y_i′ on the key {tilde over (x)}_k to bring x_i′ and {tilde over (x)}_k together is relatively accurate, as x_i′ really contains the semantic information of {tilde over (x)}_k. Meanwhile, the sum of label weights in y_i′ on remaining instance in B\{tilde over (x)}_k is scaled by θ, which also scales the possible label noise on instances in B\{tilde over (x)}_k smaller due to θ<1.

Another advantage of momentum mix-up is strong augmentation. It is observed that directly using strong augmentation in contrastive learning may lead to performance degradation, as the instance obtained by strong augmentation often heavily differs from the one with weak augmentation. As aforementioned, the component {tilde over (x)}_k in x_i′=θx_i+(1−θ){tilde over (x)}_k increases the similarity between the query instance x_i′ and the key instance {tilde over (x)}_k in B, even though (x_i, {tilde over (x)}_i) is obtained via strong augmentation. So the momentum mix-up could reduce the matching difficulty of positive instances, and promotes the learning of online network.

At step 410, a weighted sum of a contrastive loss objective computed based on one-hot labels and the second contrastive loss objective may be optionally computed as a training objective. For example, the combined training objective may be defined as:

(w)=(1−λ)_c(w,{(x_i,y_i)})+λ_c(w,{(x_i′,y_i′)}),

where _c(w,{(x_i,y_i)}) denotes the vanilla contrastive loss with one-hot label y_i, _c(w,{(x_i′,y_i′)}) denotes the momentum mix up loss with label y_i′ estimated by our self-labeling refinery, and λ is a constant. Method 400 may then proceed to step 318 in FIG. 3 to update the machine learning model.

FIG. 5 is a simplified diagram of a computing device for implementing the proposed contrastive learning with self-labeling refinement, according to some embodiments. As shown in FIG. 5, computing device 500 includes a processor 510 coupled to memory 520. Operation of computing device 500 is controlled by processor 510. And although computing device 500 is shown with only one processor 510, it is understood that processor 510 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and/or the like in computing device 500. Computing device 500 may be implemented as a stand-alone subsystem, as a board added to a computing device, and/or as a virtual machine.

Memory 520 may be used to store software executed by computing device 500 and/or one or more data structures used during operation of computing device 500. Memory 520 may include one or more types of machine readable media. Some common forms of machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.

Processor 510 and/or memory 520 may be arranged in any suitable physical arrangement. In some embodiments, processor 510 and/or memory 520 may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system-on-chip), and/or the like. In some embodiments, processor 510 and/or memory 520 may include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, processor 510 and/or memory 520 may be located in one or more data centers and/or cloud computing facilities.

In some examples, memory 520 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor 510) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory 520 includes instructions for a contrastive learning module 530 that may be used to implement and/or emulate the systems and models, and/or to implement any of the methods described further herein. In some examples, the contrastive learning module 530, may receive an input 540, e.g., such as unlabeled image instances, via a data interface 515. The data interface 515 may be any of a user interface that receives a user uploaded image instance, or a communication interface that may receive or retrieve a previously stored image instance from the database. The contrastive learning module 530 may generate an output 550, such as classification result of the input 540.

In some embodiments, the contrastive learning module 530 may further includes the self-labeling module 531 and a momentum mix-up module 532. Further functionality of the self-labeling module 531 may be discussed in relation to FIGS. 2 and 3, and further functionality of the momentum mix-up module 532 may be discussed in relation to FIG. 4. In some examples, the contrastive learning module 530 and the sub-modules 531-333 may be implemented using hardware, software, and/or a combination of hardware and software.

Example Performance

The dataset ResNet50 with a 3 layered MLP head for CIFAR10 and ImageNet are used. The contrastive learning with self-labeling refinement model, referred to as CLEAN, is first pretrained and then train a linear classifier on top of 2048-dimensional frozen features provided by ResNet50. With dictionary size 4,096, it is pretrained for 2,000 epochs on CIFAR10. Dictionary size on ImageNet is 65,536. For linear classifier, it is trained for 200 and 100 epochs on CIFAR10 and ImageNet, respectively.

Standard data augmentations are used as described in He et al., Momentum contrast for unsupervised visual representation learning. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 9729-9738, 2020, for pretraining and test unless otherwise stated. For example, for test, normalization is performed on CIFAR10, and employ center crop and normalization on ImageNet. For CLEAN, we set τ=0.2, τ′=0.8, κ=2 in Beta(κ,κ) on CIFAR10, and τ=0.2, τ′=1, κ=0.1 on ImageNet. For confidence μ, it is increased as μ_t=m₂−(m₂−m₁)(cos(πt/T)+1)/2 with current iteration t and total training iteration T. Set m₁=0, m₂=1 on CIFAR 10, and m₁=0.5, m₂=10 on ImageNet. For KNN on CIFAR10, its neighborhood number is 50 and its temperature is 0.05.

For CIFAR 10, to fairly compare with Lee et al., Mixco: Mix-up contrastive learning for visual representation. arXiv preprint arXiv:2010.06300, 2020, each image is cropped into two views to construct the loss. For ImageNet, CLSA (described in Wang & Qi, Contrastive learning with stronger augmentations, 2021) and train CLEAN in two settings. CLEAN-Single uses a single crop in momentum mixup loss _c(w,{(x_i′,y_i′)}) that crops each image to a smaller size of 96×96, without much extra computational cost to process these small images. CLEAN multi-crops each image into five sizes 224×224, 192×192, 160×160, 128×128, and 96×96 and averages their momentum mixup losses. This ensures a fair comparison with CLSA and SwAV. Moreover, we use strong augmentation strategy in CLSA.

Specifically, for the above small image, an operation is randomly selected from 14 augmentations used in CLSA, and apply it to the image with a probability of 0.5, which is repeated 5 times. “(strong)” is used to mark whether we use strong augmentations on the small images in momentum mix-up loss. Thus, CLEAN has almost the same training cost with CLSA. For vanilla contrastive loss on ImageNet, weak augmentations are always used.

From Table 1 in FIG. 6, one can observe that with weak or strong augmentations, CLEAN always surpasses the baselines on CIFAR10. Moreover, CLEAN with strong (weak) augmentation improves supervised baseline by 1%(0.6%).

Table 2 in FIG. 7 also shows that for ImageNet under weak augmentation setting, for 200(800) epochs CLEAN-Multi respectively brings 0.8% (0.6%) improvements over SwAV; with 200 (800) epochs, CLEAN-Single also beats the runner-up MixCo (i-Mix and SimSiam). Note, BYOL outperforms CLEAN-Single but was trained 1,000 epochs. With strong augmentation, CLEAN-Single and CLEAN-Multi also respectively outperform CLSA-Single and CLSA-Multi. Moreover, our self-supervised accuracy 76.4% is very close to the accuracy 76.5% of supervised baseline. These results show the superiority and robustness of CLEAN, thanks to the complementary self-labeling refinery and momentum mixup in CLEAN which both improve label quality and thus bring semantically similar samples together.

The pretrained CLEAN model is also pretrained on VOC (described in Everingham et al., The pascal visual object classes (voc) challenge, Int'l. J. Computer Vision, 88(2):303-338, 2010) and COCO (Lin et al., Microsoft coco: Common objects in context. In Proc. European Conf. Computer Vision, pp. 740-755. Springer, 2014). For classification, a linear classifier is trained upon ResNet50 100 epochs by SGD. For object detection, the same protocol is used in He et al., Momentum contrast for unsupervised visual representation learning. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 9729-9738, 2020, to fine-tune the pretrained ResNet50 based on detectron2 (described in Wu et al., detectron 2, 2019) for fairness. On VOC, detection head is trained with VOC07+12 train val data and tested on VOC07 test data. On COCO, the head is trained on train2017 set and evaluate on the val2017.

Table 3 in FIG. 8 reports the results on VOC and COCO. CLEAN consistently outperforms the compared state-of-the-art approaches on both classification and object detection tasks. Moreover, it enjoys better performance than supervised method pretrained on ImageNet. These results show the superior transferability and generalization of CLEAN.

CLEAN is trained for 1,000 epochs on CIFAR10 to investigate the effects of each component in CLEAN using strong augmentation. Table 4 in FIG. 9 shows the individual benefits of each component, including the label estimations p and q in the self-labeling refinery, and the momentum mixup. Table 5 in FIG. 9 shows the stable performance of CLEAN on CIAFR 10 when tuning regularization parameter λ in the combined contrastive loss in a relatively large range, thus testifying the robustness of CLEAN.

Then the momentum mix-up is compared with vanilla mix-up in the concurrent works (described in Kim et al., Mixco: Mix-up contrastive learning for visual representation. arXiv preprint, arXiv:2010.06300, 2020; and Lee et al., Hi-mix: A strategy for regularizing contrastive representation learning. arXiv preprint arXiv:2010.08887, 2020). Specifically, one-hot label is used in MoCo and replace {tilde over (x)}_j in with the query x_j to obtain “MoCo+mix-up”, and CLEAN with one-hot label can be viewed as “MoCo+momentum mixup”. Then these methods are trained for 1,000 epochs on CIFAR 10 with weak/strong augmentation, and 200 epochs on ImageNet with weak augmentations. Table 6 in FIG. 10 shows that with weak augmentation, momentum mix-up always outperforms vanilla mix-up in (Kim et al., 2020; Lee et al., 2020). Moreover, momentum mix-up using strong augmentation has accuracy 94.8% and improves its weak augmentation version, while vanilla mix-up with strong augmentation suffers from performance degradation. It is because that momentum mix-up can well reduce the possible label noise, especially for strong augmentations, and thus can enhance the performance more.

Additional Performance Analysis

Performance of the self-labeling refinery on label-corrupted data is analyzed as follows. Let {c_i}_i=1^K⊂^d be K vanilla samples belonging to K≤K semantic classes, and {(x_i,y_i)}_i=1ⁿ∈^d× be the random crops of {c_i}_i=1^K. Since in practice, one often cares more the semantic class prediction performance of a model which often directly reflects the performance on the downstream tasks, it is assumed that the labels {y_i}_i=1ⁿ denote corrupted semantic-class labels. Accordingly, it is analyzed whether self-labeling refinery can refine the corrupted labels {y_i}_i=1ⁿ and whether it helps a model learn the essential semantic-class knowledge of {x_i}_i=1ⁿ. Finally, while allowing for multiple classes, it is assumed the labels are scalars and take values in [−1,1] interval for simplicity. The label-corrupted dataset is defined below.

Definition 1((ρ,ε,δ)-corrupted dataset). Let {(x_i,y_i*)}_i=1ⁿ denote the pairs of crops (augmentations) and ground-truth semantic label, where crop x_i generated from the t-th sample c_t obeys ∥x_i−c_t∥₂≤ε with a constant ε, and y_i*∈{γ_t}_i=1^Kof x_i is the label of c_t. Moreover, samples and the crops are normalized, i.e. ∥c_i∥₂=∥x_k∥₂=1(∀i,k). Each c_i has n_i crops, where

$c_l\frac{n}{K}\le n_i\le c_u\frac{n}{K}$

with two constants c_l and c_u. Moreover, the classes are separated:

|γ_i−γ_k|≥δ,∥c_i−c_k∥₂>2ε,(∀i≠k),

where δ is the label separation. A (ρ,ε,δ)-corrupted dataset {(x_i,y_i)}_i=1ⁿ obeys the above conditions but with corrupted label {y_i}_i=1ⁿ. Specifically, for each sample c_i, at most ρn_i augmentations are assigned to wrong labels in {γ_i}_i=1^K.

Then a network of one hidden layer is studied as an example to investigate the label refining performance of the method:

x∈ ^d f(W,x)=v^Tϕ(Wx)

where W∈^k×d and v∈^k are network parameters, and ϕ is an activation function. v is fixed to be a unit vector where half the entries are 1/√{square root over (k)} and other half are −1/√{square root over (k)} to simplify exposition. So it is only optimized over W that contains most network parameters and will be shown to be sufficient for label refinery. Then given a (ρ,ε,δ)-corrupted dataset {(x_i,y_i)}_i=1ⁿ, at the t-iteration we train the network via minimizing the quadratic loss:

_t(W)=½Σ_i=1ⁿ(y_i^t−f(W,x_i))²=½∥y^t−f(W,X)∥₂².

Here the label y_i^t, of sample x_i is estimated at module 128 as described in FIG. 2, where p_i^t=f(W_t,{tilde over (x)}_i) denotes predicted label by using the positive {tilde over (x)}_i of x_i, i.e. ∥{tilde over (x)}_i−c_l∥₂≤ε if x_i is augmented from vanilla sample c_l. If setting β_t=0 and τ′=1 for simplicity, as (i) performing nonlinear mapping on network output greatly increases analysis difficulty; (ii) the self-label refinery is still provably sufficient to refine labels when β_t=0 and τ′=1. Then W may be updated via gradient descent algorithm:

w _t+1 =w _t−η∇_t(W_t),

where η is a learning rate. According to network convergence analysis, gradient descent and quadratic loss may be used, since (i) gradient descent is expectation version of stochastic one and often reveals similar convergence behaviors; (ii) one can expect similar results for other losses, e.g. cross entropy, but quadratic loss gives simpler gradient computation. For analysis, mild assumptions are imposed on the network and the self-labeling refinery, which are widely used in network analysis.

Assumption 1. For the network (6), suppose the activation ϕ and its first- and second-order derivatives obey |ϕ(0)|, |ϕ′(z)|, |ϕ″(z)|≤Γ for ∀z and some Γ≥1. Moreover, the entries of initialization W₀ obey i.i.d. (0,1).

Assumption 2. Define network covariance matrix Σ(C)=(CC^T)⊙_u[ϕ′(Cu)ϕ′(Cu)^T] where C=[c₁ . . . c_K]^T, u˜(0,I),⊙ is the elementwise product. Let λ(C)>0 be the minimum eigenvalue of Σ(C). For label refinery, assume

$3\sqrt{n}\sum _{t=0,}^{t_0-1}❘{\alpha }_t-{\alpha }_{t+1}❘\le {\psi }_1{f\left(W_0,X\right)-y^{*}}_2\mathrm{and}3\sqrt{n}\sum _{t=0}^{t_0-1}{\left(1-\frac{\eta {\alpha }^2}{4}\right)}^{t_0-t}❘{\alpha }_t-{\alpha }_{t+1}❘\le {\psi }_2{f\left(W_0,X\right)-y^{*}}_2^2\mathrm{where}{t}_0=\frac{c_{1}K}{\eta n\lambda \left(C\right)}\log \left(\frac{\Gamma \sqrt{n\log K}}{\left(1-{\alpha }_{\max }\right)\rho }\right)$

with three constants ψ₁,ψ₂ and c₁. Here α_max=max_i≤t≤t0 α_t.

Assumption 1 is mild, as most differential activation functions, e.g. softplus and sigmoid, satisfy it, and the Gaussian initialization is used in practice. It is assumed that Gaussian variance to be one for notation simplicity, but the technique is applicable to any constant variance. Assumption 2 requires that the discrepancy between α_t and α_t+1 until some iteration number t₀ are bounded, which holds by setting proper α_t. For λ(C), prior works empirically and theoretically show λ(C)>0. Based on the assumptions, we state our results in Theorem 2 with constants c₁˜c₆.

Theorem 2. Assume {(x_i,y_i)}_i=1ⁿ is a (ρ,ε,δ)-corrupted dataset with noiseless labels {y_i*}_i=1ⁿ. Let

$\xi =\log \left(\frac{\Gamma \sqrt{n\log K}}{\rho }\right).$

Suppose ε and the number k of hidden nodes satisfy

$\epsilon \le c_2\min \left(\frac{\lambda \left(C\right)}{K{\Gamma }^2{\xi }^6},\frac{\rho }{{\alpha }_{\max }}\right),k\ge \frac{c_3{K}^2{\Gamma }^{10}{\xi }^6{C}^4}{{\alpha }_{\max }^2{\lambda \left(C\right)}^4}.$

Let

${\psi }^{\prime }=1+\frac{{\psi }_1}{2}+\sqrt{{\psi }_2}.$

If step size

$\eta \le \frac{K}{2c_{up}n{\Gamma }^2{C}^2},$

with probability 1−3/K¹⁰⁰−K exp(−100d), after

$t_0=\frac{c_{4}K}{\eta n\lambda \left(C\right)}\log \left(\frac{\Gamma \sqrt{n\log K}}{\left(1-{\alpha }_{\max }\right)\rho }\right)$

iterations, the gradient descent (7) satisfies:(1) The discrepancy between the label y^t estimated by our refinery (5) and the true label y* of data {x_i}_i=1ⁿ is bounded:

$\frac{1}{\sqrt{n}}{{\overline{y}}^t-y^{*}}_2\le \frac{1-{\alpha }_t}{\sqrt{n}}{y-y^{*}}_2+{\alpha }_t\zeta ,$

where ζ=4ρ+c₅εψ′KΓ³ξ√{square root over (log K)}/λ(C), y*=[y₁*, . . . , y_n*]. Moreover, if

$\rho \le \frac{\delta }{32},\epsilon \le c_6\delta \min \left(\frac{{\lambda \left(C\right)}^2}{{\psi }^{\prime }{\Gamma }^5{K}^2{\xi }^3},\frac{1}{\Gamma \sqrt{d}}\right),1-\frac{3}{4}\delta \le {\alpha }_t$

the estimated label y^t predicts true label y_i* of any crop x_i: γ_k*=y_i* with k*=argmin_i≤k≤K|y_i^t−γ_k|.(2) By using the refined label y^t in (5) to train network, the error of network prediction on {x_i}_i=1ⁿ is upper bounded

$\frac{1}{\sqrt{n}}{f\left(W_t,X\right)-y^{*}}_2\le \zeta ,$

where f(W_t,X)=[f(W_t,x₁), . . . , f(W_t,x_n)]. If assumptions on ρ and ε in (1) hold, for vanilla sample c_k (∀k=1 . . . K), network f(W_t,⋅) predicts the true semantic label γ_k of its any augmentation x that obeys ∥x−c_k∥₂≤ε_i:γ_k*=γ_k with k*=argmin_i≤i≤K|f(W_t,x)−γ_i|

The first part result in Theorem 2 shows that after training iterations t₀, the discrepancy between the label y^t estimated by the label refinery and ground truth label y* of cropped training data {x_i}_i=1ⁿ is upper bounded by (∥y−y*∥₂+ζ). Both factors ∥y−y*∥₂ and ρ in the factor ζ reflect the label error of the provided corrupted label y. Another important factor in is the smallest eigenvalue λ(C) of network covariance matrix Σ(C) in Assumption 2. Typically, the performance of a network heavily relies on the data diversity even without label corruption. For instance, if two samples are nearly the same but have different labels, then the learning of a network is difficult. λ(C) can quantify this data diversity, as one can think of λ(C) as a condition number associated with the network which characterizes the diversity of the vanilla samples {c_i}_i=1ⁿ. Intuitively, if there are two similar vanilla samples, Σ(C) is trivially rank deficient and the minimum eigenvalue of Σ(C) is small, meaning more challenges for distinguishing the augmentations x generated from c_i. Otherwise, the more distinct the vanilla samples, the larger λ(C) is and the smaller label error is ζ. Moreover, when the label corruption ratio ρ and the augmentation distance E are small, the label y_i^t, estimated by the label refinery can predict the true semantic label y_i* for any crop sample x_i, and thus can supervise a network to learn the essential semantic-class knowledges of augmentation samples {x_i}_i=1ⁿ

The second result in Theorem 2 shows that by using the refined label y^t in Eqn. (5) to train network f(W,⋅), the error of network prediction on augmentations {x_i}_i=1ⁿ can be upper bounded by ζ. Similarly, the factor ρ and λ(C) in ζ respectively reflect the initial label error and the data diversity, which both reflect the learning difficulty for a model on the augmentation data {(x_i,y_i)}_i=1ⁿ. More importantly, the results also guarantee the test performance of the trained network f(W_t,⋅). Specifically, when the label corruption ratio ρ and sample augmentation distance ε are small, for any vanilla sample c_k (∀k=1 . . . K), the network f(W_t,⋅) trained by the label refinery can exactly predict the true semantic label γ_k of its any augmentation x (i.e. ∥x−c_k∥₂≤ε). These results accord with the results in Theorem 1 that shows the more accurate of training labels, the better generalization of the trained network. These results show the effectiveness of the refined labels by the method.

Some examples of computing devices, such as computing device 100 may include non-transitory, tangible, machine readable media that include executable code that when run by one or more processors (e.g., processor 110) may cause the one or more processors to perform the processes of method 300. Some common forms of machine readable media that may include the processes of method 300 are, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.

This description and the accompanying drawings that illustrate inventive aspects, embodiments, implementations, or applications should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the embodiments of this disclosure Like numbers in two or more figures represent the same or similar elements.

In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the invention should be limited only by the following claims, and it is appropriate that the claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A method for contrastive learning with self-labeling refinement, the method comprising: receiving a training batch of query samples;generating a first positive instance corresponding to the first query sample;computing a first instance probability distribution based on a first semantic similarity between the first positive instance and a set of positive instances and negative instances generated from the training batch of query samples;removing the first positive instance from the set of positive instances and negative instances;computing a second instance probability distribution based on a second semantic similarity between the first positive instance and remaining instances in the set of positive instances and negative instances;generating, for the first query sample, a first self-label by combining a first one-hot label of the first query sample, the first instance probability distribution and the second instance probability distribution;generating, by a machine learning model, an encoded output based on a contrastive input of the set of positive instances and negative instances;computing a contrastive loss objective based on the encoded output supervised by the generated self-label; andupdating the machine learning model based on the contrastive loss objective via backpropagation.

2. The method of claim 1, wherein the set of positive instances and negative instances is obtained by: generating a first number of positive instances corresponding to query samples in the training batch;generating a second number of negative instances corresponding to query samples in the training batch, wherein the second number is greater than the first number.

3. The method of claim 1, wherein the first instance probability distribution is computed by: computing a similarity metric between the first positive instance and a first instance in the set of positive instances and negative instances; anddividing the similarity metric by a sum of all similarity metrics over all instances in the set of positive instances and negative instances.

4. The method of claim 3, wherein the similarity metric is computed by: encoding, by an online network of the machine learning model, the first query sample into a first feature representation;encoding, by a target network of the machine learning model, the first instance into an encoded instance; andcomputing an exponential similarity between the first feature representation and the encoded instance in an encoded space.

5. The method of claim 1, wherein the first self-label is generated by a linear combination with a first weight and a second weight, and the method comprises: determining the first weight based on a first maximum probability among probability entries of the first instance probability distribution; anddetermining the second weight based on a second maximum probability among probability entries of the second instance probability distribution.

6. The method of claim 1, further comprising: constructing a virtual query sample by a first linear combination of the first query sample and a second positive instance corresponding to a second query sample;constructing a virtual label by a second linear combination of the first self-label and a second self-label corresponding to the second query sample.

7. The method of claim 6, wherein the second positive instance is randomly sampled from the set of positive instances.

8. The method of claim 6, wherein the first or the second linear combination includes a weight that is randomly sampled according to Beta distribution.

9. The method of claim 6, further comprising: computing the contrastive loss objective supervised by the virtual label.

10. The method of claim 1, wherein the contrastive loss objective is computed as a cross-entropy between the first self-label and a similarity metric of the first query sample to the set of positive instances and negative instances.

11. A system for contrastive learning with self-labeling refinement, the system comprising: a communication interface that receives a training batch of query samples;a memory storing a machine learning model and a plurality of processor-executable instructions; andone or more processors executing the plurality of processor-executable instructions to perform operations comprising: generating a first positive instance corresponding to the first query sample;computing a first instance probability distribution based on a first semantic similarity between the first positive instance and a set of positive instances and negative instances generated from the training batch of query samples;removing the first positive instance from the set of positive instances and negative instances;computing a second instance probability distribution based on a second semantic similarity between the first positive instance and remaining instances in the set of positive instances and negative instances;generating, for the first query sample, a first self-label by combining a first one-hot label of the first query sample, the first instance probability distribution and the second instance probability distribution;generating, by the machine learning model, an encoded output based on a contrastive input of the set of positive instances and negative instances;computing a contrastive loss objective based on the encoded output supervised by the generated self-label; andupdating the machine learning model based on the contrastive loss objective via backpropagation.

12. The system of claim 11, wherein the set of positive instances and negative instances is obtained by: generating a first number of positive instances corresponding to query samples in the training batch;generating a second number of negative instances corresponding to query samples in the training batch, wherein the second number is greater than the first number.

13. The system of claim 11, wherein the first instance probability distribution is computed by: computing a similarity metric between the first positive instance and a first instance in the set of positive instances and negative instances; anddividing the similarity metric by a sum of all similarity metrics over all instances in the set of positive instances and negative instances.

14. The system of claim 13, wherein the similarity metric is computed by: encoding, by a feature encoder of the machine learning model, the first query sample into a first feature representation;encoding, by a projection head of the machine learning model, the first instance into an encoded instance; andcomputing an exponential similarity between the first feature representation and the encoded instance in an encoded space.

15. The system of claim 11, wherein the first self-label is generated by a linear combination with a first weight and a second weight, and the operations further comprise: determining the first weight based on a first maximum probability among probability entries of the first instance probability distribution; anddetermining the second weight based on a second maximum probability among probability entries of the second instance probability distribution.

16. The system of claim 11, wherein the operations further comprise: constructing a virtual query sample by a first linear combination of the first query sample and a second positive instance corresponding to a second query sample;constructing a virtual label by a second linear combination of the first self-label and a second self-label corresponding to the second query sample.

17. The system of claim 16, wherein the second positive instance is randomly sampled from the set of positive instances.

18. The system of claim 16, wherein the first or the second linear combination includes a weight that is randomly sampled according to Beta distribution.

19. The system of claim 16, wherein the operations further comprise: computing the contrastive loss objective supervised by the virtual label.

20. The system of claim 11, wherein the contrastive loss objective is computed as a cross-entropy between the first self-label and a similarity metric of the first query sample to the set of positive instances and negative instances.