FEDERATED LEARNING SYSTEM AND METHOD OF PROTECTING DATA DIGEST

Abstract

A federated learning method of protecting data digest includes: sending a general model to multiple clients by a moderator, generating encoded features according to raw data and training by each client, the training includes: updating the general model to generate a client model, selecting at least two encoded features and at least two labels to compute a feature weighted sum and a label weighted sum, sending a digest and update parameters of the client model to the moderator, where the digest includes a sum of the feature weighted sum and a noise and the label weighted sum, and performing the following steps by the moderator: determining an absent client and a present client, generating a replacement model according to the general model and the absent client, generating an aggregation model according to the available client and the replacement model, and training the aggregation model to update the general model.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This non-provisional application claims priority under 35 U.S.C. § 119(a) on Patent Application No(s). 202310239239.3 filed in China on Mar. 13, 2023, the entire contents of which are hereby incorporated by reference.

BACKGROUND

1. Technical Field

The present disclosure relates to federated learning, and more particularly to a federated learning system and method of protecting data digest.

2. Related Art

Federated Learning (FL) addresses many privacy and data sharing issues through cross-device and distributed learning via central orchestration. Existing FL methods mostly assume a collaborative setting among clients can tolerate temporary client disconnection from the moderator.

In practice, however, extended client absence or departure can happen due to business competitions or other non-technical reasons. The performance degradation can be severe when the data are unbalanced, skewed, or non-independent-and-identically-distributed (non-IID) across clients.

Another issue arises when the moderator needs to evaluate and release the model to the consumers. As private client data are not accessible by the moderator, the representative data would be lost when clients cease to collaborate, resulting in largely biased FL gradient update and long-term training degradation. The naive approach of memorizing gradients during training is not a suitable solution, as gradients become unrepresentative very quickly as iteration progresses.

Overall, current federated learning still fails to perform well in the following three scenarios in combinations: (1) unreliable clients, (2) training after removing clients, and (3) training after adding clients.

SUMMARY

Accordingly, the present disclosure provides a federated learning system and method of protecting data digest. This is a federated learning framework that can address client absence by synthesizing representative client data at the moderator. The present disclosure proposes a feature-mixing solution to reduce the privacy concerns and uses a feature disturbance method to protect the digest.

According to an embodiment of the present disclosure, a federated learning method of protecting data digest comprises: sending a general model to each of a plurality of client devices by a moderator; executing a digest producer by each of the plurality of client devices to generate a plurality of encoded features according to a plurality of raw data; performing a training procedure by each of the plurality of client devices, wherein the training procedure comprises: updating the general model to generate a client model according to the plurality of raw data, the plurality of encoded features, a plurality of labels corresponding to the plurality of encoded features, and a present client loss function; selecting at least two of the plurality of encoded features to compute a feature weighted sum, computing a sum of the feature weighted sum and noise; selecting at least two of the plurality of labels to compute a label weighted sum, and sending the sum and the label weighted sum to the moderator as a digest when receiving a digest request; and sending an update parameter of the client model to the moderator; determining an absent client and a present client among the plurality of client devices by the moderator; generating a replacement model according to the general model, the digest of the absent client and an absent client loss function by the moderator; performing an aggregation to generate an aggregation model according to the update parameter of the client model of the present client and an update parameter of the replacement model of the absent client by the moderator; and training the aggregation model to update the general model according to a moderator loss function by the moderator.

According to an embodiment of the present disclosure, a federated learning system of protecting data digest comprises a plurality of client devices and a moderator. Each of the plurality of client devices comprises: a first processor configured to execute a digest producer to generate a plurality of encoded features according to a plurality of raw data, further configured to update a general model to generate a client model according to the plurality of raw data, the plurality of encoded features, a plurality of labels corresponding to the plurality of encoded features, and a present client loss function, and further configured to select at least two of the plurality of encoded features to compute a feature weighted sum, compute a sum of the feature weighted sum and noise and select at least two of the plurality of labels to compute a label weighted sum when receives a digest request; and a first communication circuit electrically connected to the first processor and configured to send the feature weighted sum and the label weighted sum as a digest and send an update parameter of the client model. The moderator is communicably connected to each of the plurality of client devices, and comprises: a second communication circuit configured to send the general model to each of the plurality of client devices; and a second processor electrically connected to the second communication circuit, wherein the second processor is configured to determine an absent client and a present client among the plurality of client devices, generate a replacement model according to the digest of the absent client and an absent client loss function, perform an aggregation to generate an aggregation model according to the update parameter of the client model of the present client and an update parameter of the replacement model of the absent client, and train the aggregation model to update the general model according to a moderator loss function.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from the detailed description given hereinbelow and the accompanying drawings which are given by way of illustration only and thus are not limitative of the present disclosure and wherein:

FIG. 1 is a block diagram of the federated learning system of protecting data digest according to an embodiment of the present disclosure;

FIG. 2 is an architectural diagram of the digest producer and the client model according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of computing the feature weighted sum according to an embodiment of the present disclosure;

FIG. 4 is an architectural diagram of the guidance producer and the replacement model according to an embodiment of the present disclosure;

FIG. 5 and FIG. 6 are overview diagrams of the federated learning system of protecting data digest according to an embodiment of the present disclosure;

FIG. 7 is a flow chart of the federated learning method of protecting data digest according to an embodiment of the present disclosure;

FIG. 8 is a detailed flow chart of the step in FIG. 7;

FIG. 9 is a detailed flow chart of the step in FIG. 8;

FIG. 10 is a detailed flow chart of a step in FIG. 7;

FIG. 11 is a detailed flow chart of a step in FIG. 7;

FIG. 12 is a detailed flow chart of the step in FIG. 7;

FIG. 13 is a schematic diagram showing the guidance reconstructed in different ways compared to the raw data;

FIG. 14(a) to FIG. 14(c) show the visual results of changing the parameters of feature disturbance methods;

FIG. 15 demonstrates the testing accuracy of the model with varying levels of noise added to the digest; and

FIG. 16(a) to FIG. 16(d) are performance comparison diagrams of models in different training scenarios.

DETAILED DESCRIPTION

In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. According to the description, claims and the drawings disclosed in the specification, one skilled in the art may easily understand the concepts and features of the present invention. The following embodiments further illustrate various aspects of the present invention, but are not meant to limit the scope of the present invention.

The detailed description of the embodiments of the present disclosure includes a plurality of technical terms, and the following are the definitions of these technical terms:

Client, the endpoint that contributes the data to join a distributed training or federated learning, also called “client device”.

Moderator, the service provider that collects the models from the clients to aggregate a general model for providing the service.

Raw data, the data that are held by a client and need to be protected, also called “private data”.

Digest, a sharable representation that can represent the raw data. No privacy concerns are included in the digest. The dimension of the digest is usually but not limited to lower than the raw data.

Guidance, the data to support model training with client absence. The domains of the guidance and the private data are usually the same.

Client model, the model owned by each client and trained according to the raw data by the client.

General model, the model owned by the moderator that is aggregated from the client models.

Stochastic Gradient Decent (SGD), an optimization process to update the parameters of a machine learning model based on predefined loss functions.

Federated learning (FL), a collaborative training framework to train a machine learning model without sharing client data to protect the data privacy.

Machine learning, a field of study that gives computers the ability to learn without being explicitly programmed.

Loss function: the objective functions of the optimizing process for training a machine learning model.

Differential Privacy (DP), DP is a rigorous mathematical definition of privacy. DP technologies allow sharing data information without expose any individual sample.

The present disclosure proposes a federated learning system of protecting data digest (also called FedDig framework) and an operating method using this system. FIG. 1 is a block diagram of the federated learning system of protecting data digest according to an embodiment of the present disclosure. As shown in FIG. 1, the federated learning system of protecting data digest includes a plurality of client devices Ci, Cj and a moderator Mo. The present disclosure does not limit the number of client device. For the convenience of illustration, FIG. 1 shows two client devices Ci, Cj as an example.

The hardware architecture of each of the client devices Ci, Cj is basically the same. The client device Ci in FIG. 1 is used as an example for illustration here, and the implementation example of the client device Cj can refer to the client device Ci. The client device Ci includes a first processor i1, a first communication circuit i2, and a first storage circuit i3. The first communication circuit i2 is electrically connected to the first processor i1. The first storage circuit i3 is electrically connected to the first processor i1 and the first communication circuit i2. In an embodiment, one of the following devices may be employed as the client device Ci: a server, a personal computer, a mobile computing device, and any electronic device for training a machine learning model.

The client device Ci is configured to collect raw data. The raw data include a private part and a non-private part other than the private part. For example, the raw data is an integrated circuit diagram, and the private part is a key circuit design in the integrated circuit diagram. For example, the raw data is a product design layout, and the private part is the product logo. For example, the raw data is the text, and the private part is the personal information such as name, phone, and address.

The first processor i1 is configured to execute a digest producer _R, and thus generating a plurality of encoded features according to the plurality of raw data. In the embodiment shown in FIG. 1, the digest producer _R is a software running on the first processor i1, however, the present disclosure does not limit the hardware configured to execute the digest producer _R. The digest producer _d may be stored in the first storage circuit i3 or an internal memory of the first processor i1.

In an embodiment, the federated learning system adopts an appropriate neural network model as the digest producer _R according to the type of raw data. For example, EfficientNetV2 may be adopted as the digest producer _R when the raw data is CIFAR-10 (Canadian Institute for Advanced Research), and VGG16 may be adopted as the digest producer _R when the raw data is EMINST (Extend Modified National Institute of Standards and Technology).

In an embodiment, the raw data is directly inputted to the digest producer _R to generate the encoded features. In another embodiment, the first processor i1 preprocesses the private part of the raw data before the raw data is inputted to the digest producer _R for the generation of the encoded features. For example, when the raw data is an image, the preprocessing is to crop out the private part from the image; when the raw data is a text, the preprocessing is to remove the specified field or to mask the specific string. The digest producer _R converts one piece of raw data into one encoded feature. In general, the dimension of raw data is greater than the dimension of encoded features.

If the number of samples of the raw data is K, after the digest producer _R generates K encoded features according to the K pieces of raw data, the first processor i1 updates the general model from the moderator Mo to generate the client model according to the K pieces of raw data, K encoded features, K labels corresponding to the K encoded features, and a present client loss function. In an embodiment, the number K of labels and the labels themselves are given manually.

FIG. 2 is an architectural diagram of the digest producer and the client model according to an embodiment of the present disclosure. The client model includes a first feature extractor F_R, a second feature extractor F_D, and a classifier C. The present disclosure does not limit the implementation of the client model. For example, the neural network model such as EfficientNetV2 and VGG16 may be adopted as the client model. These neural network models themselves already include the design of the feature extractor (which can be used as the first feature extractor F_R mentioned above) and a classifier. As for the second feature extractor F_D, for example, feature extractors in neural network models such as ResNet, UNet, EfficientNet, and MobieNet may be used for implementation. As shown in FIG. 2, the first processor i1 inputs the plurality of raw data Rⁱ into the first feature extractor F_R respectively to generate a plurality of first features (the number of the raw data Rⁱ is equal to the number of first features F_R). The first processor i1 inputs the plurality of raw data Rⁱ into the digest producer _R to generate the plurality of encoded features rⁱ. These encoded features rⁱ are inputted to the second feature extractor F_D to generate a plurality of second features. The first processor i1 inputs the concatenation of the first feature and the second feature to the classifier C to generate a predicted result . The first processor i1 further inputs the predicted result and an actual result yⁱ to a present client loss function, and adjusts a weight of at least one of the first feature extractor F_R, the second feature extractor F_D, and the classifier C according to an output of the present client loss function. In an embodiment, the present client loss function is shown in the following Equation 1:

$\begin{array}{cc}{ℒ}_{\mathrm{client}}^{\mathrm{avail}}={ℒ}_{\mathrm{ce}}\left({ℳ}^i\left(R^i,r^i\right),y^i\right),& \left(\mathrm{Equation}1\right)\end{array}$

• • where _client^avail is the present client loss function, _ce may adopt an appropriate loss function according to the purpose of different models while cross entropy is used as _ce in an embodiment, ⁱ is the client model of the client device Cⁱ, Rⁱ is the raw data, rⁱ denotes the encoded features, ⁱ(Rⁱ, d_Rⁱ)={tilde over (y)}^t represents the predicted result, and yⁱ is the actual result (also called label). The condition for the general model to complete training is that the output of the present client loss function _client^avail is smaller than a certain threshold. The general model trained at the client device Ci is called the client model ⁱ and is sent to the moderator device Mo.

When the first communication circuit i2 receives a digest request from the moderator Mo, the first processor i1 is further configured to select at least two of the encoded features rⁱ to compute a feature weighted sum, select at least two of the labels yⁱ to compute a label weighted sum, and compute a sum of the feature weighted sum and noise.

In an embodiment, the feature weighted sum is shown in the following Equation 2, and the label weighted sum is shown in the following Equation 3:

$\begin{array}{cc}d={\sum }_{k=1}^{\mathrm{SpD}}{w}_k{r}_k,& \left(\mathrm{Equation}2\right)\end{array}$ $\begin{array}{cc}{D}_y={\sum }_{k=1}^{\mathrm{SpD}}{w}_k{y}_k,& \left(\mathrm{Equation}3\right)\end{array}$

• • where d is the feature weighted sum, D_y is the label weighted sum, w_k is the weight, r_k is the encoded features, y_k is the label, SpD represents the number of samples included in each digest (Samples per Digest). In an embodiment, the weight w_k is set to a value of equally distributed SpD. For example, if SpD=4, then w₁=w₂=w₃=w₄=¼. However, the present disclosure does not limit the setting of weights w_k.

FIG. 3 is a schematic diagram of computing the feature weighted sum d according to an embodiment of the present disclosure. In this embodiment, it is assumed that the number of samples of the raw data Rⁱ is 6 and SpD=3. As shown in FIG. 3, the digest producer _R respectively generates 6 encoded features r1-r6 according to 6 pieces of raw data R1-R6. The first processor i1 performs a multiplication on the 6 encoded features r1-r6 and 6 default weights w₁-w₆ respectively, then performs an addition on the 3 multiplication results corresponding to r1-r3 to generate the feature weighted sum d₁, and performs an addition on the 3 multiplication results corresponding to r4-r6 to generate the feature weighted sum d₂. The present disclosure does not limit how the first processor i1 selects a plurality of multiplication results that meet the SpD value to perform the addition. For example, in the example of FIG. 3, the first processor i1 may randomly select 3 multiplication results, such as the multiplication results corresponding to r1, r3, and r6, to perform the addition to generate the feature weighted sum d₁, and then randomly select 3 multiplication results from the remaining multiplication results, such as the multiplication results corresponding to r2, r4, and r5, to perform the addition to generate the feature weighted sum d₂. It should be noted that plurality of multiplication results being selected each time are not repeated. In other words, if the first processor i1 selects r1, r3, and r6 this time, r1, r3, and r6 that have been selected will not be selected again in subsequent selections. This approach ensures the security of the feature weighted sum d. If SpD does not divide the number of samples into integers, the remaining encoded features that are not selected are discarded.

In embodiment, the sum of the feature weighted sum and the noise is computed according to the following Equation 4:

$\begin{array}{cc}{D}_R=d+\mathrm{FP}\left(\epsilon \right),& \left(\mathrm{Equation}4\right)\end{array}$

• • where d is the feature weighed sum, FP(ε) is the noise, FP(·) is the feature disturbance function, ε is the parameter to determine the level of noise, and D_R is the sum of the feature weighted sum and the noise. In an embodiment, the feature disturbance function is Laplace mechanism or Gaussian mechanism in differential privacy. When the feature disturbance function adopts the Laplace mechanism, the Equation 4 can be written as Equation 5 below:

$\begin{array}{cc}{D}_R={\sum }_{k=1}^{\mathrm{SpD}}{w}_k{y}_k+\mathrm{Laplace}\left(\epsilon \right),& \left(\mathrm{Equation}5\right)\end{array}$

In other embodiments, FP(·) may be any function that introduces feature disturbance. After the first processor i1 completes the computations of Equations 2 to Equation 4, the pair of the sum D_R and the label weighted sum D_y may be outputted as a digest D through the first communication circuit i2.

In an embodiment, one of the following devices may be employed as the first processor i1: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), system-on-a-chip (SoC), and deep learning accelerator.

The first communication circuit i2 is configured to send the sum D_R and the label weighted sum D_y as the digest D to the moderator Mo, and send an update parameter of the client model ⁱ to the moderator Mo. In an embodiment, the update parameter may be, for example, the gradient or weight of the model. The first communication circuit i2 is further configured to receive the general model and the updated general model from the moderator Mo. In an embodiment, the first communication circuit i2 performs the aforementioned transmission and reception tasks through a wired network or a wireless network.

The first storage circuit i3 is configured to store the raw data Rⁱ, the digest D, the general model , and the client model ⁱ. In an embodiment, one of the following devices may be employed as the first storage circuit i3: Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Double Data Rate Synchronuous Dynamic Random Access Memory (DDR SDRAM), flash memory, and hard disk.

The moderator Mo is communicably connected to each of the client devices Ci, Cj. The moderator Mo includes a second processor M1, a second communication circuit M2, and a second storage circuit M3. The second processor M1 is electrically connected to the second communication circuit M2, and the second storage circuit M3 is electrically connected to the second processor M1 and the second communication circuit M2. The hardware implementation of the moderator Mo and its internal components M1, M2, M3 may refer to the client device Ci and its internal component i1, i2, i3, and thus the detail is not repeated here.

The second processor M1 is configured to determine one or more absent client devices and one or more present devices among the plurality of client devices Ci, Cj. In an embodiment, the second processor M1 checks the communication connection between the second communication circuit M2 and each of the client devices Ci and thereby determining whether one or more of all the client devices Ci, Cj is (are) disconnected. The client device Ci keeping the connection is called the present client, while the client device Cj breaking the connection is called the absent client.

The second processor M1 is configured to execute a guidance producer _G, and thereby generating a piece of guidance G according to the digest D of the absent client. In the initial training stage of federated learning, each client device Ci converts the raw data R into the digest D and sends the digest D to the moderator Mo. Therefore, the guidance G recovered from the digest D is equivalent to the representative part of the raw data R, and the guidance G does not include the privacy part of the raw data R. When the moderator Mo updates the general model , the guidance producer _G is trained together with the general model , and the detail is described later. In the embodiment shown in FIG. 1, the guidance producer _G is a software running on the second processor M1, but the present disclosure does not limit the hardware configured to execute the guidance producer _G. The guidance producer G may be stored in the second storage circuit M3, or in an internal memory of the second processor M1. Since the guidance producer _G may generate the guidance G representing the raw data R of the client device Ci, the guidance producer _G should be protected by the moderator Mo from access by unauthorized clients, thereby avoiding potential data leakage or an adversarial attack.

In the initial training stage of federated learning, the second processor is further configured to initialize the general model , and send the general model to each of the client devices Ci through the second communication circuit M2. During the training process of federal learning, if the second processor M1 determines absent client (such as Cj), the second processor M1 generates a replacement model according to the general model , the digest D_R^j of the absent client Cj, and an absent client loss function.

FIG. 4 is an architectural diagram of the guidance producer and the replacement model according to an embodiment of the present disclosure. The replacement model includes a first feature extractor F_R, a second feature extractor F_D and a classifier C. As shown in FIG. 4, the second processor M1 is configured to input the digest D_R^j of the absent client Cj to the guidance producer _G to generate the guidance G^j, input the guidance G^j to the first feature extractor F_R to generate a first feature, input the digest Dg of the absent client Cj to the second feature extractor F_D to generate a second feature, and input a concatenation of the first feature and second feature to the classifier C to generate a predicted result The second processor M1 is further configured to input the predicted result and an actual result D_y^j to the absent client loss function, and adjust a weight of at least one of the first feature extractor F_R, the second feature extractor F_D and the classifier C. In an embodiment, the absent client loss function is shown in the following Equation 6:

$\begin{array}{cc}{ℒ}_{\mathrm{client}}^{\mathrm{avail}}={ℒ}_{\mathrm{ce}}\left({\stackrel{\_}{ℳ}}^j\left({𝒫}_G\left(D_R^j\right),D_R^j\right),D_y^j\right),& \left(\mathrm{Equation}6\right)\end{array}$

• • where _client^absent is the absent loss function, _ce may adopt an appropriate loss function according to the purpose of different models while cross entropy is used as _ce in an embodiment, ^j is the replacement model (assumed that the absent client is the client device Cj), _G is the guidance prodcuer, D_R^j is the digest corresponding to the absent client Cj, _G(D_R^j)=G^j represents the guidance, ^j(_G(D_R^j),D_R^j)= represents the predicted result of the replacement model ^j, D_y^j is the actual result. The condition for the replacement model ^j to complete training is that the output of the absent client loss function _client^absent is smaller than a certain threshold. The general model completing the training is called the replacement model ^j.

Overall, if the client device is not an absent client, the client device may train the client model based on the general model and the raw data. In contrast, if the client device becomes an absent client, the moderator will generate the guidance according to the digest representing the raw data and train the general model on behalf of the absent client based on the digest and the guidance to generate a replacement model. From FIG. 2 and FIG. 4, it can be seen that the client model and the replacement model have the same architecture because both models are trained based on the general model, the difference lies in the different training data.

The second processor M1 is further configured to perform an aggregation to generate an aggregation model according to the general model , the update parameter of the client model ⁱ of the present client Ci and the update parameter of the replacement model ^j of the absent client Cj. In an embodiment, the update parameter of the model may be, for example, gradient or weight. In an embodiment, the aggregation is shown in the following Equation 7:

$\begin{array}{cc}{𝒜}_t={ℳ}_t+{\sum }_i{w}_t^i\nabla {ℳ}_t^i+{\sum }_j{w}_t^j\nabla {\stackrel{\_}{ℳ}}_t^j,& \left(\mathrm{Equation}7\right)\end{array}$

where _t is the aggregation model, t is the general model (t represents the t-th iteration), w_tⁱ is the weight corresponding to the present client Ci, ∇_tⁱ is the update parameter of the client model _tⁱ of the present client Ci, w_t^j is the weight corresponding to the absent client Cj, ∇_t^j, is the update parameter of the replacement model ^j of the absent client Cj.

In an embodiment, the weight w_tⁱ corresponding to the present client Ci and the weight w_t^j corresponding to the absent client Cj satisfy the following Equation 8:

$\begin{array}{cc}{\sum }_i{w}_t^i+{\sum }_j{w}_t^j=1.& \left(\mathrm{Equation}8\right)\end{array}$

In other embodiments, the aggregation may be FedAvg, FedProx, or FedNova, and the present disclosure does not limit thereof.

The second processor M1 is further configured to train the aggregation model _t to update the general model _t according to the moderator loss function. In an embodiment, the moderator loss function is shown in the following Equation 9:

$\begin{array}{cc}{ℒ}_{\mathrm{server}}={ℒ}_{\mathrm{ce}}\left({𝒜}_t\left({𝒫}_G\left(D_R\right),D_R\right),D_y\right),& \left(\mathrm{Equation}9\right)\end{array}$

• • where _server is the moderator loss function, _ce may adopt an appropriate loss function according to the purpose of different models while cross entropy is used as _ce in an embodiment, _t is the aggregation model, _G is the guidance producer, D_R is the sums of all client devices (as previously mentioned, each sum is the addition result of the feature weighted sum of a client device and the noise), D_y is the label weighted sum of all client devices. The condition for the aggregation model _t to complete training is that the output of the moderator loss function _server is smaller than a certain threshold. It should be noted that the output of the moderator loss function _server gradually reduces in the training process, and this process also implements the training of the guidance producer _G.

The second communication circuit M2 is configured to send the general model _t and the digest producer _R to each of the client devices Ci, Cj. In other words, the moderator Mo and each of the client devices Ci, Cj have identical digest producer _R. In addition, in the initial training stage of federated learning, the second processor M1 controls the second communication circuit M2 to send the digest request to each of the client devices Ci, Cj, and then to receive the digest D returned from each of the client devices Ci, Cj.

The second storage circuit M is configured to store digests D of all client devices Ci, Cj, and further store the digest producer _R, the guidance G, the general model _t, and the replacement model _J.

FIG. 5 and FIG. 6 are overview diagrams of the federated learning system of protecting data digest according to an embodiment of the present disclosure. FIG. 5 and FIG. 6 represent two different timings in the training process respectively, and the timing corresponding to FIG. 6 is later than the timing corresponding to FIG. 5. FIG. 5 and FIG. 6 represent two conditions of FedDig training respectively, where FIG. 5 shows that system collects digests when the client devices Ci, Cj are available and FIG. 6 shows that the system uses the guidance G^j to continue training when the client device Ci is absent.

Before the timing corresponding to FIG. 5, the client device Ci has already received the general model from the moderator Mo. At the timing corresponding to FIG. 5, the client devices Ci, Cj exist and perform the training respectively. Taking the client device Ci as an example, the digest producer _R converts the plurality of raw data Rⁱ into the plurality of encoded features rⁱ, mixes these encoded features rⁱ to generate the digest Dⁱ, and sends the digest Dⁱ to the moderator Mo. The client device Ci performs the training according to the raw data Rⁱ, the encoded features rⁱ and the general model , and thereby generating the client model ⁱ. The operation of the client device Cj is identical to that of the client device Ci, and the description is not repeated here.

The moderator Mo receives the digests Dⁱ, D^j from the client devices Ci, Cj and stores thereof. The moderator Mo receives the update parameters of the client models ⁱ,^j from the client devices Ci, Cj, performs the aggregation according to the update parameters of the client models ⁱ,^j, and thereby updating the general model . Finally, the trained general model may be deployed on the device of the consumer U.

At the the timing corresponding to FIG. 6, the client device Ci is the present client. The client device Cj leaves and becomes the absent client. Therefore, the guidance producer _G of the moderator Mo generates the guidance G^j according to the digest D^j corresponding to the absent client Cj. The moderator Mo further generates the replacement model ^j according to the digest D′ corresponding to the absent client Cj and the guidance G^j, performs the aggregation according to the replacement model ^j and client model ⁱ of the present client Ci, and thereby updating the general model .

In this way, regardless of whether the client device Ci exists or not, the training of the federated learning system of protecting data digest proposed by the present disclosure will not be interrupted.

FIG. 7 is a flow chart of the federated learning method of protecting data digest according to an embodiment of the present disclosure and includes steps S1-S7. Step S1 shows “the moderator sends a general model and a digest producer to each client device”, step S2 shows “each client device executes the digest producer to generate encoded features”, step S3 shows “each client device performs a training procedure to generate a client model”, step S4 shows “the moderator determines an absent client and a present client among the plurality of client devices”, step S5 shows “the moderator generates a replacement model according to the digest of the absent client and the general model”, step S6 shows “the moderator performs an aggregation to generate an aggregation model according to the general model, an update parameter of the client model of the present client, and an update parameter of the replacement model of the absent client”, and step S7 shows “the moderator trains the aggregation model to update the general model and sends the updated general model to each client device”.

The training of federated learning includes a plurality of iterations, and steps S3-S7 in FIG. 7 show the detail of one of the iterations. In an embodiment, the method shown in FIG. 7 may be implemented by the system shown in FIG. 1, FIG. 5 and FIG. 6.

In an embodiment, step S1 is performed in the first iteration of federated learning. In step S1, the moderator initializes a general model, and sends the general model to each client device. In addition, the moderator sends the digest producer to each client device to ensure that all client devices have the identical digest producer.

In step S2, each client device inputs the plurality of raw data into the digest producer to generate the plurality of encoded features, and selects some of the plurality of encoded features to mix according to the specified number, and thereby generating the digest to send to the moderator. In an embodiment, step S2 is performed in the first iteration of the federated learning. In another embodiment, step S2 is performed as long as the client device receives the digest request from the moderator.

In step S3, the details of the training procedure may refer to FIG. 8. FIG. 8 is a detailed flow chart of step S3 in FIG. 7 and includes steps S31-36. In step S31, the client device updates the general model to generate the client model according to the plurality of raw data, the plurality of encoded features, a plurality of labels corresponding to the plurality of encoded features, and a present client loss function. Please refer to FIG. 9 for the details of step S31. FIG. 9 is a detailed flow chart of step S31 in FIG. 8 and includes steps S311-S314. Step S311 shows “inputting the raw data to a first feature extractor to generate a first feature”, step S312 shows “inputting the encoded features to a second feature extractor to generate a second feature”, step S313 shows “inputting a concatenation of the first feature and the second feature to a classifier to generate a predicted result”, and step S314 shows “inputting the predicted result and an actual result to a present client loss function, and adjusting a weight of at least one of the first feature extractor, the second feature extractor, and the classifier according to an output of the present client loss function”.

In step S32, the client device determines whether a digest request has been received. Step S33 is performed if the determination is “yes”. Step S35 is performed if the determination is “no”. In step S33, the client device selects at least two encoded features from the plurality of encoded features to compute a feature weighted sum and selects at least two labels from the plurality of labels to compute a label weighted sum. In step S34, the client device computes the sum of feature weighted sum and the noise. In step S35, the client device sends the sum and the label weighted sum as the digest to the moderator. In step S36, the client device sends the update parameter of the client device to the moderator.

In step S4, the moderator detects the connection between itself and each client device, determines the client device that keeps the connection as a present client, and determines the client device that breaks the connection as an absent client.

In step S5, the details of generating the replacement model may refer to FIG. 10. FIG. 10 is a detailed flow chart of a step S5 in FIG. 7 and includes steps S51-S55. Step S51 shows “inputting the digest of the absent client to the guidance producer to generate the guidance”, step S52 shows “inputting the guidance to a first feature extractor to generate a first feature”, step S53 shows “inputting the digest of the absent client to a second feature extractor to generate a second feature”, step S54 shows “inputting a concatenation of the first feature and the second feature to a classifier to generate a predicted result”, and step S55 shows “inputting the predicted result and an actual result to an absent client loss function, and adjusting a weight of at least one of the first feature extractor, the second feature extractor, and the classifier according to an output of the absent client loss function”.

In step S6, the details of generating the aggregation model may refer to FIG. 11. FIG. 11 is a detailed flow chart of a step S6 in FIG. 7 and includes steps S61-S63. Step S61 shows “computing a first weighted sum of an update parameter of the client model of each present client and a first weight”, step S62 shows “computing a second weighted sum of an update parameter of the replacement model of each absent client and a second weight”, and step S63 shows “summing the update parameter of the general model, the first weighted sum, and the second weighted sum to generate an update parameter of the aggregation model”.

In step S7, the details of updating the general model may refer to FIG. 12. FIG. 12 is a detailed flow chart of a step S7 in FIG. 7 and includes steps S71-S73. Step S71 shows “inputting the digest of each client device to the guidance producer to generate the guidance”, step S72 shows “inputting the guidance and the digest of each client device to the aggregation model to generate the predicted result”, and step S73 shows “inputting the predicted result and the actual result to the moderator loss function, adjusting the parameter of the aggregation model according to an output of the moderator loss function, and updating the guidance producer”. After step S73 is completed, the trained aggregation model may be sent to each client device as the updated general model.

The following algorithm is the pseudo code of the federated learning method of protecting data digest according to an embodiment of the present disclosure:

01 Initialize:   and G
02 for Each training iteration t do
03 Moderator pushes server model Mt to all clients
04 (Client side)
05 for available client i = 1, 2, ..., n in parallel do
06 Generate encoded features ri = PR(Ri)
07 Generate ∇Mti with loss clientavail and update Mti
08 Push model gradient ∇Mti to the moderator
09 if t = 0 then
10 Produce digests Di = (DRi, Dyi)
11 Push Di to the moderator
12 end if
13 end for
14 (Moderator side)
15 for absent client j = 1, 2, ..., k in parallel do
16 if the digests Dj exists then
17 Generate replacement model Mtj from Mt
18 Generate ∇Mti with loss clientabsent and update Mtj
19 end if
20 end for
21 Moderator updates t with Mt, ∇Mti, and ∇Mti
22 Moderator updates ∇Mt with loss server
23 Moderator updates Mt+1 = Mt + ∇Mt
24 end for

• • where is the initialized general model, _G is the initialized guidance producer, t is the number of iterations, _t is the general model at the t-th iteration, n is the number of client devices, rⁱ is the encoded feature, P_R is the digest producer, Rⁱ is the raw data, ∇_tⁱ is the update parameter of the client model of the present client Ci, _client^avail is the present client loss function, _tⁱ is the client model of present client Ci, Dⁱ is the digest of the present client Ci, D_R is the sum of feature weighted sum and the noise, D_yⁱ is the label weighted sum, k is the number of absent clients, D^j is the digest of the absent client, _t^j is the replacement model of the absent client Cj, ∇_t^j is the update parameter of the replacement model of the absent client Cj, _client^absent is the absent client loss function, _t is the aggregation model, _server is the moderator loss function, ∇M_t is the parameter of updated general model, and _t+1 is the updated general model sent to the client device at the (t+1)th iteration.

Please refer to FIG. 8 to FIG. 12 and the algorithm above. Line 3 of the algorithm corresponds to step S1, lines 4 to 6 correspond to step S2, lines 7 to 11 correspond to step S3, where line 7 corresponds to step S31, line 8 corresponds to step S36, line 9 corresponds to step 32, line 10 corresponds to step S33 and step S34, line 11 corresponds to step S35, line 14 to 15 corresponds to step S4, line 16 to 18 corresponds to step S5, line 21 corresponds to step S6, and lines 22 to 23 correspond to step S7.

In view of the above, the present disclosure provides a federated learning method of protecting data digest. This is a federated learning framework that can address client absence by synthesizing representative client data at the moderator. The present disclosure proposes a data memorizing mechanism to handle the client's absence effectively. Specifically, the present disclosure handles the following three scenarios: (1) unreliable clients, (2) training after removing clients, and (3) training after adding clients.

The present disclosure deals with potential client absence during FL training is to encode and aggregate information of the raw data and corresponding labels as data digests, and add a mechanism of the feature disturbance into the digest. When clients leave, the moderator may recover information from these digests to generate training guidance that can mitigate the catastrophic forgetting caused by the absent data. Since digests may be shared and stored at the moderator for training use, information that can lead to data privacy infringement should not be recoverable from the digests. To increase privacy protection of the proposed data digest, the present disclosure introduces the sample disturbance by mixing features extracted from the raw data, and add the noise generated based on differential privacy to protect the privacy of the data digest. Furthermore, the present disclosure introduces a trainable guidance producer into the ordinary FL training process, such that the moderator may learn to extract information and generate training guidance from the digests automatically. The digest and guidance proposed by the present disclosure are adaptable to most FL systems.

FIG. 13 is a schematic diagram showing the guidance reconstructed in different ways compared to the raw data. As shown in FIG. 13, R0 represents the samples of the raw data, including 16 handwritten characters. G_mix represents the guidance reconstructed based on the digest of the mixed encoded features only. G1 represents the guidance reconstructed based on the digest of the encoded features mixed with Laplace noise, where the parameter ε controlling the level of noise is set to 0.005. It can be seen from FIG. 13 that visually, it is more difficult to identify corresponding features of G1 in the original samples compared to G_mix.

FIG. 14(a) to FIG. 14(c) show the visual results obtained by changing the parameters of feature disturbance methods, and it is evident from the figures that the visual quality degrades as the parameters are changed. FIG. 14(a) shows the samples of the raw data, FIG. 14(b) shows the result with noise added with & set to 0.005, and FIG. 14(c) shows the result with noise added with ε set to 0.0005. As shown in FIG. 14(a) to FIG. 14(c), a smaller value of ε corresponds to more noise being added. In practice, the setting value of ε can be adaptively determined based on the property of the raw data owned by the client device. In other words, the present disclosure may control different privacy levels to meet the privacy requirements of users by adjusting the setting value of ε.

FIG. 15 demonstrates the testing accuracy of the model with varying levels of noise added to the digest. The testing accuracy is a quantitative indicator for evaluating the performance of machine learning models. As shown in FIG. 15, the testing accuracy is approximately 87% when the feature mixing is not adopted, the noise is added only, and ε is set to 0.005. The testing accuracy is approximately 83% when the feature mixing is not adopted, the noise is added only, and ε is set to 0.0005. The testing accuracy is approximately 77% when the feature mixing is adopted without adding noise. The testing accuracy is approximately 78% when the feature mixing is adopted, the noise is added, and ε is set to 0.005. From FIG. 15, it can be seen that adding noise with ε set to 0.005 under the mechanism of feature mixing does not significantly affect the testing accuracy of the model (increased from 77% to 78%). In addition, the impact of the value of ε on testing accuracy can be seen from the two bar graphs on the left of FIG. 15.

FIG. 16(a) to FIG. 16(d) are performance comparison diagrams of models in different training scenarios.

In the training process of FL, the following four training scenarios are common: (1) a client temporarily leaves during the FL training, (2) a client leaves the training forever, (3) all clients leave the FL training sequentially, and (4) multiple client groups join the FL training in different time slots. FIG. 16(a) to FIG. 16(d) correspond to the above four scenarios respectively and show the accuracy of the general model, where C0, C1, C2, C3 represent different client devices, and the dataset is EMNIST. In the tests of FIG. 16(a) to FIG. 16(d), the client device with largest data amount is asked to leave the FL training to highlight the performance impact. As it can be observed in FIG. 16(a) to FIG. 16(d), none of the common FL algorithms, such as FedAvg, FedNova, FedProx, survive in the target four scenarios on the testing accuracy. On the other hand, the federated learning system and method of protecting data digest achieve a stable testing accuracy on the scenarios. The experiment results show the robustness of the federated learning system and method of protecting data digest.

Claims

1. A federated learning method of protecting data digest comprising: sending a general model to each of a plurality of client devices by a moderator;executing a digest producer by each of the plurality of client devices to generate a plurality of encoded features according to a plurality of raw data;performing a training procedure by each of the plurality of client devices, wherein the training procedure comprises: updating the general model to generate a client model according to the plurality of raw data, the plurality of encoded features, a plurality of labels corresponding to the plurality of encoded features, and a present client loss function;selecting at least two of the plurality of encoded features to compute a feature weighted sum, computing a sum of the feature weighted sum and noise, selecting at least two of the plurality of labels to compute a label weighted sum, and sending the sum and the label weighted sum to the moderator as a digest when receiving a digest request; andsending an update parameter of the client model to the moderator;determining an absent client and a present client among the plurality of client devices by the moderator;generating a replacement model according to the general model, the digest of the absent client and an absent client loss function by the moderator;performing an aggregation to generate an aggregation model according to the update parameter of the client model of the present client and an update parameter of the replacement model of the absent client by the moderator; andtraining the aggregation model to update the general model according to a moderator loss function by the moderator.

2. The federated learning method of protecting data digest of claim 1, further comprising: generating the noise according to a Laplace mechanism or Gaussian mechanism of differential privacy.

3. The federated learning method of protecting data digest of claim 1, wherein the general model comprises a first feature extractor, a second feature extractor and a classifier, and updating the general model to generate the client model according to the plurality of raw data, the plurality of encoded features, the plurality of labels corresponding to the plurality of encoded features, and the present client loss function comprises: inputting the plurality of raw data to the first feature extractor to generate a first feature;inputting the plurality of encoded features to the second feature extractor to generate a second feature;inputting a concatenation of the first feature and the second feature to the classifier to generate a predicted result; andinputting the predicted result and an actual result to the present client loss function, and adjusting a weight of at least one of the first feature extractor, the second feature extractor, and the classifier according to an output of the present client loss function.

4. The federated learning method of protecting data digest of claim 1, wherein the general model comprises a first feature extractor, a second feature extractor and a classifier, and generating the replacement model according to the digest of the absent client and the absent client loss function comprises: inputting the digest of the absent client to a guidance producer to generate a piece of guidance;inputting the piece of guidance to the first feature extractor to generate a first feature;inputting the digest of the absent client to the second feature extractor to generate a second feature;inputting a concatenation of the first feature and the second feature to the classifier to generate a predicted result; andinputting the predicted result and an actual result to the absent client loss function, and adjusting a weight of at least one of the first feature extractor, the second feature extractor, and the classifier according to an output of the absent loss function; wherein the replacement model is the general model with an updated weight.

5. The federated learning method of protecting data digest of claim 1, wherein performing the aggregation to generate the aggregation model according to the update parameter of the client model of the present client and the update parameter of the replacement model of the absent client comprises: computing a first weighted sum of the update parameter of the client model of the present client and a first weight;computing a second weighted sum of the update parameter of the replacement model and a second weight; andsumming a parameter of the general model, the first weighted sum and the second weighted sum to generate a parameter of the aggregation model.

6. The federated learning method of protecting data digest of claim 1, wherein training the aggregation model to update the general model according to the moderator loss function by the moderator comprises: inputting the digest of each of the plurality of client devices to a guidance producer to generate a piece of guidance;inputting the piece of guidance and the digest of each of the plurality of client devices to the aggregation model to generate a predicted result; andinputting the predicted result and an actual result to the moderator loss function, and adjusting a parameter of the aggregation model according to an output of the moderator loss function.

7. A federated learning system of protecting data digest comprises: a plurality of client devices, wherein each of the plurality of client devices comprises: a first processor configured to execute a digest producer to generate a plurality of encoded features according to a plurality of raw data, further configured to update a general model to generate a client model according to the plurality of raw data, the plurality of encoded features, a plurality of labels corresponding to the plurality of encoded features, and a present client loss function, and further configured to select at least two of the plurality of encoded features to compute a feature weighted sum, compute a sum of the feature weighted sum and noise, and select at least two of the plurality of labels to compute a label weighted sum when receives a digest request; anda first communication circuit electrically connected to the first processor and configured to send the sum and the label weighted sum as a digest and send an update parameter of the client model; anda moderator communicably connected to each of the plurality of client devices, wherein the moderator comprises: a second communication circuit configured to send the general model to each of the plurality of client devices; anda second processor electrically connected to the second communication circuit, wherein the second processor is configured to determine an absent client and a present client among the plurality of client devices, generate a replacement model according to the general model, the digest of the absent client and an absent client loss function, perform an aggregation to generate an aggregation model according to the update parameter of the client model of the present client and an update parameter of the replacement model of the absent client, and train the aggregation model to update the general model according to a moderator loss function.

8. The federated learning system of protecting data digest of claim 7, wherein the first processor further generates the noise according to a Laplace mechanism or a Gaussian mechanism of differential privacy.

9. The federated learning system of protecting data digest of claim 7, wherein the general model comprises a first feature extractor, a second feature extractor and a classifier, and the first processor is further configured to: input the plurality of raw data to the first feature extractor to generate a first feature;input the plurality of encoded features to the second feature extractor to generate a second feature;input a concatenation of the first feature and the second feature to the classifier to generate a predicted result; andinput the predicted result and an actual result to the present client loss function, and adjusting a weight of at least one of the first feature extractor, the second feature extractor, and the classifier according to an output of the present client loss function.

10. The federated learning system of protecting data digest of claim 7, wherein the general model comprises a first feature extractor, a second feature extractor and a classifier, and the second processor is further configured to: input the digest of the absent client to a guidance producer to generate a piece of guidance;input the piece of guidance to the first feature extractor to generate a first feature;input the digest of the absent client to the second feature extractor to generate a second feature;input a concatenation of the first feature and the second feature to the classifier to generate a predicted result; andinput the predicted result and an actual result to the absent client loss function, and adjusting a weight of at least one of the first feature extractor, the second feature extractor, and the classifier according to an output of the absent loss function; wherein the replacement model is the general model with an updated weight.