Joint dynamic correction algorithms for local and global drifts in federated learning
Objective Federated learning enables multiple parties to collaboratively train a machine learning model without communicating their local data.In practical applications,the data between nodes usually follow a non-independent identi-cal distribution(non-IID).In the local update,each client model will be optimized toward its local optima(i.e.,fitting its individual feature distribution)instead of the global optimal objective and raises a client update drift.Meanwhile,in global updates that aggregate these diverged local models,the server model is further distracted by the set of mismatching local optima,which subsequently leads to a global drift at the server model.To solve the problems of slow global conver-gence and increasing number of training communication rounds caused by non-IID data,this paper proposes a joint dynamic correction federated learning algorithm(FedJDC)that is optimized from the client and server.Method To reduce the influence of non-IID on federated learning,this paper carries out a joint optimization from the two aspects of local model update and global model update and proposes the FedJDC algorithm.This paper then uses the cosine similarity between the local and global update directions to measure the offset of each participating client.Afterward,given that each client has a different degree of non-IID,if the degree of the model offset is only determined by the cosine similarity calculated in this round,then the model update may become unstable.Therefore,the FedJDC algorithm defines the cumulative offset and introduces the attenuation coefficient p.In calculating the cumulative offset of the model,the current and historical cumula-tive offsets are taken into account.In addition,by changing p to reduce the proportion of the cumulative offset of the cur-rent round,the influence of the offset of the current round on the final result can be reduced.This paper also proposes a strategy for dynamically adjusting the constraint terms for local model update offset.Specifically,the constraint terms of the local loss function are dynamically adjusted according to the calculated cumulative offset of the local model,and the algorithm is automatically adapted to various non-IID settings without a careful selection of hyperparameters,thus improv-ing the flexibility of the algorithm.To dynamically change the weight of global model aggregation in each round and effec-tively improve the convergence speed and model accuracy,this paper also designs a dynamic weighted aggregation strategy that takes the accumulated offset uploaded by all clients as the weight of global model aggregation in each round of commu-nication.Result The proposed method is tested on three dataset using different deep learning models.LeNeT5,the VGG16 network model,and the ResNetl8 network model are used for training in the MNIST,FMNIST,and CIFAR10 datasets,respectively.Four experiments are designed to prove the effectiveness of the proposed algorithm.To verify the accuracy of FedJDC at different degrees of non-IID,the hyperparameter fi of the Dirichlet distribution is varied,and the performance of different algorithms is compared.Experimental results show that FedJDC can improve the model accuracy by 5.48％,1.62％,2.10％,and 2.28％on average compared with FedAvg,FedProx,FedAdp,and FedLAW,respectively.To evaluate the communication efficiency of FedJDC,the number of communication rounds is counted as FedJDC reaches a target accuracy,and this number is compared with that obtained by other algorithms.Experimental results show that under different degrees of non-IID,FedJDC can reduce communication rounds by 62.29％,20.90％,24.93％,and 20.47％on average compared with FedAvg,FedProx,FedAdp,and FedLAW,respectively.This paper also investigates the effect of the number of local epochs on the accuracy of the final model.Experimental results show that FedJDC outperforms the other four methods under different epochs in terms of final model accuracy.FedJDC also demonstrates better robustness against the larger offset caused by more local update epochs.Ablation experiments also show that each optimization method performs well on all datasets,and FedJDC combines these two strategies to achieve the global optimal performance.Conclusion This paper optimizes the local and global model offsets from two aspects and proposes a joint dynamic correc-tion algorithm for these offsets in federated learning.The cumulative offset is defined,and the attenuation coefficient is introduced into the calculation of the cumulative offset.Considering the historical and current offset information,the size of the cumulative offset is dynamically adjusted to ensure the stability of the training parameter update.The dynamic con-straint strategy takes the cumulative offset calculated by the client in each round as the constraint parameter of the client model.The dynamic weighted aggregation strategy changes the weight of each local model during the global model aggrega-tion based on the cumulative offset of each participating client so as to dynamically update the global model in each round.The combination of the two optimization strategies has achieved good results,effectively alleviated the performance degrada-tion of the federated learning model caused by non-IID data,and provided a good foundation for the further implementation of federated learning in this field.
NETL
NSTL
万方数据
