Gradient sparsification for efficient wireless federated learning with differential privacy

Kang WEI ¹Jun LI ¹Chuan MA ²Ming DING ³Feng SHU ⁴Haitao ZHAO ⁵Wen CHEN ⁶Hongbo ZHU⁵

扫码查看

作者信息

1. School of Electronic and Optical Engineering,Nanjing University of Science and Technology,Nanjing 210096,China
2. Zhejiang Lab,Hangzhou 311121,China;Key Laboratory of Computer Network and Information Integration(Southeast University),Ministry of Education,Nanjing 211189,China
3. Data61,Commonwealth Scientific and Industrial Research Organisation,Sydney 2015,Australia
4. School of Information and Communication Engineering,Hainan University,Haikou 570228,China;School of Electronic and Optical Engineering,Nanjing University of Science and Technology,Nanjing 210096,China
5. School of Communications and Information Engineering,Nanjing University of Posts and Telecommunications,Nanjing 210003,China
6. School of Electronic Information and Electrical Engineering,Shanghai Jiao Tong University,Shanghai 200240,China
折叠

Abstract

Federated learning(FL)enables distributed clients to collaboratively train a machine learning model without sharing raw data with each other.However,it suffers from the leakage of private informa-tion from uploading models.In addition,as the model size grows,the training latency increases due to the limited transmission bandwidth and model performance degradation while using differential privacy(DP)protection.In this paper,we propose a gradient sparsification empowered FL framework with DP over wire-less channels,to improve training efficiency without sacrificing convergence performance.Specifically,we first design a random sparsification algorithm to retain a fraction of the gradient elements in each client's local model,thereby mitigating the performance degradation induced by DP and reducing the number of transmission parameters over wireless channels.Then,we analyze the convergence bound of the proposed algorithm,by modeling a non-convex FL problem.Next,we formulate a time-sequential stochastic optimiza-tion problem for minimizing the developed convergence bound,under the constraints of transmit power,the average transmitting delay,as well as the client's DP requirement.Utilizing the Lyapunov drift-plus-penalty framework,we develop an analytical solution to the optimization problem.Extensive experiments have been implemented on three real-life datasets to demonstrate the effectiveness of our proposed algorithm.We show that our proposed algorithms can fully exploit the interworking between communication and computation to outperform the baselines,i.e.,random scheduling,round robin,and delay-minimization algorithms.

Key words

federated learning/differential privacy/gradient sparsification/Lyapunov drift/convergence analysis

引用本文复制引用

基金项目

国家自然科学基金(62071296)

国家自然科学基金(62002170)

国家自然科学基金(62071234)

国家自然科学基金(U22A2002)

国家重点研发计划(2020YFB1807700)

中央高校基本科研业务费专项(30921013104)

Key Technologies Research and Development Program of Jiangsu(Prospective and Key Technologies for Industry)(BE2023022)

Key Technologies Research and Development Program of Jiangsu(Prospective and Key Technologies for Industry)(BE2023022-2)

Future Network Grant of Provincial Education Board in Jiangsu()

Major Science and Technology Plan of Hainan Province(ZDKJ2021022)

Scientific Research Fund Project of Hainan University(KYQDZR-21008)

Youth Foundation Project of Zhejiang Lab(K2023PD0AA01)

Collaborative Innovation Center of Information Technology,Hainan University(XTCX2022XXC07)

Sciences and Technology Commission of Shanghai Municipality(22JC1404000)

Sciences and Technology Commission of Shanghai Municipality(20JC1416502)

Sciences and Technology Commission of Shanghai Municipality(PKX2021-D02)

出版年

2024

中国科学:信息科学(英文版)

中国科学院

中国科学:信息科学(英文版)

CSTPCDEI

影响因子：0.715

ISSN：1674-733X

参考文献量38

段落导航