基于梯度选择的轻量化差分隐私保护联邦学习
Lightweight Differential Privacy Federated Learning Based on Gradient Dropout
王周生 1杨庚 2戴华2
作者信息
- 1. 南京邮电大学计算机学院 南京 210023
- 2. 南京邮电大学计算机学院 南京 210023;江苏省大数据安全与智能处理重点实验室 南京 210023
- 折叠
摘要
为了应对机器学习过程中可能出现的用户隐私问题,联邦学习作为首个无需用户上传真实数据、仅上传模型更新的协作式在线学习解决方案,已经受到人们的广泛关注与研究.然而,它要求用户在本地训练且上传的模型更新中仍可能包含敏感信息,从而带来了新的隐私保护问题.与此同时,必须在用户本地进行完整训练的特点也使得联邦学习过程中的运算与通信开销问题成为一项挑战,亟需人们建立一种轻量化的联邦学习架构体系.出于进一步的隐私需求考虑,文中使用了带有差分隐私机制的联邦学习框架.另外,首次提出了基于Fisher信息矩阵的Dropout机制——FisherDropout,用于对联邦学习过程中在客户端训练产生梯度更新的每个维度进行优化选择,从而极大地节约运算成本、通信成本以及隐私预算,建立了一种兼具隐私性与轻量化优势的联邦学习框架.在真实世界数据集上的大量实验验证了该方案的有效性.实验结果表明,相比其他联邦学习框架,FisherDropout机制在最好的情况下可以节约76.8％～83.6％的通信开销以及23.0％～26.2％的运算开销,在差分隐私保护中隐私性与可用性的均衡方面同样具有突出优势.
Abstract
To address the privacy issues in the traditional machine learning,federated learning has received widespread attention and research as the first collaborative online learning solution,that does not require users to upload real data but only model up-dates.However,it requires users to train locally and upload model updates that may still contain sensitive information,which rai-ses new privacy concerns.At the same time,the fact that the complete training must be performed locally by the user makes the computational and communication overheads particularly critical.So,there is also an urgent need for a lightweight federated lear-ning architecture.In this paper,a federated learning framework with differential privacy mechanism is used,for further privacy re-quirements.In addition,a Fisher information matrix-based Dropout mechanism,FisherDropout,is proposed for the first time for optimal selection of each dimension in the gradients updated by client-side.This mechanism greatly saves computing cost,commu-nication cost,and privacy budget,and establishes a federated learning framework with both privacy and lightweight advantages.Extensive experiments on real-world datasets demonstrate the effectiveness of the scheme.Experimental results show that the FisherDropout mechanism can save 76.8％～83.6％of communication overhead and 23.0％～26.2％of computational overhead in the best case compared with other federated learning frameworks,and also has outstanding advantages in balancing privacy and usability in differential privacy.
关键词
联邦学习/差分隐私/Fisher信息矩阵/Dropout机制/轻量化Key words
Federated learning/Differential privacy/Fisher information matrix/Dropout/Lightweight引用本文复制引用
基金项目
国家自然科学基金面上项目(61872197)
国家自然科学基金面上项目(61972209)
国家自然科学基金面上项目(62372244)
江苏省研究生科研与实践创新计划项目(KYCX21_0791)
出版年
2024
NETL
NSTL
万方数据