Objective The current development of deep learning has caused significant changes in numerous research fields and has had profound impacts on every aspect of societal and industrial sectors,including computer vision,natural lan-guage processing,multi-modal learning,and medical analysis.The success of deep learning heavily relies on large-scale data.However,the public and scientific communities have become increasingly aware of the need for data privacy.In the real world,data are commonly distributed among different entities such as edge devices and companies.With the increas-ing emphasis on data sensitivity,strict legislation has been proposed to govern data collection and utilization.Thus,the tra-ditional centralized training model,which requires data aggregation,is unusable in the practical setting.In response to such real-world challenges,federated learning(FL)has emerged as a popular research field because it can train a global model for different participants without centralizing data owned by the distributed parties.FL is a privacy-preserving multi-party collaboration model that adheres to privacy protocols without data leakage.Typically,FL requires clients to share a global model architecture for the central server to aggregate parameters from participants and then redistributes the global model(averaged parameters).However,this prerequisite largely restricts the flexibility of the client model architecture.In recent years,the concept of objective model heterogeneous FL has garnered substantial attention because it allows par-ticipants to independently design unique models in FL without compromising privacy.Specifically,participants may need to design special model architecture to ease the communication burden or refuse to share the same architecture due to intel-lectual property concerns.However,existing methods often rely on publicly shared related data or a global model for com-munication,limiting their applicability.In addition,FL is proposed to handle privacy concerns in the distributed learning environment.A pioneering FL method trains a global model by aggregating local model parameters.However,its perfor-mance is impeded due to decentralized data,which results in non-i.i.d distribution(called data heterogeneity).Each par-ticipant optimizes toward the local empirical risk minimum,which is inconsistent with the global direction.Therefore,the average global model has a slow convergence speed and achieves limited performance improvement.Method Model hetero-geneity largely impedes the local model section flexibility,and data heterogeneity hinders federated performance.To address model and data heterogeneity,this paper introduces a groundbreaking approach called adaptive heterogeneous fed-erated(AHF)learning,which employs a unique strategy by utilizing a randomly generated input signal,such as random noise and public unrelated samples,to facilitate direct communication among heterogeneous model architectures.This task is achieved by aligning the output logit distributions,fostering collaborative knowledge sharing among participants.The pri-mary advantage of AHF is its ability to address model heterogeneity without depending on additional related data collection or shared model design.To further enhance AHF's effectiveness in handling data heterogeneity,the paper proposes adap-tive weight updating on both model and sample levels,which enables AHF participants to acquire rich and diverse knowl-edge by leveraging dissimilarities in model output on unrelated data while emphasizing the importance of meaningful samples.Result Empirical validation of the proposed AHF method is conducted through a meticulous series of extensive empirical experiments.Random noise inputs are employed in the context of two distinct federated learning tasks:Digits and Office-Caltech scenarios.Specifically,our solution presents the stable generalization performance on the more chal-lenging scenario,Office-Caltech.Notably,when a larger domain gap exists among private data,AHF achieves higher over-all generalization performance on these different unrelated data samples and obtains stable improvements on most unseen private data.By contrast,competing methods achieve limited generalization performance in the Office-Caltech scenario.The empirical findings validate our solution's ability,showcasing a marked improvement in within-domain accuracy and demonstrating superior cross-domain generalization performance compared with existing methodologies.Conclusion In summary,the AHF learning method,as extensively examined in this thorough investigation,not only presents a straightfor-ward yet remarkably efficient foundation for future progress in the domain of federated learning but also emerges as a trans-formative paradigm in comprehensively addressing model and data heterogeneity.AHF not only lays the groundwork for more resilient and adaptable FL models but also serves as a guide for the transformation of collaborative knowledge sharing in the upcoming era of machine learning.Studying AHF is more than an exploration of an innovative FL methodology;it provides numerous opportunities that arise given the complexities of model and data heterogeneity in the development of machine learning models.
federated learning(FL)model heterogeneitydata heterogeneityrandom noiseheterogeneous federal learning
NETL
NSTL
万方数据
