FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
复制标题
DOI：
10.18653/v1/2022.findings-naacl.13
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Bill Yuchen Lin;Chaoyang He;ZiHang Zeng;Hulin Wang;Yufen Huang;M. Soltanolkotabi;Xiang Ren;S. Avestimehr
中科院分区：
文献类型：
--
作者：
Bill Yuchen Lin;Chaoyang He;ZiHang Zeng;Hulin Wang;Yufen Huang;M. Soltanolkotabi;Xiang Ren;S. Avestimehr
对数据隐私和稀疏性的关注和监管日益增加，需要研究用于自然语言处理（NLP）任务的隐私保护、分散学习方法。联邦学习（FL）为大量客户端（例如，个人设备或组织）提供了有前途的方法来协作学习共享的全局模型，以使所有客户端受益，同时允许用户将其数据保留在本地。尽管有兴趣研究用于NLP任务的FL方法，但文献中缺乏系统的比较和分析。在这里，我们提出了FedNLP，这是一个基准框架，用于评估四种不同任务公式上的联邦学习方法：文本分类、序列标记、问答和seq2seq。我们在各种非iid分区策略下提出了基于transformer的语言模型（例如BERT， BART）和FL方法（例如FedAvg， FedOPT等）之间的通用接口。我们对FedNLP的大量实验提供了FL方法之间的经验比较，并帮助我们更好地理解这个方向的内在挑战。综合分析指出了有趣和令人兴奋的未来研究，旨在开发用于NLP任务的FL方法。
Increasing concerns and regulations about data privacy and sparsity necessitate the study of privacy-preserving, decentralized learning methods for natural language processing (NLP) tasks. Federated learning (FL) provides promising approaches for a large number of clients (e.g., personal devices or organizations) to collaboratively learn a shared global model to benefit all clients while allowing users to keep their data locally. Despite interest in studying FL methods for NLP tasks, a systematic comparison and analysis is lacking in the literature. Herein, we present the FedNLP, a benchmarking framework for evaluating federated learning methods on four different task formulations: text classification, sequence tagging, question answering, and seq2seq. We propose a universal interface between Transformer-based language models (e.g., BERT, BART) and FL methods (e.g., FedAvg, FedOPT, etc.) under various non-IID partitioning strategies. Our extensive experiments with FedNLP provide empirical comparisons between FL methods and helps us better understand the inherent challenges of this direction. The comprehensive analysis points to intriguing and exciting future research aimed at developing FL methods for NLP tasks.