Projected Federated Averaging with Heterogeneous Differential Privacy
Projected Federated Averaging with Heterogeneous Differential Privacy
复制标题
DOI：
10.14778/3503585.3503592
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Junxu Liu;Jian Lou;Li Xiong;Jinfei Liu;Xiaofeng Meng
中科院分区：
文献类型：
--
作者：
Junxu Liu;Jian Lou;Li Xiong;Jinfei Liu;Xiaofeng Meng
联邦学习（FL）是一个很有前途的框架，多个客户可以在不直接共享数据的情况下学习联合模型。除了联合模型的高实用性之外，严格的数据隐私保护和通信效率是重要的设计目标。许多现有的努力实现严格的隐私，通过确保中间模型参数的差分隐私，然而，他们假设一个统一的隐私参数的所有客户端。实际上，不同的客户端可能由于不同的策略或偏好而具有不同的隐私要求。在本文中，我们专注于明确建模和利用不同客户端的异构隐私要求，并研究如何优化效用的联合模型，同时最大限度地减少通信成本。由于不同的隐私扰动影响模型效用，一个自然的想法是更好地利用具有较高隐私预算的客户端（称为“公共”客户端，相反称为“私人”客户端）提交的信息。挑战在于如何在不偏向联合模型的情况下使用这些信息。我们提出了投影F-平均（PFA），它提取了“公共”客户端提交的模型更新的顶部奇异子空间，并利用它们来投影“私人”客户端的模型更新，然后再聚合它们。然后，我们提出了通信效率的PFA+，它允许“私人”客户端上传预测的模型更新，而不是原来的。我们的实验验证了这两种算法的效用提升相比，基线方法，PFA+实现了99%以上的上行链路通信减少“私人”客户端。
Federated Learning (FL) is a promising framework for multiple clients to learn a joint model without directly sharing the data. In addition to high utility of the joint model, rigorous privacy protection of the data and communication efficiency are important design goals. Many existing efforts achieve rigorous privacy by ensuring differential privacy for intermediate model parameters, however, they assume a uniform privacy parameter for all the clients. In practice, different clients may have different privacy requirements due to varying policies or preferences. In this paper, we focus on explicitly modeling and leveraging the heterogeneous privacy requirements of different clients and study how to optimize utility for the joint model while minimizing communication cost. As differentially private perturbations affect the model utility, a natural idea is to make better use of information submitted by the clients with higher privacy budgets (referred to as "public" clients, and the opposite as "private" clients). The challenge is how to use such information without biasing the joint model. We propose P rojected F ederated A veraging (PFA), which extracts the top singular subspace of the model updates submitted by "public" clients and utilizes them to project the model updates of "private" clients before aggregating them. We then propose communication-efficient PFA+, which allows "private" clients to upload projected model updates instead of original ones. Our experiments verify the utility boost of both algorithms compared to the baseline methods, whereby PFA+ achieves over 99% uplink communication reduction for "private" clients.