SPACE: fully decentralised distributed learning for tradeoff of privacy, accuracy, communication complexity, and efficiency

机器学习应用程序的训练数据可能从不同的地理位置收集，但将分布式数据传输到中央服务器进行训练可能在法律上或实际上是不可能的。这导致了机器学习中一个快速增长的领域，即联邦学习。然而，人们发现，在原始形式的联邦学习中，服务器迭代地聚合从分布式用户接收到的本地梯度，由于攻击者可以从本地梯度推断出敏感信息，因此可能会导致隐私泄露。从那时起，已经提出了许多改进来减轻这种隐私泄露，但它们通常通过牺牲其他关键属性（如准确性和通信复杂性）来实现更好的隐私。在本文中，我们提出了一种基于完全分散的分布式学习的新解决方案。我们的解决方案能够优化多个属性，并在多个属性之间实现平衡，包括隐私保护、准确性、通信复杂性、效率和对用户故障的容忍度。我们的解决方案首先根据所需的属性合成用户之间的通信拓扑，然后应用完全分散的分布式学习，其中服务器不参与计算。在去中心化学习中，局部梯度的聚合被简化为用户之间的分布式共识。最后，在添加差分隐私噪声后，将用户的商定值发送给服务器。我们正在对美国/英国隐私增强技术挑战中的两个用例进行实验，以验证我们的解决方案。

Training data of a machine learning application may be collected from geographically different places, but transferring distributed data to a central server for training can be legally or practically impossible. This has led to a fast-growing area in machine learning, i.e., federated learning. However, it was discovered that the federated learning in its original form, where a server iteratively aggregates local gradients received from the distributed users, may suffer from privacy leakage as an attacker can infer sensitive information from the local gradients. Since then, many improvements have been presented to mitigate this privacy leakage, but they often achieve better privacy with the compromise of other critical properties such as the accuracy and communication complexity. In this paper, we propose a novel solution that is based on a fully decentralised distributed learning. Our solution enables the optimisation over, and achieve a balance between, multiple properties, including privacy preservation, accuracy, communication complexity, efficiency, and tolerance to user failures. Our solution proceeds by first synthesising a communication topology between users according to the required properties and then applying a fully decentralised distributed learning where the server is not involved in the computation. In the decentralised learning, the aggregation of local gradients is reduced to a distributed consensus between users. Finally, the agreed value of the users is sent to the server after added a differential privacy noise. We are conducting experiments on both use cases in the US/UK privacy enhancing technologies challenge to validate our solution.