SPACE: fully decentralised distributed learning for tradeoff of privacy, accuracy, communication complexity, and efficiency

机器学习应用的训练数据可以从地理上不同的地方收集，但将分布式数据传输到中央服务器进行训练在法律上或实践上都是不可能的。这导致了机器学习中一个快速增长的领域，即联合学习。然而，人们发现，原始形式的联合学习，其中服务器迭代地聚集从分布式用户接收的局部梯度，可能会遭受隐私泄露，因为攻击者可以从局部梯度推断敏感信息。从那时起，已经提出了许多改进来缓解这种隐私泄露，但它们往往以牺牲其他关键特性(如准确性和通信复杂性)来实现更好的隐私。在本文中，我们提出了一种新的解决方案，该方案基于完全分散的分布式学习。我们的解决方案能够优化多个属性，并在多个属性之间实现平衡，包括隐私保护、准确性、通信复杂性、效率和对用户故障的容忍度。我们的解决方案首先根据所需的属性综合用户之间的通信拓扑，然后在服务器不参与计算的情况下应用完全分散的分布式学习。在分散式学习中，局部梯度的聚集被归结为用户之间的分布式共识。最后，在添加了差分隐私噪声后，将用户的协议值发送到服务器。我们正在对美国/英国隐私增强技术挑战赛中的两个用例进行实验，以验证我们的解决方案。

Training data of a machine learning application may be collected from geographically different places, but transferring distributed data to a central server for training can be legally or practically impossible. This has led to a fast-growing area in machine learning, i.e., federated learning. However, it was discovered that the federated learning in its original form, where a server iteratively aggregates local gradients received from the distributed users, may suffer from privacy leakage as an attacker can infer sensitive information from the local gradients. Since then, many improvements have been presented to mitigate this privacy leakage, but they often achieve better privacy with the compromise of other critical properties such as the accuracy and communication complexity. In this paper, we propose a novel solution that is based on a fully decentralised distributed learning. Our solution enables the optimisation over, and achieve a balance between, multiple properties, including privacy preservation, accuracy, communication complexity, efficiency, and tolerance to user failures. Our solution proceeds by first synthesising a communication topology between users according to the required properties and then applying a fully decentralised distributed learning where the server is not involved in the computation. In the decentralised learning, the aggregation of local gradients is reduced to a distributed consensus between users. Finally, the agreed value of the users is sent to the server after added a differential privacy noise. We are conducting experiments on both use cases in the US/UK privacy enhancing technologies challenge to validate our solution.