SPACE: fully decentralised distributed learning for tradeoff of privacy, accuracy, communication complexity, and efficiency

点击翻译按钮获取中文摘要

Training data of a machine learning application may be collected from geographically different places, but transferring distributed data to a central server for training can be legally or practically impossible. This has led to a fast-growing area in machine learning, i.e., federated learning. However, it was discovered that the federated learning in its original form, where a server iteratively aggregates local gradients received from the distributed users, may suffer from privacy leakage as an attacker can infer sensitive information from the local gradients. Since then, many improvements have been presented to mitigate this privacy leakage, but they often achieve better privacy with the compromise of other critical properties such as the accuracy and communication complexity. In this paper, we propose a novel solution that is based on a fully decentralised distributed learning. Our solution enables the optimisation over, and achieve a balance between, multiple properties, including privacy preservation, accuracy, communication complexity, efficiency, and tolerance to user failures. Our solution proceeds by first synthesising a communication topology between users according to the required properties and then applying a fully decentralised distributed learning where the server is not involved in the computation. In the decentralised learning, the aggregation of local gradients is reduced to a distributed consensus between users. Finally, the agreed value of the users is sent to the server after added a differential privacy noise. We are conducting experiments on both use cases in the US/UK privacy enhancing technologies challenge to validate our solution.