CIF: Small: Communication-efficient and robust learning from distributed data

在互联设备或数据中心的分布式网络上分配机器学习工作流的趋势越来越大。对于支持大数据应用的分布式数据网络，在计算节点之间传输数据或模型参数的通信开销已经成为所有分布式机器学习算法的共同瓶颈。该项目为分布式学习开发了通信高效和可靠的技术，特别是在缺乏中央协调的分散网络中。其核心思想是加强通信审查，其中分布式节点基于对本地信息变化的重要性的自主评估而不频繁地发送其本地更新。这项研究的结果预计将使大量资源受限的分布式学习应用受益，例如关键基础设施、位置感知服务、物联网和移动医疗的结构监控。这个项目的目标是开发通信高效和健壮的分布式随机优化方法，以便在大数据计算中从本地存储的私有数据中学习。在方差减少随机优化技术的设计中引入了通信审查框架，以有效地减少分布式节点之间的消息移动，同时全局优化共享学习模型，即使在没有任何中央协调或同步的情况下也具有可证明的收敛。此外，开发了分布式稳健聚合技术来对抗恶意攻击、故障节点和传输链路故障的影响，并增加了对数据隐私的保护。发展的通信审查和稳健聚集的理论和机制是分布式节点在没有数据共享的情况下，即使在没有中央协调的情况下，也可以协作地评估计算的信息量和联合评估稳健统计的关键思想。严格的分析描述了收敛条件、收敛速度以及在效率和稳健性之间的权衡。这些进展为推动实用的分布式机器学习系统在广泛的应用中的成功实施提供了至关重要的工具。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

There is an increasing trend of allocating machine learning workflows over a distributed network of connected devices or data centers. For distributed data networks supporting big data applications, the communication cost of moving either data or model parameters among computing nodes has become a common bottleneck of all distributed machine learning algorithms. This project develops communication-efficient and robust techniques for distributed learning, particularly for decentralized networks in the absence of central coordination. The key idea is to enforce communication censoring, in which distributed nodes transmit their local updates infrequently based on autonomous assessment of the significance of local information changes. The outcomes of this research are expected to benefit a plethora of resource-constrained distributed learning applications, such as structural monitoring for critical infrastructure, location-aware services, Internet of Things, and mobile healthcare. The goal of this project is to develop communication-efficient and robust approaches to distributed stochastic optimization, for learning from locally stored private data in big data computing. A communication-censoring framework is introduced into the design of variance-reduced stochastic optimization techniques in order to effectively reduce message movement among distributed nodes, while globally optimizing a shared learning model with provable convergence, even in the absence of any central coordination or synchronism. Further, distributed robust aggregation techniques are developed to combat the impacts of malicious attacks, malfunctional nodes and transmission link failure, with added protection of data privacy. The developed theory and mechanisms on communication censoring and robust aggregation feature in key ideas for distributed nodes to collaboratively evaluate the informativeness of computing and jointly assess robust statistics without data sharing, even in the absence of central coordination. Rigorous analyses are conducted to delineate the convergence conditions, convergence rates, and tradeoff between efficiency and robustness. Such advances offer vital tools to propel the successful implementation of practical distributed machine learning systems in broad applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.