CIF: Small: Communication-efficient and robust learning from distributed data

在连接设备或数据中心的分布式网络上分配机器学习工作流的趋势越来越大。对于支持大数据应用的分布式数据网络，在计算节点之间移动数据或模型参数的通信成本已经成为所有分布式机器学习算法的共同瓶颈。该项目为分布式学习开发了高效通信和强大的技术，特别是在缺乏中央协调的分散网络中。关键思想是加强通信审查，其中分布式节点基于对本地信息变化重要性的自主评估，不频繁地传输其本地更新。这项研究的结果有望使大量资源受限的分布式学习应用受益，例如关键基础设施的结构监测、位置感知服务、物联网和移动医疗保健。该项目的目标是开发高效通信和健壮的分布式随机优化方法，以便在大数据计算中从本地存储的私有数据中学习。为了有效地减少分布节点之间的消息移动，将通信审查框架引入到方差减少随机优化技术的设计中，同时即使在没有任何中心协调或同步的情况下，也能全局优化具有可证明收敛性的共享学习模型。此外，开发了分布式健壮聚合技术，以对抗恶意攻击，故障节点和传输链路故障的影响，并增加了数据隐私保护。发展的通信审查和鲁棒聚合的理论和机制是分布式节点协作评估计算的信息性和联合评估鲁棒统计的关键思想，即使在没有中心协调的情况下也没有数据共享。进行了严格的分析，以描绘收敛条件，收敛速度，以及效率和鲁棒性之间的权衡。这些进步为推动实际分布式机器学习系统在广泛应用中的成功实施提供了重要的工具。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

There is an increasing trend of allocating machine learning workflows over a distributed network of connected devices or data centers. For distributed data networks supporting big data applications, the communication cost of moving either data or model parameters among computing nodes has become a common bottleneck of all distributed machine learning algorithms. This project develops communication-efficient and robust techniques for distributed learning, particularly for decentralized networks in the absence of central coordination. The key idea is to enforce communication censoring, in which distributed nodes transmit their local updates infrequently based on autonomous assessment of the significance of local information changes. The outcomes of this research are expected to benefit a plethora of resource-constrained distributed learning applications, such as structural monitoring for critical infrastructure, location-aware services, Internet of Things, and mobile healthcare. The goal of this project is to develop communication-efficient and robust approaches to distributed stochastic optimization, for learning from locally stored private data in big data computing. A communication-censoring framework is introduced into the design of variance-reduced stochastic optimization techniques in order to effectively reduce message movement among distributed nodes, while globally optimizing a shared learning model with provable convergence, even in the absence of any central coordination or synchronism. Further, distributed robust aggregation techniques are developed to combat the impacts of malicious attacks, malfunctional nodes and transmission link failure, with added protection of data privacy. The developed theory and mechanisms on communication censoring and robust aggregation feature in key ideas for distributed nodes to collaboratively evaluate the informativeness of computing and jointly assess robust statistics without data sharing, even in the absence of central coordination. Rigorous analyses are conducted to delineate the convergence conditions, convergence rates, and tradeoff between efficiency and robustness. Such advances offer vital tools to propel the successful implementation of practical distributed machine learning systems in broad applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.