MLWiNS: Optimization and Coding Theory for Fast and Robust Wireless Distributed Learning

无线分布式学习系统可以实现各种新的应用，包括工业自动化、语义学习、自主驾驶、医疗保健应用等。无线分布式学习在带来新机遇的同时，也面临着两大挑战，严重限制了其效率、可靠性和可扩展性：(1)网络异构性，这是由于边缘设备的计算能力不同造成的。这一挑战也称为Sttggler瓶颈，由于计算节点比其他节点慢得多，会导致巨大的延迟和故障；以及(2)通信瓶颈，这是由于必须在网络中移动大量原始或已处理的数据。为了解决这些瓶颈，本项目从编码理论和优化理论出发，开发了具有强大理论保证和经验性能的分布式学习算法。无线分布式学习系统是通过在许多无线边缘节点上向外扩展计算来驱动的。然而，出现了两个主要的系统瓶颈：(1)由于等待最慢的节点完成其任务的延迟而导致的Stggler延迟瓶颈；(2)由于必须在节点之间移动的大量数据而导致的数据洗牌瓶颈。此外，还存在共享本地敏感数据的隐私问题，以及易受敌意攻击的问题。这项提议旨在从编码理论和优化理论中开发新的技术来解决上述瓶颈和问题。该项目开发了用于稳健梯度聚合的新的“编码计算”算法，以及用于分布式学习的新的优化算法。然后在两种网络环境中使用这些算法来开发通信高效、抗掉队和健壮的分布式学习框架：(I)将学习任务分配给网络的多个边缘节点的协作环境。在此设置中，可以对数据点进行编码并将其卸载到边缘节点，以针对系统瓶颈提供恢复能力；(Ii)联合设置，其中数据点在边缘设备本地收集，并且由于隐私问题必须保留在本地。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，认为值得支持。

Wireless distributed learning systems can enable a variety of new applications including industrial automation, semantic learning, autonomous driving, health-care applications, etc. While wireless distributed learning brings about new opportunities, it faces two major challenges that severely limit its efficiency, reliability, and scalability: (1) Network heterogeneity, which is due to varying computational capabilities of edge devices. This challenge, also known as Straggler bottleneck, incurs large delays and failures due to computing nodes that are significantly slower than the rest; and (2) Communication Bottleneck, which is due to the massive amounts of raw or processed data that must be moved around the network. To tackle these bottlenecks, this project proposes techniques from coding theory and optimization theory to develop distributed learning algorithms with strong theoretical guarantees and empirical performance. Wireless distributed learning systems are driven by scaling out computations across many wireless edge nodes. There are, however, two major systems bottlenecks that arise: (1) Straggler Delay Bottleneck, which is due to the latency in waiting for slowest nodes to finish their tasks; (2) Data Shuffling Bottleneck, which is due to the massive amounts of data that must be moved among nodes. Moreover, there are privacy concerns about sharing sensitive local data, as well as vulnerabilities to adversarial attacks. This proposal aims to develop novel techniques from coding theory and optimization theory to tackle the mentioned bottlenecks and concerns. The project develops new "coded computing" algorithms for robust gradient aggregation, as well as new optimization algorithms for distributed learning. These algorithms are then used in two network settings to develop communication-efficient, straggler-resilient, and robust distributed learning frameworks: (i) a collaborative setting where a learning task is allocated to multiple edge nodes of the network. In this setting, data points can be encoded and offloaded to the edge nodes to provide resiliency against system bottlenecks; (ii) a federated setting where data points are gathered locally at edge devices and have to remain local due to privacy concerns.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.