MLWiNS: Optimization and Coding Theory for Fast and Robust Wireless Distributed Learning

无线分布式学习系统可以实现各种新的应用，包括工业自动化，语义学习，自动驾驶，医疗保健应用等，虽然无线分布式学习带来了新的机遇，但它面临着两个主要挑战，严重限制了其效率，可靠性和可扩展性：（1）网络异构性，这是由于边缘设备的计算能力不同。这种挑战，也被称为Straggler瓶颈，由于计算节点比其他节点慢得多，会导致大的延迟和故障;（2）通信瓶颈，这是由于大量的原始数据或处理后的数据必须在网络中移动。为了解决这些瓶颈，本项目提出了编码理论和优化理论的技术，以开发具有强大理论保证和经验性能的分布式学习算法。无线分布式学习系统是通过在许多无线边缘节点上扩展计算来驱动的。然而，出现了两个主要的系统瓶颈：（1）掉队者延迟瓶颈，这是由于等待最慢的节点完成其任务的延迟;（2）数据洗牌瓶颈，这是由于必须在节点之间移动大量数据。此外，共享敏感的本地数据也存在隐私问题，以及对抗性攻击的脆弱性。该提案旨在从编码理论和优化理论中开发新技术，以解决上述瓶颈和问题。该项目开发了新的“编码计算”算法，用于强大的梯度聚合，以及新的分布式学习优化算法。这些算法，然后在两个网络设置中使用，以开发通信效率，离散弹性，和强大的分布式学习框架：（i）协作设置，其中学习任务被分配给网络的多个边缘节点。在这种情况下，数据点可以被编码并卸载到边缘节点，以提供对系统瓶颈的弹性;（ii）联合设置，其中数据点在边缘设备本地收集，并且由于隐私问题而必须保持本地。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Wireless distributed learning systems can enable a variety of new applications including industrial automation, semantic learning, autonomous driving, health-care applications, etc. While wireless distributed learning brings about new opportunities, it faces two major challenges that severely limit its efficiency, reliability, and scalability: (1) Network heterogeneity, which is due to varying computational capabilities of edge devices. This challenge, also known as Straggler bottleneck, incurs large delays and failures due to computing nodes that are significantly slower than the rest; and (2) Communication Bottleneck, which is due to the massive amounts of raw or processed data that must be moved around the network. To tackle these bottlenecks, this project proposes techniques from coding theory and optimization theory to develop distributed learning algorithms with strong theoretical guarantees and empirical performance. Wireless distributed learning systems are driven by scaling out computations across many wireless edge nodes. There are, however, two major systems bottlenecks that arise: (1) Straggler Delay Bottleneck, which is due to the latency in waiting for slowest nodes to finish their tasks; (2) Data Shuffling Bottleneck, which is due to the massive amounts of data that must be moved among nodes. Moreover, there are privacy concerns about sharing sensitive local data, as well as vulnerabilities to adversarial attacks. This proposal aims to develop novel techniques from coding theory and optimization theory to tackle the mentioned bottlenecks and concerns. The project develops new "coded computing" algorithms for robust gradient aggregation, as well as new optimization algorithms for distributed learning. These algorithms are then used in two network settings to develop communication-efficient, straggler-resilient, and robust distributed learning frameworks: (i) a collaborative setting where a learning task is allocated to multiple edge nodes of the network. In this setting, data points can be encoded and offloaded to the edge nodes to provide resiliency against system bottlenecks; (ii) a federated setting where data points are gathered locally at edge devices and have to remain local due to privacy concerns.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.