Encoded computing for efficient and robust large-scale distributed optimization

大数据的可用性以及在训练人工智能（AI）系统中使用这些数据正在改变公司在从金融到娱乐再到药物发现等广泛行业中开展业务的方式。这些进步依赖于计算、网络和数据存储基础设施，其不断增长的能力是实现人工智能承诺的关键。然而，随着数据集和处理需求的大规模扩展，经典的单处理器计算范式无法跟上。因此，在理解如何将大规模并行化应用于学习特定工作负载的算法设计方面出现了复兴。正如我们在本提案中所概述的那样，我们可以利用数字通信和纠错编码的技术、观点和解决方案来开发新的鲁棒和资源高效的并行计算方法。我们的长期愿景是，通过借鉴数字通信的理论和实践，我们可以对计算的理论和实践产生新的和意想不到的影响。 在短期内，我们将通过以下三个耦合的研究主题来实现我们的愿景。 这些主题借鉴并有助于纠错编码和分布式优化的理论和实践，以实现大规模，强大和高效的计算系统。 在第一个主题中，我们设计了大规模的分布式优化技术，提供理想化的性能，同时应对现实世界的云计算系统的非理想性，包括“掉队”节点和网络延迟的影响。 在第二部分中，我们从信息论和纠错的角度来开发新的“计算编码”范式，提供强大而有效的计算。 在最后一个主题中，我们与系统组密切合作，以帮助将我们的结果应用于应用，并提高我们对计算系统面临的无数现实问题的理解，激励进一步的研究和深化合作。

The availability of big data and the use of these data in training artificial intelligence (AI) systems is changing the way companies do business in a wide swath of industries, from finance to entertainment to drug discovery. These advances rely on a computing, networking, and data storage infrastructure, the growing capabilities of which are key to realizing the promise of AI. However, as data sets and processing requirements scale up massively, classic single-processor computing paradigms cannot keep up. There has therefore been a renaissance in understanding how to bring large-scale parallelization to bear on algorithmic design for learning-specific workloads. As we outline in this proposal, we can draw on techniques, perspectives, and solutions from digital communications and error-correction coding to develop novel robust and resource-efficient approaches to parallelized computing. Our long-term vision is that by drawing on the theory and practice of digital communications we can deliver novel and unexpected impact on the theory and practice of computation. In the shorter term, we will build towards our vision by following three coupled research themes. These themes draw on and contribute to the theory and practice of error-correction coding and distributed optimization to enable large-scale, robust, and efficient computing systems. In the first theme we design large-scale distributed optimization techniques that deliver idealized performance while coping with the non-idealities of real-world cloud computing systems including the effects of “straggler” nodes and network delays. In the second, we draw on perspectives from information theory and error correction to develop novel “coding-for-computing” paradigms that deliver robust and efficient computation. In the final theme we work closely with systems groups to help push our results into application and to improve our understanding of the myriad real-world issues facing computing systems, motivating further research and deepening collaborations.