Encoded computing for efficient and robust large-scale distributed optimization

点击翻译按钮获取中文摘要

The availability of big data and the use of these data in training artificial intelligence (AI) systems is changing the way companies do business in a wide swath of industries, from finance to entertainment to drug discovery. These advances rely on a computing, networking, and data storage infrastructure, the growing capabilities of which are key to realizing the promise of AI. However, as data sets and processing requirements scale up massively, classic single-processor computing paradigms cannot keep up. There has therefore been a renaissance in understanding how to bring large-scale parallelization to bear on algorithmic design for learning-specific workloads. As we outline in this proposal, we can draw on techniques, perspectives, and solutions from digital communications and error-correction coding to develop novel robust and resource-efficient approaches to parallelized computing. Our long-term vision is that by drawing on the theory and practice of digital communications we can deliver novel and unexpected impact on the theory and practice of computation. In the shorter term, we will build towards our vision by following three coupled research themes. These themes draw on and contribute to the theory and practice of error-correction coding and distributed optimization to enable large-scale, robust, and efficient computing systems. In the first theme we design large-scale distributed optimization techniques that deliver idealized performance while coping with the non-idealities of real-world cloud computing systems including the effects of "straggler" nodes and network delays. In the second, we draw on perspectives from information theory and error correction to develop novel "coding-for-computing" paradigms that deliver robust and efficient computation. In the final theme we work closely with systems groups to help push our results into application and to improve our understanding of the myriad real-world issues facing computing systems, motivating further research and deepening collaborations.