CIF: Medium: Collaborative Research: Coded Computing for Large-Scale Machine Learning

CIF：媒介：协作研究：大规模机器学习的编码计算

• 批准号: 1763561
• 负责人: Pulkit Grover
• 金额: $ 30万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1763561&HistoricalAwards=false
• 关键词: CIF; Medium; Collaborative; Research; Coded

Deep learning models are breaking new ground in data science tasks including image recognition, automatic translation and autonomous driving. This is achieved by neural networks that can be hundreds of layers deep and involve hundreds of millions of parameters. Training such large models requires distributed computations, very long training times and expensive hardware. This project studies coding theoretic techniques that can accelerate distributed machine learning and allow training with cheaper commodity hardware. Beyond the development of theoretical foundations, this project develops new algorithms for providing fault tolerance over unreliable cloud infrastructure that can significantly reduce the cost of large-scale machine learning. The research outcomes of the project will be broadly disseminated and integrated into education. The specific focus of this research program is on mitigating the bottlenecks of distributed machine learning. Currently, scaling benefits are limited because of two reasons: first, communication is typically the bottleneck and second, straggler effects limit performance. Both problems can be mitigated using coding theoretic methods. This work proposes "coded computing", a transformative framework that combines coding theory with distributed computing to inject computational redundancy in a novel coded form. This framework is then used to develop three research thrusts: a) Coding for Linear Algebraic Computations b) Coding for Iterative Computations and c) Coding for General Distributed Computations. Each of the thrusts operates on a different layer of a machine learning pipeline but all rely on coding theoretic tools and distributed information processing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习模型正在数据科学任务中开辟新天地，包括图像识别、自动翻译和自动驾驶。这是通过神经网络来实现的，神经网络可以有数百层，涉及数亿个参数。训练如此大的模型需要分布式计算，非常长的训练时间和昂贵的硬件。该项目研究编码理论技术，可以加速分布式机器学习，并允许使用更便宜的商品硬件进行训练。除了开发理论基础之外，该项目还开发了新的算法，用于在不可靠的云基础设施上提供容错，从而大大降低大规模机器学习的成本。该项目的研究成果将广泛传播并纳入教育。该研究计划的具体重点是缓解分布式机器学习的瓶颈。目前，扩展的好处是有限的，因为两个原因：第一，通信通常是瓶颈，第二，掉队者影响限制性能。这两个问题都可以使用编码理论方法来缓解。这项工作提出了“编码计算”，一个变革性的框架，结合编码理论与分布式计算注入计算冗余在一个新的编码形式。这个框架，然后用来开发三个研究重点：a）线性代数计算的编码B）迭代计算的编码和c）一般分布式计算的编码。每个项目都在机器学习管道的不同层上运行，但都依赖于编码理论工具和分布式信息处理。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Coded QR Decomposition for Solving System of Linear Equations

求解线性方程组的编码QR分解

• 发表时间: 2020
• 期刊: IEEE International Symposium on Information Theory (ISIT
• 作者: Quang Minh Nguyen, Haewon Jeong

Addressing Unreliability in Emerging Devices and Non-von Neumann Architectures Using Coded Computing

• DOI: 10.1109/jproc.2020.2986362
• 发表时间: 2020-08-01
• 期刊: PROCEEDINGS OF THE IEEE
• 影响因子: 20.6
• 作者: Dutta, Sanghamitra;Jeong, Haewon;Grover, Pulkit
• 通讯作者: Grover, Pulkit

Neural silences can be localized rapidly using noninvasive scalp EEG.

• DOI: 10.1038/s42003-021-01768-0
• 发表时间: 2021-03-30
• 期刊: Communications biology
• 影响因子: 5.9
• 作者: Chamanzar A;Behrmann M;Grover P
• 通讯作者: Grover P

Masterless Coded Computing: A Fully-Distributed Coded FFT Algorithm

Masterless编码计算：一种全分布式编码FFT算法

• DOI: 10.1109/allerton.2018.8636047
• 发表时间: 2018
• 期刊: and Computing (Allerton
• 作者: Jeong, Haewon;Low, Tze Meng;Grover, Pulkit
• 通讯作者: Grover, Pulkit

Robust Molecular Dynamics Simulations Using Coded FFT Algorithm

使用编码 FFT 算法进行稳健的分子动力学模拟

• DOI: 10.1109/icassp.2019.8682276
• 发表时间: 2019
• 期刊: Speech and Signal Processing (ICASSP
• 作者: Wong, Yuk;Zhang, Yuqiu;Jeong, Haewon;Grover, Pulkit
• 通讯作者: Grover, Pulkit