CIF: Small: Coding Techniques for Distributed Machine Learning

CIF：小型：分布式机器学习的编码技术

• 批准号: 1910447
• 负责人: Jun Li
• 金额: $ 50万
• 依托单位: Florida International University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910447&HistoricalAwards=false
• 关键词: CIF; Small; Coding; Techniques; Distributed

Modern machine learning models have achieved great success and have been widely deployed across many sectors. As the size of data used to train machine learning models keeps growing, it is now routine to use distributed computing infrastructures such as the cloud. This strategy allows the computation of training to be distributed among a large number of nodes hosted in the cloud, where each node processes a partition of the whole data set. However, the performance of nodes in the cloud is often unreliable, due to system failures, resource contention, load imbalance, etc., and that unreliability can significantly delay the training process. This project pursues a coding-based framework that not only tolerates the effects of faulty nodes, but also further enhances the performance of machine learning training by dynamically taking advantage of the resources available on all nodes, whether they are faulty or not. The outcomes of this project should lead to a significant performance boost for distributed training of machine learning models.To enable the efficient use of distributed computing across unreliable infrastructure for training machine learning models from big data sets, the technical objectives of this project are divided into three levels. This project will first study coding theory for distributed matrix multiplication, a universal operation in various machine learning algorithms, and propose a coding framework with both fault tolerance and a significant performance boost. This framework will then be applied into parameter servers at the architecture level and deep neural networks at the model level, respectively. Combining these three parts, this work will lead to a practical coding framework that can efficiently scale out computation on heterogeneous unreliable nodes, where the coding schemes will be applied to distributed machine learning at different levels including fundamental arithmetic, architectures, and models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代机器学习模型取得了巨大成功，并已广泛应用于许多领域。随着用于训练机器学习模型的数据规模不断增长，使用云等分布式计算基础设施现在已成为惯例。该策略允许训练计算分布在云中托管的大量节点中，其中每个节点处理整个数据集的一部分。然而，由于系统故障、资源争用、负载不平衡等原因，云中节点的性能往往不可靠，并且这种不可靠性会显着延迟训练过程。该项目追求一种基于编码的框架，不仅可以容忍故障节点的影响，而且可以通过动态利用所有节点上的可用资源（无论是否故障）来进一步增强机器学习训练的性能。该项目的成果应该会显着提高机器学习模型的分布式训练的性能。为了能够跨不可靠的基础设施有效地使用分布式计算来训练来自大数据集的机器学习模型，该项目的技术目标分为三个层次。该项目将首先研究分布式矩阵乘法的编码理论，这是各种机器学习算法中的通用运算，并提出一种兼具容错性和显着性能提升的编码框架。然后，该框架将分别应用于架构级别的参数服务器和模型级别的深度神经网络。结合这三个部分，这项工作将产生一个实用的编码框架，可以有效地扩展异构不可靠节点上的计算，其中编码方案将应用于不同级别的分布式机器学习，包括基础算法、架构和模型。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查进行评估，被认为值得支持。 标准。

项目成果

Local Re-encoding for Coded Matrix Multiplication

编码矩阵乘法的本地重新编码

• DOI: 10.1109/isit44484.2020.9174041
• 发表时间: 2020
• 期刊: 2020 IEEE International Symposium on Information Theory (ISIT
• 作者: Su, Xian;Zhong, Xiaomei;Fan, Xiaodi;Li, Jun
• 通讯作者: Li, Jun

Straggler-free Coding for Concurrent Matrix Multiplications

• DOI: 10.1109/isit44484.2020.9174239
• 发表时间: 2020-06
• 期刊: 2020 IEEE International Symposium on Information Theory (ISIT)
• 作者: Pedro Soto;Jun Li