CIF: Small: Coding Techniques for Distributed Machine Learning

现代机器学习模型已经取得了巨大的成功，并在许多领域得到了广泛的部署。随着用于训练机器学习模型的数据量持续增长，现在使用云等分布式计算基础设施已成为惯例。这种策略允许在云中托管的大量节点之间分布训练计算，其中每个节点处理整个数据集的一个分区。然而，由于系统故障、资源争用、负载不平衡等原因，云中节点的性能往往是不可靠的，这种不可靠会显著延迟训练过程。该项目追求一种基于编码的框架，该框架不仅可以容忍故障节点的影响，而且通过动态利用所有节点上的可用资源来进一步增强机器学习训练的性能，无论这些节点是否有故障。该项目的成果应该会显著提高机器学习模型的分布式训练的性能。为了能够在不可靠的基础设施上高效地使用分布式计算来训练来自大数据集的机器学习模型，该项目的技术目标分为三个层次。该项目将首先研究分布式矩阵乘法的编码理论，这是各种机器学习算法中的一种通用操作，并提出一种既具有容错能力又具有显著性能提升的编码框架。然后，该框架将分别应用于体系结构级别的参数服务器和模型级别的深度神经网络。将这三个部分结合在一起，这项工作将产生一个实用的编码框架，它可以有效地扩展异类不可靠节点上的计算，其中编码方案将应用于不同级别的分布式机器学习，包括基本算法、体系结构和模型。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern machine learning models have achieved great success and have been widely deployed across many sectors. As the size of data used to train machine learning models keeps growing, it is now routine to use distributed computing infrastructures such as the cloud. This strategy allows the computation of training to be distributed among a large number of nodes hosted in the cloud, where each node processes a partition of the whole data set. However, the performance of nodes in the cloud is often unreliable, due to system failures, resource contention, load imbalance, etc., and that unreliability can significantly delay the training process. This project pursues a coding-based framework that not only tolerates the effects of faulty nodes, but also further enhances the performance of machine learning training by dynamically taking advantage of the resources available on all nodes, whether they are faulty or not. The outcomes of this project should lead to a significant performance boost for distributed training of machine learning models.To enable the efficient use of distributed computing across unreliable infrastructure for training machine learning models from big data sets, the technical objectives of this project are divided into three levels. This project will first study coding theory for distributed matrix multiplication, a universal operation in various machine learning algorithms, and propose a coding framework with both fault tolerance and a significant performance boost. This framework will then be applied into parameter servers at the architecture level and deep neural networks at the model level, respectively. Combining these three parts, this work will lead to a practical coding framework that can efficiently scale out computation on heterogeneous unreliable nodes, where the coding schemes will be applied to distributed machine learning at different levels including fundamental arithmetic, architectures, and models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.