III: Small: Applying Relational Database Design Principles to Machine Learning System Design

TensorFlow和PyTorch等现代机器学习系统彻底改变了机器学习模型的发展，使其能够在短时间内生成新的复杂模型。然而，这些系统具有显著的局限性。它们很难用于在大型数据集上训练大型模型。用户必须手动地将计算和数据映射到分布式设置中的硬件。不正确的操作会导致系统故障。模型不能轻易地在不同的硬件基础设施上并行分解和训练。当用户希望通过添加更多机器/工人来加速学习时，在许多情况下，结果将是更长的培训时间。该项目考虑的基本问题是：机器学习系统应该建立在什么基础上，以便它们可以轻松地在最大的数据集上进行最大模型的分布式训练？该项目将研究使用关系模型作为机器学习系统设计的基础，其中矩阵和张量被分解并存储在关系中。关系模型长期以来一直是数据库系统的基础，它使用大型机器集群成功地处理巨大的数据集。然而，一些研究问题需要解决的关系系统是机器学习系统实现的首选平台。首先，有很多方法可以将张量分解并存储在关系中。在数据表示、正在运行的计算以及计算到硬件的映射之间存在复杂的交互。所有这些如何才能共同优化？其次，与经典的关系计算不同，机器学习计算是迭代的，多次重复相同的计算。计算内核（矩阵乘法，卷积等）非常昂贵，使得基于成本的优化变得困难。该项目将研究一种全新的关系优化范式，而不是静态优化，迭代执行的关系计算被视为马尔可夫决策过程，必须在其执行生命周期内进行优化，以便在所有执行中实现最小成本。最后，当机器学习计算以关系方式表达时，底层元组存储分解的张量片段。这些元组非常大，并且具有诸如键的连续性之类的约束，这些约束在一般的关系计算中不存在。该项目将研究基于优化的关系算法的使用，该算法使用这些约束来仔细放置数据和计划通信，以最大限度地减少此类大型对象的通信。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern machine learning systems such as TensorFlow and PyTorch have revolutionized the development of machine learning models, making it possible to produce new and complex models in a short period of time. However, these systems have significant limitations. They are difficult to use for training large models over large data sets. A user must manually map computations and data to hardware in a distributed setting. Doing so incorrectly leads to system failure. Models cannot easily be decomposed and trained in parallel across different hardware infrastructures. When a user wishes to speed learning by adding more machines/workers, the result will in many cases be a longer training time. This project considers the fundamental question: What is the foundation upon which machine learning systems should be built so that they can easily facilitate distributed training of the largest models over the largest data sets?The project will investigate use of the relational model as the basis for machine learning system design, where matrices and tensors are decomposed and stored in relations. The relational model has long been the basis for database systems, which successfully process huge data sets using large clusters of machines. However, a number of research problems need to be addressed for relational systems to be the preferred platform for machine learning system implementation. First, there are many ways that a tensor can be decomposed and stored in a relation. There are complex interactions between the data representation, the computation being run, and the mapping of the computation to hardware. How can all of these be co-optimized? Second, unlike classical relational computations, machine learning computations are iterative, repeating the same computation many times. The compute kernels (matrix multiplications, convolutions, etc.) are very expensive, making cost-based optimization difficult. The project will investigate an entirely new paradigm for relational optimization, where rather than being statically optimized, a relational computation that is executed iteratively is treated as a Markov decision process that must be optimized over its lifetime of executions, so as to achieve minimum cost over all executions. Finally, when machine learning computations are expressed relationally, the underlying tuples store pieces of decomposed tensors. Those tuples are very large, and have constraints such as continuity of keys that are not present in general, relational computations. The project will investigate the use of optimization-based relational algorithms that use those constraints to carefully place the data and plan communication so as to minimize the communication of such large objects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.