III: Small: Applying Relational Database Design Principles to Machine Learning System Design

现代机器学习系统，如TensorFlow和PyTorch，已经彻底改变了机器学习模型的发展，使得在短时间内产生新的复杂模型成为可能。然而，这些系统有很大的局限性。它们很难用于在大型数据集上训练大型模型。用户必须手动将计算和数据映射到分布式设置中的硬件。这样做不正确会导致系统故障。模型不容易被分解，也不容易在不同的硬件基础结构中并行训练。当用户希望通过增加更多的机器/工人来加速学习时，结果在许多情况下将是更长的培训时间。这个项目考虑了一个基本的问题：机器学习系统应该建立在什么基础上，这样它们才能轻松地在最大的数据集上促进最大模型的分布式训练？该项目将研究使用关系模型作为机器学习系统设计的基础，其中矩阵和张量被分解并存储在关系中。关系模型长期以来一直是数据库系统的基础，它成功地使用大型机器集群处理巨大的数据集。然而，要使关系系统成为机器学习系统实现的首选平台，还需要解决许多研究问题。首先，有很多方法可以将张量分解并存储在关系中。在数据表示、正在运行的计算和计算到硬件的映射之间存在复杂的交互。如何将所有这些协同优化？其次，与经典的关系计算不同，机器学习计算是迭代的，多次重复相同的计算。计算核（矩阵乘法，卷积等）非常昂贵，使得基于成本的优化变得困难。该项目将研究一种全新的关系优化范例，在这种范例中，迭代执行的关系计算被视为一个马尔可夫决策过程，必须在执行的整个生命周期中进行优化，以便在所有执行中实现最小的成本。最后，当机器学习计算以关系表示时，底层元组存储分解张量的片段。这些元组非常大，并且具有约束，例如键的连续性，这在一般的关系计算中是不存在的。该项目将研究基于优化的关系算法的使用，该算法使用这些约束来仔细放置数据和计划通信，从而最大限度地减少此类大型对象的通信。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern machine learning systems such as TensorFlow and PyTorch have revolutionized the development of machine learning models, making it possible to produce new and complex models in a short period of time. However, these systems have significant limitations. They are difficult to use for training large models over large data sets. A user must manually map computations and data to hardware in a distributed setting. Doing so incorrectly leads to system failure. Models cannot easily be decomposed and trained in parallel across different hardware infrastructures. When a user wishes to speed learning by adding more machines/workers, the result will in many cases be a longer training time. This project considers the fundamental question: What is the foundation upon which machine learning systems should be built so that they can easily facilitate distributed training of the largest models over the largest data sets?The project will investigate use of the relational model as the basis for machine learning system design, where matrices and tensors are decomposed and stored in relations. The relational model has long been the basis for database systems, which successfully process huge data sets using large clusters of machines. However, a number of research problems need to be addressed for relational systems to be the preferred platform for machine learning system implementation. First, there are many ways that a tensor can be decomposed and stored in a relation. There are complex interactions between the data representation, the computation being run, and the mapping of the computation to hardware. How can all of these be co-optimized? Second, unlike classical relational computations, machine learning computations are iterative, repeating the same computation many times. The compute kernels (matrix multiplications, convolutions, etc.) are very expensive, making cost-based optimization difficult. The project will investigate an entirely new paradigm for relational optimization, where rather than being statically optimized, a relational computation that is executed iteratively is treated as a Markov decision process that must be optimized over its lifetime of executions, so as to achieve minimum cost over all executions. Finally, when machine learning computations are expressed relationally, the underlying tuples store pieces of decomposed tensors. Those tuples are very large, and have constraints such as continuity of keys that are not present in general, relational computations. The project will investigate the use of optimization-based relational algorithms that use those constraints to carefully place the data and plan communication so as to minimize the communication of such large objects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.