III: Small: Applying Relational Database Design Principles to Machine Learning System Design

点击翻译按钮获取中文摘要

Modern machine learning systems such as TensorFlow and PyTorch have revolutionized the development of machine learning models, making it possible to produce new and complex models in a short period of time. However, these systems have significant limitations. They are difficult to use for training large models over large data sets. A user must manually map computations and data to hardware in a distributed setting. Doing so incorrectly leads to system failure. Models cannot easily be decomposed and trained in parallel across different hardware infrastructures. When a user wishes to speed learning by adding more machines/workers, the result will in many cases be a longer training time. This project considers the fundamental question: What is the foundation upon which machine learning systems should be built so that they can easily facilitate distributed training of the largest models over the largest data sets?The project will investigate use of the relational model as the basis for machine learning system design, where matrices and tensors are decomposed and stored in relations. The relational model has long been the basis for database systems, which successfully process huge data sets using large clusters of machines. However, a number of research problems need to be addressed for relational systems to be the preferred platform for machine learning system implementation. First, there are many ways that a tensor can be decomposed and stored in a relation. There are complex interactions between the data representation, the computation being run, and the mapping of the computation to hardware. How can all of these be co-optimized? Second, unlike classical relational computations, machine learning computations are iterative, repeating the same computation many times. The compute kernels (matrix multiplications, convolutions, etc.) are very expensive, making cost-based optimization difficult. The project will investigate an entirely new paradigm for relational optimization, where rather than being statically optimized, a relational computation that is executed iteratively is treated as a Markov decision process that must be optimized over its lifetime of executions, so as to achieve minimum cost over all executions. Finally, when machine learning computations are expressed relationally, the underlying tuples store pieces of decomposed tensors. Those tuples are very large, and have constraints such as continuity of keys that are not present in general, relational computations. The project will investigate the use of optimization-based relational algorithms that use those constraints to carefully place the data and plan communication so as to minimize the communication of such large objects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.