Nautilus: An Optimized System for Deep Transfer Learning over Evolving Training Datasets
Nautilus: An Optimized System for Deep Transfer Learning over Evolving Training Datasets
复制标题
DOI：
10.1145/3514221.3517846
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Supun Nakandala;Arun Kumar
中科院分区：
文献类型：
--
作者：
Supun Nakandala;Arun Kumar
深度学习（DL）彻底改变了非结构化数据分析。但在大多数情况下，深度学习需要大量标记数据集和大型计算集群，这阻碍了它的采用。这些限制可以使用一种称为深度迁移学习（DTL）的流行范式来克服。使用DTL，可以适应预训练的DL模型，而不是从头开始训练模型。因此，DTL减少了训练模型所需的大量训练数据和计算量。在适应过程中，一个常见的做法是冻结大多数预先训练的模型部分，只适应剩余的部分。由于没有单一的适应方案是普遍最好的，人们往往会对几种方案进行评估，这也被称为模式选择。我们还观察到，DTL的数据标记很少是一次性的过程。人们经常通过添加新的标记数据来间歇性地更新他们的标记数据，并进行模型选择来评估训练模型的准确性。今天，人们通过对整个预训练模型执行计算并在每个模型选择周期中重复它来执行此工作负载。这种方法导致在固定的模型部件中进行冗余计算，并导致可用性和系统效率低下问题。在这项工作中，我们将存在冻结层的DTL模型选择重新想象为多查询优化的一个实例，并提出了两种减少冗余计算和训练开销的优化方法。我们在一个名为Nautilus的数据系统中实现了我们的优化。在基准数据集上对端到端工作负载进行的实验表明，与当前的实践相比，Nautilus将DTL模型选择的运行时间减少了5倍。
Deep learning (DL) has revolutionized unstructured data analytics. But in most cases, DL needs massive labeled datasets and large compute clusters, which hinders its adoption. These limitations can be overcome using a popular paradigm called deep transfer learning (DTL). With DTL, one adapts a pre-trained DL model instead of training a model from scratch. Thus, DTL reduces the massive training data and compute requirements to train a model. During adaptation, a common practice is to freeze most pre-trained model parts and adapt only the remaining. Since no single adaptation scheme is universally the best, one often evaluates several schemes, which is also called model selection. We also observed that data labeling for DTL is seldom a one-off process. One often updates their labeled data intermittently by adding new labeled data and performs model selection to evaluate the accuracy of the trained models. Today, one executes this workload by performing computations for the entire pre-trained model and repeats it for every model selection cycle. This approach results in redundant computations in frozen model parts and causes usability and system inefficiency issues. In this work, we reimagine DTL model selection in the presence of frozen layers as an instance of multi-query optimization and propose two optimizations that reduce redundant computations and training overheads. We implement our optimizations into a data system called Nautilus. Experiments with end-to-end workloads on benchmark datasets show that Nautilus reduces DTL model selection runtimes by up to 5X compared to the current practice.