Cerebro: A Layered Data Platform for Scalable Deep Learning-猫眼课题宝

DOI：

发表时间：

2021

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Arun Kumar;Advitya Gemawat;Kabir;Nagrecha;Yuhao Zhang;Side Li

得益于 TensorFlow 等工具以及更容易访问 GPU，深度学习 (DL) 在许多领域越来越受欢迎。但对于除了大型科技公司之外的所有人来说，构建大规模深度学习应用程序仍然过于资源密集且痛苦。造成这种痛苦的一个关键原因是让深度学习正常运行所需的昂贵的模型选择过程。现有的深度学习系统将此过程视为事后的想法，导致大量资源浪费和可用性混乱。为了解决这些问题，受数据库世界经验教训的启发，我们提出了针对可扩展深度学习的首个数据平台 Cerebro 的愿景。我们利用实践中固有的更高级别的 API 来提升深度学习模型选择过程，并设计了一系列新颖的多查询优化技术来大幅提高资源效率。这份愿景文件介绍了我们的系统设计理念和架构、我们最近的研究和开放研究问题、初步结果，以及对实现实际影响的切实途径的讨论。 1 深度学习：承诺，痛苦深度学习 (DL) 彻底改变了机器学习 (ML)，为现代语音识别、机器翻译、电子商务、放射学等提供动力。深度学习的成功（主要是在网络巨头中）引起了领域科学家、企业、小型网络公司甚至医疗保健公司对尝试使用深度学习完成分析任务的浓厚兴趣。令人兴奋的主要是由于深度学习能够解锁有价值的非结构化数据，例如公共卫生中的传感器时间序列、农业中的卫星图像以及金融中的文本语料库。虽然 Spark 和 Flink 等工具控制了“大数据”分析的“数量”和“速度”方面，但“多样性”问题在很大程度上仍然悬而未决。深度学习是驾驭分析的第三个 V 的关键。当然，TensorFlow 等新工具旨在让深度学习更易于使用。云计算也让 GPU 变得更容易使用。尽管取得了所有这些进展，深度学习仍然难以大规模使用、资源密集且成本高昂。虽然现成的预训练模型可以缓解一些定义明确的预测任务的这个问题，但大多数机器学习用户仍然需要手动或使用 AutoML 程序为定制数据集和应用程序构建自定义模型。这个过程很痛苦，很大程度上是由于深度学习的灵活性，这在机器学习中是前所未有的：输入/输出几乎可以是任何数据结构，内部层可以通过多种方式构建[27]。这种端到端的学习使深度学习更加准确，但也使得深度学习训练对软件商品化具有异常的抵抗力。即使是云“鲸鱼”也没有为 DL 训练提供超出基本 IaaS 支持的内容（尽管针对某些推理任务存在 SaaS）。本文根据知识共享署名许可 (http://creativecommons.org/licenses/by/3.0/) 发布，该许可允许在任何媒体上分发和复制，并允许衍生作品，前提是您将原始作品归属于作者和 CIDR 2021。第 11 届创新数据系统研究年会 (CIDR ‘21)，2021 年 1 月 10 日至 13 日，美国加利福尼亚州查米纳德这也是其成本的一个关键原因：模型选择，这是一种将替代训练配置与不同的输入/输出表示、神经架构和/或超参数进行比较的经验过程[37]。由于机器学习精度 [57] 中偏差-方差-噪声权衡的基本理论原因以及许多任务的定制性质，这个过程是不可避免的。如果没有正确的模型选择，用户只会浪费标记数据的力量，从而损害应用程序。遗憾的是，没有可靠的方法可以在不实际运行给定数据集的情况下判断配置的准确性，从而导致大量经验主义。如果调整 3 个超参数，每个超参数有 4 个可能值，则配置数量已经是 64 种。将其与对不同输入/输出表示和神经架构的更多试验相乘。难怪从业者通常会构建数十个甚至数千个模型来选择一个模型 [6]。总体而言，模型选择对于有效使用深度学习至关重要，但它极大地放大了深度学习的成本和能源足迹，并妨碍了可用性。 1.1 现有工具缺少什么？为什么？ TensorFlow 等深度学习系统满足相当低级的需求：指定神经架构，使用随机梯度下降（SGD）或其变体对其进行训练，然后运行推理 [12]。他们在这方面做得相当好，至少在单个节点上是这样。但用户还必须以某种方式指定模型选择过程，这就是 Keras [21]、AutoKeras [31] 和 Hyperband [42] 等 API 出现的主要原因。在深度学习系统中，模型选择是事后才想到的，是由用户承担的负担，需要通过临时外循环和库来处理，而不是作为关键任务的过程。这种设计导致了两个问题：（1）可用性混乱：如果没有一流的模型选择支持，用户会以临时方式处理额外的代码和元数据。这通常会导致不可重复的执行并阻碍跨组织协作。虽然一些云供应商和机器学习平台开始更好地支持此过程，但它在很大程度上仍然是零碎的，并且通常仅限于超参数调整等基本任务。 (2)高资源浪费：在模型选择中尝试的不同配置通常在数据和/或计算上有很大的重叠。忽略这种结构会导致计算、存储/内存和/或网络资源的大量浪费，尤其是在大规模情况下。这个问题与加快单模型训练速度和设计更好的 AutoML 程序来导航搜索空间是正交的。虽然世界上的谷歌、Facebook 和亚马逊可能会在不考虑整体资源效率的情况下，贪婪地将数百个 GPU/TPU 和工程师投入到他们的 DL 任务中，但大多数其他 DL 用户负担不起，尤其是在即用即付的云上。尽管一些网络巨头可能会忽略某些机器学习任务的可重复性，但这对大多数企业和领域科学家来说都是一个令人头疼的问题。网络巨头和云鲸鱼之外的深度学习用户群快速增长，迫切需要解决这些问题。迁移学习高级

Deep learning (DL) is gaining popularity across many domains thanks to tools such as TensorFlow and easier access to GPUs. But building large-scale DL applications is still too resource-intensive and painful for all but the big tech firms. A key reason for this pain is the expensivemodel selection process needed to get DL to work well. Existing DL systems treat this process as an afterthought, leading to massive resource wastage and a usability mess. To tackle these issues, we present our vision of a first-of-its-kind data platform for scalable DL, Cerebro, inspired by lessons from the database world. We elevate the DL model selection process with higherlevel APIs already inherent in practice and devise a series of novel multi-query optimization techniques to substantially raise resource efficiency. This vision paper presents our system design philosophy and architecture, our recent research and open research questions, initial results, and a discussion of tangible paths to practical impact. 1 DEEP LEARNING: THE PROMISE, THE PAIN Deep learning (DL) has revolutionized machine learning (ML), poweringmodern speech recognition, machine translation, e-commerce, radiology, and more. The successes of DL, primarily at Web giants, has led to high interest among domain scientists, enterprises, smaller Web companies, and even healthcare firms in trying DL for their analytics tasks. The excitement is mainly due to DL’s power to unlock valuable unstructured data, e.g., sensor time series in public health, satellite images in agriculture, and text corpora in finance. While tools such as Spark and Flink tame the “Volume” and “Velocity” aspects of “Big Data” analytics, the “Variety” issue is still largely open. DL is key to taming this third V for analytics. Naturally, new tools such as TensorFlow aim to make DL easier to use. Cloud computing is also making GPUs more accessible. In spite of all this progress, DL is still too hard to use at scale, resource-intensive, and costly. While off-the-shelf pre-trained models mitigate this issue for a few well-defined prediction tasks, most ML users still need to build custom models for bespoke datasets and applications, either manually or with AutoML procedures. This process is painful in large part due to DL’s flexibility, which is unprecedented in ML: the input/output can be almost any data structure and internal layers can be built in many ways [27]. Such end-to-end learning makes DL more accurate but it also makes DL training unusually resistant to software commodification. Even the cloud “whales” do not offer much beyond basic IaaS support for DL training (although SaaS exists for some inference tasks). This article is published under a Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits distribution and reproduction in any medium as well allowing derivative works, provided that you attribute the original work to the author(s) and CIDR 2021. 11th Annual Conference on Innovative Data Systems Research (CIDR ‘21), January 10–13, 2021, Chaminade, CA, USA DL’s flexibility also underlies a key reason for its cost: model selection, an empirical process comparing alternate training configurations with varying input/output representations, neural architectures, and/or hyperparameters [37]. This process is unavoidable due to fundamental theoretical reasons on the bias-variance-noise tradeoff in ML accuracy [57] and the bespoke nature of many tasks. Without proper model selection, users will just squander the power of their labeled data, hurting the application. Alas, there exist no reliable ways to tell the accuracy of a configuration on a given dataset without actually running it, leading to mass empiricism. If one tuned, say, 3 hyperparameters with, say, 4 possible values each, the number of configurations is 64 already. Multiply that with more trials for varying input/output representations and neural architectures. No wonder practitioners routinely build 10s to even 1000s of models to pick just one [6]. Overall, model selection is critical for effective use of DL but it massively amplifies DL’s cost and energy footprints and impedes usability. 1.1 What do Existing Tools Lack? Why? DL systems such as TensorFlow meet a rather low-level need: specify a neural architecture, train it with stochastic gradient descent (SGD) or its variants, and run inference [12]. They do this rather well, at least on a single node. But the user must also specify the model selection process somehow, which is mainly why APIs such as Keras [21], AutoKeras [31], and Hyperband [42] arose. Model selection is an afterthought in DL systems, a burden dumped on the user to be handled with ad hoc outer loops and libraries, not as the mission-critical process it is. This design has led to two issues: (1)Usabilitymess:Without first-class support for model selection, users handle extra code and metadata in ad hoc ways. This often results in unreproducible executions and stymies cross-organizational collaboration.While some cloud vendors andML platforms are starting to support this process better, it is still largely piecemeal and often limited to only basic tasks such as hyperparameter tuning. (2)High resourcewastage:Different configurations tried inmodel selection often overlap substantially on data and/or computations. Ignoring this structure leads to high wastage of compute, storage/memory, and/or network resources, especially at scale. This issue is orthogonal to both making single-model training faster and to devising better AutoML procedures to navigate the search space. While the Googles, Facebooks, and Amazons of the world may gluttonously throw 100s of GPUs/TPUs and engineers at their DL tasks with low regard for overall resource efficiency, most other DL users cannot afford that, especially on pay-as-you-go clouds. And while some Web giants may ignore reproducibility for some ML tasks, it is a showstopper issue for most enterprises and domain scientists. There is a pressing need to meet these concerns of the fast-growing user base of DL outside Web giants and cloud whales. Transfer Learning High-level

Cerebro: A Layered Data Platform for Scalable Deep Learning

文献摘要

被引文献

相似文献