Benchmark of DNN Model Search at Deployment Time
Benchmark of DNN Model Search at Deployment Time
复制标题
DOI：
10.1145/3538712.3538725
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Lixi Zhou;Arindam Jain;Zijie Wang;Amitabh Das;Yingzhen Yang;Jia Zou
中科院分区：
文献类型：
--
作者：
Lixi Zhou;Arindam Jain;Zijie Wang;Amitabh Das;Yingzhen Yang;Jia Zou
深度学习已经成为机器学习和人工智能中最热门的方向。然而，训练数据的准备以及模型训练通常非常耗时，并成为端到端机器学习生命周期的瓶颈。重用模型来推断数据集可以避免重新训练的成本。然而，当存在多个候选模型时，发现正确的模型以供重用是具有挑战性的。尽管存在ModelDB、TensorFlow Hub、PyTorch Hub和DLHub等多个模型共享平台，但这些系统大多需要模型上传者手动指定每个模型和模型下载者的详细信息，以筛选关键字搜索结果来选择模型。我们缺乏一个高效的模型搜索工具，它可以选择模型进行部署，而不需要任何手动检查和/或标记目标领域的数据。本文提出了多模型搜索策略，包括各种基于相似性的方法和非相似性的方法。我们设计，实现和评估这些方法在多个模型推理场景，包括活动识别，图像识别，文本分类，自然语言处理和实体匹配。实验评估表明，我们提出的非对称的基于相似性的测量，自适应性，优于对称的基于相似性的测量和非基于相似性的测量在大多数的工作负载。
Deep learning has become the most popular direction in machine learning and artificial intelligence. However, the preparation of training data, as well as model training, are often time-consuming and become the bottleneck of the end-to-end machine learning lifecycle. Reusing models for inferring a dataset can avoid the costs of retraining. However, when there are multiple candidate models, it is challenging to discover the right model for reuse. Although there exist a number of model sharing platforms such as ModelDB, TensorFlow Hub, PyTorch Hub, and DLHub, most of these systems require model uploaders to manually specify the details of each model and model downloaders to screen keyword search results for selecting a model. We are lacking a highly productive model search tool that selects models for deployment without the need for any manual inspection and/or labeled data from the target domain. This paper proposes multiple model search strategies including various similarity-based approaches and non-similarity-based approaches. We design, implement and evaluate these approaches on multiple model inference scenarios, including activity recognition, image recognition, text classification, natural language processing, and entity matching. The experimental evaluation showed that our proposed asymmetric similarity-based measurement, adaptivity, outperformed symmetric similarity-based measurements and non-similarity-based measurements in most of the workloads.