An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry
An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry
复制标题
DOI：
10.1109/icse48619.2023.00206
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Wenxin Jiang;Nicholas Synovic;Matt Hyatt;Taylor R. Schorlemmer;R. Sethi;Yung-Hsiang Lu;G. Thiruvathukal;James C. Davis
中科院分区：
文献类型：
--
作者：
Wenxin Jiang;Nicholas Synovic;Matt Hyatt;Taylor R. Schorlemmer;R. Sethi;Yung-Hsiang Lu;G. Thiruvathukal;James C. Davis
深度神经网络（DNN）被用作软件系统中的组件。随着最先进的体系结构的增长越来越复杂，从头开始创建和专门化DNN变得越来越困难。遵循传统软件工程的路径，机器学习工程师已经开始重复使用大规模的预训练模型（PTMS），并为下游任务微调这些模型。先前的工作已经研究了传统软件包的重复使用实践，以指导软件工程师更好的软件包维护和依赖性管理。我们缺乏类似的知识基础来指导预训练的模型生态系统中的行为。在这项工作中，我们介绍了PTM重用的首次实证研究。我们采访了来自最受欢迎的PTM生态系统的12位从业者，拥抱面孔，以学习PTM Reuse的实践和挑战。从这些数据中，我们为PTM重复使用的决策过程建模。基于确定的实践，我们描述了模型重复使用的有用属性，包括出处，可重复性和便携性。 PTM重复使用的三个挑战是缺少属性，索赔与实际绩效之间的差异以及模型风险。我们通过在拥抱面对生态系统中进行系统的测量来证实这些确定的挑战。我们的工作通过自动衡量有用的属性和潜在攻击，并设想对模型注册机构的基础架构和标准化的未来研究，从而为未来的方向提供了优化深度学习生态系统的指示。
Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as state-of-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems. In this work, we present the first empirical investigation of PTM reuse. We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. From this data, we model the decision-making process for PTM reuse. Based on the identified practices, we describe useful attributes for model reuse, including provenance, reproducibility, and portability. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks. We substantiate these identified challenges with systematic measurements in the Hugging Face ecosystem. Our work informs future directions on optimizing deep learning ecosystems by automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries.