Mistify: Automating DNN Model Porting for On-Device Inference at the Edge
Mistify: Automating DNN Model Porting for On-Device Inference at the Edge
复制标题
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Peizhen Guo;Bo Hu;Wenjun Hu
中科院分区：
文献类型：
--
作者：
Peizhen Guo;Bo Hu;Wenjun Hu
由深度学习推理驱动的AI应用程序越来越多地在边缘设备上本地运行，以提供更好的交互式用户体验。这通常需要将最初在云中设计和训练的模型适配到具有一系列硬件功能的边缘设备，到目前为止，这一直依赖于耗时的手动工作。在本文中，我们量化了手动生成大量压缩模型的挑战，然后构建了一个系统框架Mistify，以自动将基于云的模型移植到针对设计空间中各个点的边缘设备的一套模型。Mistify添加了一个中间“层”，它将模型设计和部署阶段合并在一起。通过暴露配置API来满足对深入嵌入到原始模型中的代码更改的需求，Mistify向模型设计者隐藏了运行时问题，并向模型用户隐藏了模型内部，从而减少了两者所需的专业知识。为了获得更好的可扩展性，Mistify合并了多个模型裁剪请求，以最大限度地减少重复计算。此外，Mistify以隐私感知的方式利用本地可用的边缘数据，并执行运行时模型自适应，以提供可扩展的边缘支持和准确的推理结果。广泛的评估表明，Mistify将DNN移植所需的时间减少了10倍以上，以满足广泛的边缘部署场景，从而减少了几个数量级的手动工作。
AI applications powered by deep learning inference are increasingly run natively on edge devices to provide better interactive user experience. This often necessitates fitting a model originally designed and trained in the cloud to edge devices with a range of hardware capabilities, which so far has relied on time-consuming manual effort. In this paper, we quantify the challenges of manually generating a large number of compressed models and then build a system framework, Mistify , to automatically port a cloud-based model to a suite of models for edge devices targeting various points in the design space. Mistify adds an intermediate “layer” that decouples the model design and deployment phases. By exposing configuration APIs to obviate the need for code changes deeply embedded into the original model, Mistify hides run-time issues from model designers and hides the model internals from model users, hence reducing the expertise needed in either. For better scalability, Mistify consoli-dates multiple model tailoring requests to minimize repeated computation. Further, Mistify leverages locally available edge data in a privacy-aware manner, and performs run-time model adaptation to provide scalable edge support and accurate inference results. Extensive evaluation shows that Mistify reduces the DNN porting time needed by over 10 × to cater to a wide spectrum of edge deployment scenarios, incurring orders of magnitude less manual effort.