Dystri: A Dynamic Inference based Distributed DNN Service Framework on Edge
Dystri: A Dynamic Inference based Distributed DNN Service Framework on Edge
复制标题
DOI：
10.1145/3605573.3605598
复制
发表时间：
2023-08
期刊：
影响因子：
--
通讯作者：
Xueyu Hou;Yongjie Guan;Tao Han
中科院分区：
文献类型：
--
作者：
Xueyu Hou;Yongjie Guan;Tao Han
由于高请求强度、并发多用户场景和各种异构服务类型，深度神经网络（DNN）推理在服务计算请求方面提出了独特的挑战。同时，移动的和边缘设备为用户提供了增强的计算能力，使他们能够利用本地资源进行深度推理处理。此外，动态推理技术允许基于内容的计算成本选择每个请求。本文介绍了Dystri，一个创新的框架，旨在促进分布式边缘基础设施的动态推理，从而容纳多个异构用户。Dystri在实际环境中提供了广泛的适用性，包括异构设备类型，基于DNN的应用程序和动态推理技术，超越了最先进的（SOTA）方法。通过分布式控制器和全局协调器，Dystri允许按请求、按用户调整服务质量，确保即时、灵活和离散的控制。Dystri中的解耦工作流自然支持用户异构性和可扩展性，解决了现有SOTA工作忽略的关键方面。我们的评估涉及三个部署在分布式边缘基础设施上的多用户、异构DNN推理服务平台，包括七个DNN应用程序。结果表明，Dystri实现了接近零的最后期限错过，并擅长适应不同的用户数量和请求强度。Dystri优于基线，精度提高高达95倍。
Deep neural network (DNN) inference poses unique challenges in serving computational requests due to high request intensity, concurrent multi-user scenarios, and diverse heterogeneous service types. Simultaneously, mobile and edge devices provide users with enhanced computational capabilities, enabling them to utilize local resources for deep inference processing. Moreover, dynamic inference techniques allow content-based computational cost selection per request. This paper presents Dystri, an innovative framework devised to facilitate dynamic inference on distributed edge infrastructure, thereby accommodating multiple heterogeneous users. Dystri offers a broad applicability in practical environments, encompassing heterogeneous device types, DNN-based applications, and dynamic inference techniques, surpassing the state-of-the-art (SOTA) approaches. With distributed controllers and a global coordinator, Dystri allows per-request, per-user adjustments of quality-of-service, ensuring instantaneous, flexible, and discrete control. The decoupled workflows in Dystri naturally support user heterogeneity and scalability, addressing crucial aspects overlooked by existing SOTA works. Our evaluation involves three multi-user, heterogeneous DNN inference service platforms deployed on distributed edge infrastructure, encompassing seven DNN applications. Results show Dystri achieves near-zero deadline misses and excels in adapting to varying user numbers and request intensities. Dystri outperforms baselines with accuracy improvement up to 95 ×.