Adaptive and Resilient Model-Distributed Inference in Edge Computing Systems
Adaptive and Resilient Model-Distributed Inference in Edge Computing Systems
复制标题
DOI：
10.1109/ojcoms.2023.3280174
复制
发表时间：
2023
影响因子：
7.9
通讯作者：
Pengzhen Li;Erdem Koyuncu;H. Seferoglu
中科院分区：
文献类型：
--
作者：
Pengzhen Li;Erdem Koyuncu;H. Seferoglu
边缘计算系统中分布式深度神经网络（DNN）推理的传统方法是数据分布式推理。在此范例中，每个工作人员都有一个预先训练的 DNN 模型。使用 DNN 模型，工作器处理卸载到自身的数据。数据分布式推理方法（i）具有很高的通信成本，特别是当数据量很大时，并且（ii）在内存方面效率不高，因为整个模型应该在每个工作器中存储和计算。模型分布式推理正在成为一种有前途的解决方案，其中 DNN 模型分布在各个工作人员之间。尽管在模型分布式训练方面有大量的工作，但模型分布对于推理的好处还没有得到很好的理解。在本文中，我们分析了模型分布式推理在边缘计算系统中的潜力。然后，我们基于最佳模型分配公式开发了自适应和弹性模型分布式推理 (AR-MDI) 算法。 AR-MDI 以轻量级和去中心化的方式执行模型分配，并且能够抵御延迟的工作人员和故障。我们在由 NVIDIA Jetson TX2 组成的真实测试台中实施 AR-MDI，结果表明，当数据量较大（例如 ImageNet）时，与基线相比，AR-MDI 显着缩短了推理时间。
The traditional approach to distributed deep neural network (DNN) inference in edge computing systems is data-distributed inference. In this paradigm, each worker has a pre-trained DNN model. Using the DNN model, the worker processes the data that is offloaded to itself. The data-distributed inference approach (i) has high communication cost especially when the size of data is large, and (ii) is not efficient in terms of memory as the whole model should be stored and computed in each worker. Model-distributed inference is emerging as a promising solution, where a DNN model is distributed across workers. Although there is a huge amount of work on model-distributed training, the benefit of model distribution for inference is not understood well. In this paper, we analyze the potential of model-distributed inference in edge computing systems. Then, we develop an Adaptive and Resilient Model-Distributed Inference (AR-MDI) algorithm based on our optimal model allocation formulation. AR-MDI performs model allocation in a lightweight and decentralized way and it is resilient against delayed workers and failures. We implement AR-MDI in a real testbed consisting of NVIDIA Jetson TX2s and show that AR-MDI improves the inference time significantly as compared to baselines when the size of data is large, such as ImageNet.