Real-Time Video Inference on Edge Devices via Adaptive Model Streaming
Real-Time Video Inference on Edge Devices via Adaptive Model Streaming
复制标题
DOI：
10.1109/iccv48922.2021.00453
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Mehrdad Khani Shirkoohi;Pouya Hamadanian;Arash Nasr-Esfahany;Mohammad Alizadeh
中科院分区：
文献类型：
--
作者：
Mehrdad Khani Shirkoohi;Pouya Hamadanian;Arash Nasr-Esfahany;Mohammad Alizadeh
由于深度神经网络的高计算成本，在移动的手机和无人机等边缘设备上进行实时视频推理具有挑战性。我们提出了自适应模型流（AMS），这是一种提高边缘设备上视频推理的高效轻量级模型性能的新方法。AMS使用远程服务器来不断训练和调整在边缘设备上运行的小型模型，使用来自大型最先进模型的在线知识蒸馏来提高其在实时视频上的性能。我们讨论了视频推理的网络模型自适应的挑战，并提出了几种技术来降低这种方法的通信成本：避免过度拟合，更新一小部分重要的模型参数，以及在边缘设备上对训练帧进行自适应采样。在视频语义分割任务中，我们的实验结果显示，与多个视频数据集的预训练模型相比，Intersection-over-Union平均提高了0.4- 17.8%。我们的原型可以在三星Galaxy S10+移动的手机上以每秒30帧的速度执行视频分割，相机到标签延迟为40毫秒，设备上的上行链路和下行链路带宽小于300 Kbps。
Real-time video inference on edge devices like mobile phones and drones is challenging due to the high computation cost of Deep Neural Networks. We present Adaptive Model Streaming (AMS), a new approach to improving the performance of efficient lightweight models for video inference on edge devices. AMS uses a remote server to continually train and adapt a small model running on the edge device, boosting its performance on the live video using online knowledge distillation from a large, state-of-the-art model. We discuss the challenges of over-the-network model adaptation for video inference and present several techniques to reduce communication the cost of this approach: avoiding excessive overfitting, updating a small fraction of important model parameters, and adaptive sampling of training frames at edge devices. On the task of video semantic segmentation, our experimental results show 0.4–17.8 percent mean Intersection-over-Union improvement compared to a pretrained model across several video datasets. Our prototype can perform video segmentation at 30 frames-per-second with 40 milliseconds camera-to-label latency on a Samsung Galaxy S10+ mobile phone, using less than 300 Kbps uplink and downlink bandwidth on the device.