Predicting Inference Latency of Neural Architectures on Mobile Devices
Predicting Inference Latency of Neural Architectures on Mobile Devices
复制标题
DOI：
10.1145/3578244.3583735
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Zhuojin Li;Marco Paolieri;L. Golubchik
中科院分区：
文献类型：
--
作者：
Zhuojin Li;Marco Paolieri;L. Golubchik
由于移动设备上的推理任务扩散，最新的神经体系结构通常是使用神经体系结构搜索（NAS）设计的，以在机器学习准确性和推理潜伏期之间实现良好的权衡。尽管在NAS期间测量大量候选架构的推理潜伏期是不可行的，但由于硬件异质性，机器学习框架应用的优化以及神经体系结构的多样性，移动设备的延迟预测是具有挑战性的。在这些挑战中，我们首先定量评估对推理潜伏期有重大影响的神经体系结构和移动设备的特征。基于此评估，我们提出了一个按操作的框架，该框架通过开发操作的延迟预测因子来解决这些挑战，并在端到端的潜伏期预测中实现了很高的准确性，如我们使用Multicore CPU和多个移动设备对多个移动设备的全面评估所示GPU。为了说明我们的方法不需要昂贵的数据收集，我们还表明，仅使用少量分析数据就可以在现实世界的神经体系结构上实现准确的预测。
Due to the proliferation of inference tasks on mobile devices, state-of-the-art neural architectures are typically designed using Neural Architecture Search (NAS) to achieve good tradeoffs between machine learning accuracy and inference latency. While measuring inference latency of a huge set of candidate architectures during NAS is not feasible, latency prediction for mobile devices is challenging, because of hardware heterogeneity, optimizations applied by machine learning frameworks, and diversity of neural architectures. Motivated by these challenges, we first quantitatively assess the characteristics of neural architectures and mobile devices that have significant effects on inference latency. Based on this assessment, we propose an operation-wise framework which addresses these challenges by developing operation-wise latency predictors and achieves high accuracy in end-to-end latency predictions, as shown by our comprehensive evaluations on multiple mobile devices using multicore CPUs and GPUs. To illustrate that our approach does not require expensive data collection, we also show that accurate predictions can be achieved on real-world neural architectures using only small amounts of profiling data.