ED-Batch: Efficient Automatic Batching of Dynamic Neural Networks via Learned Finite State Machines
ED-Batch: Efficient Automatic Batching of Dynamic Neural Networks via Learned Finite State Machines
复制标题
DOI：
10.48550/arxiv.2302.03851
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
Siyuan Chen-;Pratik Fegade;Tianqi Chen;Phillip B. Gibbons;T. Mowry
中科院分区：
文献类型：
--
作者：
Siyuan Chen-;Pratik Fegade;Tianqi Chen;Phillip B. Gibbons;T. Mowry
批处理对深度神经网络（DNN）的执行效率有着根本性的影响。然而，对于动态DNN，有效的递归是特别具有挑战性的，因为递归图根据输入实例而变化。因此，最先进的框架使用导致次优决策的策略。此外，DRAM对内存邻接有严格的限制，并可能导致高数据移动成本。在本文中，我们提供了一种基于有限状态机的动态DNN的方法，该方法可以通过强化学习自动发现专门用于每个DNN的策略。此外，我们发现，内存规划，知道的策略可以节省显着的数据移动开销，这是自动化的PQ树为基础的算法，我们介绍。实验结果表明，我们的框架在CPU和GPU上对基于链、基于树和基于网格的DNN的平均速度分别提高了1.15倍、1.39倍和2.45倍。
Batching has a fundamental influence on the efficiency of deep neural network (DNN) execution. However, for dynamic DNNs, efficient batching is particularly challenging as the dataflow graph varies per input instance. As a result, state-of-the-art frameworks use heuristics that result in suboptimal batching decisions. Further, batching puts strict restrictions on memory adjacency and can lead to high data movement costs. In this paper, we provide an approach for batching dynamic DNNs based on finite state machines, which enables the automatic discovery of batching policies specialized for each DNN via reinforcement learning. Moreover, we find that memory planning that is aware of the batching policy can save significant data movement overheads, which is automated by a PQ tree-based algorithm we introduce. Experimental results show that our framework speeds up state-of-the-art frameworks by on average 1.15x, 1.39x, and 2.45x for chain-based, tree-based, and lattice-based DNNs across CPU and GPU.