SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
复制标题
DOI：
10.1145/3373376.3378530
复制
发表时间：
2020-03
期刊：
影响因子：
--
通讯作者：
Chien-chin Huang;Gu Jin;Jinyang Li
中科院分区：
文献类型：
--
作者：
Chien-chin Huang;Gu Jin;Jinyang Li
众所周知，更深层和更广泛的神经网络可以实现更好的准确性。但是，由于GPU内存有限，很难继续增加趋势以增加模型大小。一种有希望的解决方案是支持在GPU和CPU内存之间交换。但是，现有关于交换的工作仅处理某些型号，并且无法实现令人满意的性能。深度学习计算通常表示为数据流图，可以分析以改善交换。我们提出了SwapAdvisor，该Swapadvisor基于给定的数据流图沿3个维度执行关节优化：操作员调度，内存分配和交换决策。 Swapadvisor使用定制设计的遗传算法探索了广阔的搜索空间。使用各种大型模型进行评估表明，Swapadvisor可以训练模型的12倍GPU内存限制，同时使用无限GPU存储器实现假设基线的吞吐量的53-99％。
It is known that deeper and wider neural networks can achieve better accuracy. But it is difficult to continue the trend to increase model size due to limited GPU memory. One promising solution is to support swapping between GPU and CPU memory. However, existing work on swapping only handle certain models and do not achieve satisfactory performance. Deep learning computation is commonly expressed as a dataflow graph which can be analyzed to improve swapping. We propose SwapAdvisor, which performs joint optimization along 3 dimensions based on a given dataflow graph: operator scheduling, memory allocation, and swap decisions. SwapAdvisor explores the vast search space using a custom-designed genetic algorithm. Evaluations using a variety of large models show that SwapAdvisor can train models up to 12 times the GPU memory limit while achieving 53-99% of the throughput of a hypothetical baseline with infinite GPU memory.