SHADE: Enable Fundamental Cacheability for Distributed Deep Learning Training
SHADE: Enable Fundamental Cacheability for Distributed Deep Learning Training
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Redwan Ibne Seraj Khan;Ahmad Hossein Yazdani;Yuqi Fu;Arnab K. Paul;Bo Ji;Xun Jian;Yue Cheng;A. R. Butt
中科院分区：
文献类型：
--
作者：
Redwan Ibne Seraj Khan;Ahmad Hossein Yazdani;Yuqi Fu;Arnab K. Paul;Bo Ji;Xun Jian;Yue Cheng;A. R. Butt
深度学习培训（DLT）应用程序对储存系统设计构成新的挑战是I/O的独特I/O工作负载。为了支持这些应用程序。分布式DLT。重要的和不同的数据样本有助于提高模型的准确性。利用重要性采样来利用数据局部性。 Shade采用了一种基于等级的方法，该方法捕获了不同的小型阴影的数据样本的相对重要性。通过这些技术，所有样本都可以显着改善DLT作业的命中率，因此，使用代表性的计算机愿景（CV）改善了培训性能。与LRU缓存策略相比，将缓存命中率提高了4.5倍。
Deep learning training (DLT) applications exhibit unique I/O workload behaviors that pose new challenges for storage system design. DLT is I/O intensive since data samples need to be fetched continuously from a remote storage. Accelerators such as GPUs have been extensively used to support these applications. As accelerators become more powerful and more data-hungry, the I/O performance lags behind. This creates a crucial performance bottleneck, especially in distributed DLT. At the same time, the exponentially growing dataset sizes make it impossible to store these datasets entirely in memory. While today’s DLT frameworks typically use a random sampling policy that treat all samples uniformly equally, recent findings indicate that not all samples are equally important and different data samples contribute differently towards improving the accuracy of a model. This observation creates an opportunity for DLT I/O optimizations by exploiting the data locality enabled by importance sampling. To this end, we design and implement SHADE, a new DLTaware caching system that detects fine-grained importance variations at per-sample level and leverages the variance to make informed caching decisions for a distributed DLT job. SHADE adopts a novel, rank-based approach, which captures the relative importance of data samples across different minibatches. SHADE then dynamically updates the importance scores of all samples during training. With these techniques, SHADE manages to significantly improve the cache hit ratio of the DLT job, and thus, improves the job’s training performance. Evaluation with representative computer vision (CV) models shows that SHADE, with a small cache, improves the cache hit ratio by up to 4.5× compared to the LRU caching policy.