NeSSA: Near-Storage Data Selection for Accelerated Machine Learning Training
NeSSA: Near-Storage Data Selection for Accelerated Machine Learning Training
复制标题
DOI：
10.1145/3599691.3603404
复制
发表时间：
2023-07
期刊：
影响因子：
--
通讯作者：
Neha Prakriya;Yu Yang;Baharan Mirzasoleiman;Cho-Jui Hsieh;J. Cong
中科院分区：
文献类型：
--
作者：
Neha Prakriya;Yu Yang;Baharan Mirzasoleiman;Cho-Jui Hsieh;J. Cong
大规模机器学习（ML）模型依赖于非常大的数据集来学习其呈指数增长的参数数量。虽然这些模型取得了前所未有的成功，但所需的训练时间和硬件资源的增加是不可持续的。此外，我们发现随着数据集大小的增加，数据移动成为整体训练时间的重要组成部分。我们提出了NeSSA，一种新型的SmartSSD+GPU训练架构，可以智能地选择存储附近的大型数据集的重要子集，这样子集上的训练就可以模仿完整数据集上的训练，并且精度损失非常小。据我们所知，这是第一次提出这种近存储数据选择模型来进行有效的ML训练。我们已经对CIFAR-10，SVHN，CINIC-10，CIFAR-100，TinyImageNet和ImageNet-100数据集进行了评估。我们还测试了ResNet-20、ResNet-18和ResNet-50模型。
Large-scale machine learning (ML) models rely on extremely large datasets to learn their exponentially growing number of parameters. While these models achieve unprecedented success, the increase in training time and hardware resources required is unsustainable. Further, we find that as dataset sizes increase, data movement becomes a significant component of overall training time. We propose NeSSA, a novel SmartSSD+GPU training architecture to intelligently select important subsets of large datasets near-storage, such that training on the subset mimics training on the full dataset with a very small loss in accuracy. To the best of our knowledge, this is the first work to propose such a near-storage data selection model for efficient ML training. We have evaluated our method for the CIFAR-10, SVHN, CINIC-10, CIFAR-100, TinyImageNet, and ImageNet-100 datasets. We also test across ResNet-20, ResNet-18, and ResNet-50 models.