Machine learning driven system level heterogeneous memory management for high-performance computing

用于高性能计算的机器学习驱动的系统级异构内存管理

• 批准号: 19K11993
• 负责人: GEROFI BALAZS
• 金额: $ 2.75万
• 依托单位: Institute of Physical and Chemical Research
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (C)
• 财政年份: 2019
• 资助国家: 日本
• 起止时间: 2019-04-01 至 2023-03-31
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-19K11993/
• 关键词: Memory access tracing; Runtime approximation; Distributed ML; Neural network training; I/O of deep learning; Distributed learning; Memory access tracking; heterogeneous memory; gem5; architectural simulator; non-uniform memory; machine learning

Results have been achieved in two parallel efforts of the project.We found that system-software-level heterogeneous memory management solutions utilizing machine learning, in particular nonsupervised learning- based methods such as reinforcement learning, require rapid estimation of execution runtime as a function of the data layout across memory devices for exploring different data placement strategies, which renders architecture-level simulators impractical for this purpose. We proposed a differential tracing-based approach using memory access traces obtained by high-frequency sampling-based methods (e.g., Intel's PEBS) on real hardware using of different memory devices. We developed a runtime estimator based on such traces that provides an execution time estimate orders of magnitude faster than full-system simulators. On a number of HPC mini applications we showed that the estimator predicts runtime with an average error of 4.4% compared to measurements on real hardware.For the deep learning data shuffling subtopic, we investigated the viability of partitioning the dataset among DL workers and performing only a partial distributed exchange of samples in each training epoch. Through extensive experiments on up to 2048 GPUs of ABCI and 4096 compute nodes of Fugaku, we demonstrated that in practice validation accuracy of global shuffling can be maintained when carefully tuning the partial distributed exchange. We provided an implementation in PyTorch that enables users to control the proposed data exchange scheme.

我们发现，利用机器学习的系统软件级异质内存管理解决方案，特别是基于强化学习的非监督学习方法，需要根据存储设备上的数据布局快速估计执行时间，以探索不同的数据放置策略，这使得架构级仿真器不适用于此目的。我们提出了一种基于差分跟踪的方法，使用基于高频采样的方法(如Intel的PEB)在使用不同存储设备的实际硬件上获得内存访问轨迹。我们开发了一个基于这样的跟踪的运行时估计器，它提供的执行时间估计比全系统模拟器快几个数量级。在一些HPC迷你应用程序上，我们表明该估计器预测运行时间的平均误差为4.4%，与在真实硬件上的测量结果相比。对于深度学习数据洗牌子主题，我们研究了在每个训练周期中在DL工作者之间划分数据集并仅执行部分分布式样本交换的可行性。通过在高达2048个ABCI的GPU和4096个Fugaku的计算节点上的广泛实验，我们证明了在实践中，当仔细调整局部分布式交换时，全局洗牌的验证精度可以保持不变。我们在PyTorch中提供了一个实现，使用户能够控制建议的数据交换方案。

项目成果

Why Globally Re-shuffle? Revisiting Data Shuffling in Large Scale Deep Learning

• DOI: 10.1109/ipdps53621.2022.00109
• 发表时间: 2022-05
• 期刊: 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
• 作者: Thao Nguyen;François Trahay;Jens Domke;Aleksandr Drozd;Emil;Vatai;Jianwei Liao;M. Wahib

Directions for Operating Systems Research

操作系统研究方向

• 发表时间: 2021
• 作者: Fajardo-Diaz Juan L.;Morelos-Gomez Aaron;Cruz-Silva Rodolfo;Matsumoto Akito;Ueno Yutaka;Takeuchi Norihiro;Kitamura Kotaro;Miyakawa Hiroki;Tejima Syogo;Takeuchi Kenji;Tsuzuki Koichi;Endo Morinobu;田中 紘生，木原 尚，安倍 賢一;Balazs Gerofi
• 通讯作者: Balazs Gerofi

2020 SIAM Conference on Parallel Processing for Scientific Computing

2020 SIAM 科学计算并行处理会议

• 发表时间: 2020
• 作者: 宮地英生;川原慎太郎;Balazs Gerofi
• 通讯作者: Balazs Gerofi

Towards Intelligent Management of Heterogeneous Memory: A Reinforcement Learning Approach

走向异构内存的智能管理：强化学习方法

• 发表时间: 2019
• 作者: 宮地英生;川原慎太郎;廣渡 祥太，木原 尚，安倍 賢一;Balazs Gerofi
• 通讯作者: Balazs Gerofi

Argonne National Laboratory(米国)

阿贡国家实验室（美国）