HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder
HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder
复制标题
DOI：
10.1109/cluster48925.2021.00064
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Jaime Cernuda Garcia;H. Devarajan;Luke Logan;Keith Bateman;N. Rajesh;Jie Ye;Anthony Kougkas;Xian-He Sun
中科院分区：
文献类型：
--
作者：
Jaime Cernuda Garcia;H. Devarajan;Luke Logan;Keith Bateman;N. Rajesh;Jie Ye;Anthony Kougkas;Xian-He Sun
现代应用程序是高度数据密集型的，导致了众所周知的I/O瓶颈问题。科学家们提出了快速中间存储资源的放置，旨在掩盖I/O损失。为了管理这些资源，领导级计算设施中使用了三种核心软件抽象：IO转发器、突发缓冲器和数据分段器。然而，随着HPC系统中多租户部署的兴起，这些软件抽象是：隔离管理和维护，导致低效的交互;静态分配，导致负载不平衡;在中间存储之间完全分叉，导致资源利用不足，并且在许多情况下，不支持原位操作。为此，我们提出了HFlow，一类新的数据转发系统，利用实时数据移动的范例。HFlow引入了统一的数据移动抽象（ByteFlow），提供了可以在任何地方执行的数据独立任务，从而实现了动态资源配置。此外，执行字节流的处理元件被设计为是短暂的，并且因此使得能够对中间存储资源进行弹性管理。我们的研究结果表明，与最先进的软件解决方案相比，在HFlow下运行的应用程序显示出3倍的性能提升。
Modern applications are highly data-intensive, leading to the well-known I/O bottleneck problem. Scientists have proposed the placement of fast intermediate storage resources which aim to mask the I/O penalties. To manage these resources, three core software abstractions are being used in leadership-class computing facilities: IO Forwarders, Burst Buffers, and Data Stagers. Yet, with the rise of multi-tenant deployment in HPC systems, these software abstractions are: managed and maintained in isolation, leading to inefficient interactions; allocated statically, leading to load imbalance; exclusively bifurcated between the intermediate storage, leading to under-utilization of resources, and, in many cases, do not support in-situ operations. To this end, we present HFlow, a new class of data forwarding system that leverages a real-time data movement paradigm. HFlow introduces a unified data movement abstraction (the ByteFlow) providing data-independent tasks that can be executed anywhere and thus, enabling dynamic resource provisioning. Moreover, the processing elements executing the ByteFlows are designed to be ephemeral and, hence, enable elastic management of intermediate storage resources. Our results show that applications running under HFlow display an increase in performance of 3x when compared with state-of-the-art software solutions.