Use Only What You Need: Judicious Parallelism For File Transfers in High Performance Networks
Use Only What You Need: Judicious Parallelism For File Transfers in High Performance Networks
复制标题
DOI：
10.1145/3577193.3593722
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Md. Arifuzzaman;Engin Arslan
中科院分区：
文献类型：
--
作者：
Md. Arifuzzaman;Engin Arslan
并行性是传输大量数据时有效利用高速研究网络的关键。然而，现有传输应用程序的整体设计要求文件传输的读、写和网络操作具有相同级别的并行性。这反过来又会加重系统资源的负担，因为为最慢的组件设置并行级别会导致其他组件不必要的高并行度。使用超过必要的并行性会导致系统资源开销增加以及竞争传输之间的不公平资源分配。在本文中，我们引入模块化文件传输架构Marlin，将文件传输的I/O和网络操作分开，以便可以独立调整每个组件的并行度。 Marlin 采用在线梯度下降算法快速搜索解空间并找到读取、传输和写入操作的最佳并行度。在各种网络设置下收集的实验结果表明，Marlin 可以识别并使用每个组件的最小并行级别，从而提高竞争传输和 CPU 利用率之间的公平性。最后，将网络传输与写入操作分开，使得 Marlin 在传输小型数据集时的性能比最先进的解决方案高出 2 倍以上。
Parallelism is key to efficiently utilizing high-speed research networks when transferring large volumes of data. However, the monolithic design of existing transfer applications requires the same level of parallelism to be used for read, write, and network operations for file transfers. This, in turn, overburdens system resources since setting the parallelism level for the slowest component results in unnecessarily high parallelism for other components. Using more than necessary parallelism lead to increased overhead on system resources and unfair resource allocation among competing transfers. In this paper, we introduce modular file transfer architecture, Marlin, to separate I/O and network operations for file transfers so that parallelism can be independently adjusted for each component. Marlin adopts online gradient descent algorithm to swiftly search the solution space and find the optimal level of parallelism for read, transfer, and write operations. Experimental results collected under various network settings show that Marlin can identify and use a minimum parallelism level for each component, improving fairness among competing transfers and CPU utilization. Finally, separating network transfers from write operations allows Marlin to outperform the state-of-the-art solutions by more than 2x when transferring small datasets.