Data Transfer between Scientific Facilities – Bottleneck Analysis, Insights and Optimizations
Data Transfer between Scientific Facilities – Bottleneck Analysis, Insights and Optimizations
复制标题
科学设施之间的数据传输 – 瓶颈分析、见解和优化
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Ian T Foster
中科院分区：
文献类型：
--
作者：
Yuanlai Liu;Zhengchun Liu;R. Kettimuthu;N. Rao;Zizhong Chen;Ian T Foster
广域文件传输在许多科学应用中起着重要的作用。文件传输工具通常为具有少量大文件的数据集提供最高性能，但许多科学数据集由许多小文件组成。因此，重要的是要了解导致具有许多小文件的数据集的广域数据传输性能下降的因素。为此，我们（i）对两个HPC设施之间的端到端文件传输所涉及的子系统的性能进行基准测试，用于代表生产科学传输的多文件数据集;（ii）表征由不同子系统引入的每个文件开销;（iii）识别潜在的依赖关系和瓶颈;（iv）研究同时转移多份档案是否有效，以减少每份档案的间接费用;以及（v）原型化预取机制作为并发的替代方案以减少源存储系统上的每个文件开销。我们表明，并发和预取可以帮助减少每个文件的开销显着。合理水平的并发性与预取相结合可以将每个文件的开销降低到可以忽略不计的水平。
Wide area file transfers play an important role in many science applications. File transfer tools typically deliver the highest performance for datasets with a small number of large files, but many science datasets consist of many small files. Thus it is important to understand the factors that contribute to the decrease in wide area data transfer performance for datasets with many small files. To this end, we (i) benchmark the performance of subsystems involved in end-to-end file transfer between two HPC facilities for a many-file dataset that is representative of production science transfers; (ii) characterize the per-file overhead introduced by different subsystems; (iii) identify potential dependencies and bottlenecks; (iv) study the effectiveness of transferring many files concurrently as a means of reducing per-file overheads; and (v) prototype a prefetching mechanism as an alternative of concurrency to reduce the per-file overhead on source storage system. We show that both concurrency and prefetching can help reduce the per-file overhead significantly. A reasonable level of concurrency combined with prefetching can bring the per-file overhead down to a negligible level.