Reliable Wide-Area Data Transfers for Streaming Workflows
Reliable Wide-Area Data Transfers for Streaming Workflows
复制标题
DOI：
10.1109/tpds.2022.3158673
复制
发表时间：
2022
影响因子：
5.3
通讯作者：
Hemanta Sapkota;Engin Arslan
中科院分区：
文献类型：
--
作者：
Hemanta Sapkota;Engin Arslan
许多大型科学项目依赖远程集群进行（近）实时数据处理，因此它们需要可靠的广域数据传输性能以实现平滑的端到端工作流程执行。然而，由于网络（例如后台流量）和数据集（例如平均文件大小）条件的变化，数据传输经常会受到性能变化的影响，因此需要自适应解决方案来满足延迟敏感的流媒体工作流程的严格性能要求。在本文中，我们提出 FStream++ 通过动态调整传输设置以适应不断变化的传输条件，为大型流科学应用提供可靠的传输性能。 FStream++结合动态调优、在线分析、历史分析三种优化方法，快速准确地发现满足工作流程要求的最佳传输设置。动态调整使用启发式模型根据数据集特征和网络设置来预测传输参数的值。由于启发式模型无法纳入许多重要因素，例如 I/O 吞吐量和资源干扰，因此我们通过在线分析对其进行补充，以对传输设置的子集执行实时搜索。最后，历史分析通过存储和分析以前的性能观察结果来利用流工作流的长期运行特性，以缩短在线分析的执行时间。我们通过在高性能生产网络中传输多个合成的和真实的工作负载来评估 FStream++ 的性能，结果表明，与传统传输应用程序相比，它的性能提升高达 3.6 倍，比我们之前的 FStream 工作提升高达 24%。
Many large science projects rely on remote clusters for (near) real-time data processing, thus they demand reliable wide-area data transfer performance for smooth end-to-end workflow executions. However, data transfers are often exposed to performance variations due to the changing network (e.g., background traffic) and dataset (e.g., average file size) conditions, necessitating adaptive solutions to meet stringent performance requirements of delay-sensitive streaming workflows. In this paper, we propose FStream++ to provide reliable transfer performance for large streaming science applications by dynamically adjusting transfer settings to adapt to changing transfer conditions. FStream++ combines three optimization methods as dynamic tuning, online profiling, and historical analysis to swiftly and accurately discover optimal transfer settings that can meet workflow requirements. Dynamic tuning uses a heuristic model to predict the values of transfer parameters based on dataset characteristics and network settings. Since heuristic models fall short to incorporate many important factors such as I/O throughput and resource interference, we complement it with online profiling to execute a real-time search for a subset of transfer settings. Finally, historical analysis takes advantage of the long-running nature of streaming workflows by storing and analyzing previous performance observations to shorten the execution time of online profiling. We evaluate the performance of FStream++ by transferring several synthetic and real-world workloads in high-performance production networks and show that it offers up to 3.6x performance improvement over legacy transfer applications and up to 24% over our previous work FStream.