Falcon: Fair and Efficient Online File Transfer Optimization
Falcon: Fair and Efficient Online File Transfer Optimization
复制标题
DOI：
10.1109/tpds.2023.3282872
复制
发表时间：
2023-08
影响因子：
5.3
通讯作者：
Md. Arifuzzaman;B. Bockelman;James Basney;Engin Arslan
中科院分区：
文献类型：
--
作者：
Md. Arifuzzaman;B. Bockelman;James Basney;Engin Arslan
研究网络在研究机构和教育机构之间提供高速广域网络连接，以促进大规模数据传输。然而，SCP和FTP等传统传输应用的可扩展性问题阻碍了这些网络的有效利用。尽管研究人员通过利用I/O和网络并行来扩展传统传输应用以提高其性能，但这些解决方案要求用户微调并行级别，由于网络的动态性质，即使对于经验丰富的用户来说，这也是一项具有挑战性的任务。在本文中，我们提出了一个在线优化算法Falcon，用于调整文件传输的并行度，以最大化传输吞吐量，同时将系统开销保持在最小。由于研究网络是共享的基础设施，我们引入了一个博弈论启发的新型效用函数来评估各种并行级别的性能，从而保证竞争传输收敛到一个公平和稳定的解决方案。我们评估了Falcon在隔离和生产型高速网络中的性能，发现它可以在短短20秒内发现最优的传输并行性，并比最先进的解决方案高出2美元\x$2倍以上。此外，在博弈论启发的效用函数的帮助下，当多个转移竞争相同的资源时，猎鹰保证收敛到纳什均衡。最后，我们证明了Falcon还可以用作中央传输调度器，以加快收敛时间，增加稳定性，并在共享网络中强制实施系统/用户级资源限制。
Research networks provide high-speed wide-area network connectivity between research and education institutions to facilitate large-scale data transfers. However, scalability issues of legacy transfer applications such as scp and FTP hinder the effective utilization of these networks. Although researchers extended the legacy transfer applications to increase their performance by exploiting I/O and network parallelism, these solutions necessitate users to fine-tune parallelism level, a task that is challenging even for experienced users due to the dynamic nature of networks. In this article, we propose an online optimization algorithm, Falcon, to tune the degree of parallelism for file transfers to maximize transfer throughput while keeping system overhead at a minimum. As research networks are shared infrastructures, we introduce a game theory-inspired novel utility function to evaluate the performance of various parallelism levels such that competing transfers are guaranteed to converge to a fair and stable solution. We assessed the performance of Falcon in isolated and production high-speed networks and found that it can discover optimal transfer parallelism in as little as 20 seconds and outperform the state-of-the-art solutions by more than $2\times$2×. Moreover, Falcon is guaranteed to converge to Nash Equilibrium when multiple transfers compete for the same resources with the help of its game theory-inspired utility function. Finally, we demonstrate that Falcon can also be used as a central transfer scheduler to speed up convergence time, increase stability, and enforce system/user-level resource limitations in shared networks.