Application-Level Optimization of Big Data Transfers through Pipelining, Parallelism and Concurrency
Application-Level Optimization of Big Data Transfers through Pipelining, Parallelism and Concurrency
复制标题
DOI：
10.1109/tcc.2015.2415804
复制
发表时间：
2016
影响因子：
6.5
通讯作者：
E. Yildirim;Engin Arslan;Jangyoung Kim;T. Kosar
中科院分区：
文献类型：
--
作者：
E. Yildirim;Engin Arslan;Jangyoung Kim;T. Kosar
在端到端数据传输中，存在影响数据传输吞吐量的若干因素，诸如网络特性（例如，网络带宽、往返时间、后台业务）;端系统特性（例如，NIC容量、CPU内核数量及其时钟速率、磁盘驱动器数量及其I/O速率）;以及数据集特征（例如，平均文件大小、数据集大小、文件大小分布）。通过云间和云内网络优化大数据传输是一项具有挑战性的任务，需要联合考虑所有这些参数。当传输由异构文件大小组成的数据集（即，大文件和小文件混合）。以前在这方面的工作只集中在终端系统和网络的特点，但没有提供关于数据集的特点模型。在这项研究中，我们分析了三个最重要的传输参数，用于提高数据传输吞吐量的影响：流水线，并行性和并发性。我们提供的模型和准则，以设置这些参数的最佳值，并提出了两种不同的传输优化算法，使用开发的模型。在高速网络和云测试平台上进行的测试表明，在大多数情况下，我们的算法优于最流行的数据传输工具，如Globus Online和UDT。
In end-to-end data transfers, there are several factors affecting the data transfer throughput, such as the network characteristics (e.g., network bandwidth, round-trip-time, background traffic); end-system characteristics (e.g., NIC capacity, number of CPU cores and their clock rate, number of disk drives and their I/O rate); and the dataset characteristics (e.g., average file size, dataset size, file size distribution). Optimization of big data transfers over inter-cloud and intra-cloud networks is a challenging task that requires joint-consideration of all of these parameters. This optimization task becomes even more challenging when transferring datasets comprised of heterogeneous file sizes (i.e., large files and small files mixed). Previous work in this area only focuses on the end-system and network characteristics however does not provide models regarding the dataset characteristics. In this study, we analyze the effects of the three most important transfer parameters that are used to enhance data transfer throughput: pipelining,parallelism and concurrency. We provide models and guidelines to set the best values for these parameters and present two different transfer optimization algorithms that use the models developed. The tests conducted over high-speed networking and cloud testbeds show that our algorithms outperform the most popular data transfer tools like Globus Online and UDT in majority of the cases.