Big data transfer optimization based on offline knowledge discovery and adaptive sampling
Big data transfer optimization based on offline knowledge discovery and adaptive sampling
复制标题
DOI：
10.1109/bigdata.2017.8257959
复制
发表时间：
2017-12
期刊：
影响因子：
--
通讯作者：
M. S. Q. Z. Nine;Kemal Guner;Ziyun Huang;Xiangyu Wang;Jinhui Xu;T. Kosar
中科院分区：
文献类型：
--
作者：
M. S. Q. Z. Nine;Kemal Guner;Ziyun Huang;Xiangyu Wang;Jinhui Xu;T. Kosar
超过专用和非专用网络链接的数据量的增加速度要比网络容量增加的速度快得多，但是当前的解决方案也无法保证即使承诺的可实现的传输吞吐量也无法保证。在本文中，我们提出了一种基于数学建模的新型动态吞吐量优化模型，其离线知识发现/分析和自适应在线决策。在离线分析中，我们挖掘了历史转移日志，以执行有关转移特征的知识发现。在线阶段使用了从离线分析中发现的知识以及对网络条件的实时调查以优化协议参数。由于实时调查很昂贵，并且提供了有关当前网络状态的部分知识，因此我们的模型使用有关网络和数据的历史知识来减少实时调查开销，同时确保每次转移的最佳吞吐量。我们的新颖方法在不同数据集的不同网络上进行了测试，并以1.7倍的速度优于其最接近的竞争对手，默认情况下的情况则以5倍。与这些网络上最佳可实现的吞吐量相比，它还达到了93％的精度。
The amount of data moved over dedicated and non-dedicated network links increases much faster than the increase in the network capacity, but the current solutions fail to guarantee even the promised achievable transfer throughputs. In this paper, we propose a novel dynamic throughput optimization model based on mathematical modeling with offline knowledge discovery/analysis and adaptive online decision making. In offline analysis, we mine historical transfer logs to perform knowledge discovery about the transfer characteristics. Online phase uses the discovered knowledge from the offline analysis along with real-time investigation of the network condition to optimize the protocol parameters. As real-time investigation is expensive and provides partial knowledge about the current network status, our model uses historical knowledge about the network and data to reduce the real-time investigation overhead while ensuring near optimal throughput for each transfer. Our novel approach is tested over different networks with different datasets and outperformed its closest competitor by 1.7x and the default case by 5x. It also achieved up to 93% accuracy compared with the optimal achievable throughput possible on those networks.