Dynamic Load-Balanced Multicast for Data-Intensive Applications on Clouds
Dynamic Load-Balanced Multicast for Data-Intensive Applications on Clouds
复制标题
DOI：
10.1109/ccgrid.2010.63
复制
发表时间：
2010-05
期刊：
影响因子：
--
通讯作者：
Tatsuhiro Chiba;M. Burger;T. Kielmann;S. Matsuoka
中科院分区：
文献类型：
--
作者：
Tatsuhiro Chiba;M. Burger;T. Kielmann;S. Matsuoka
云上的数据密集型并行应用程序需要尽快将大数据集从云的存储设施部署到所有计算节点。针对集群和网格环境，已经提出了许多组播算法。最常见的方法是根据网络拓扑和网络监控数据构建一棵或多棵生成树，以最大化可用带宽并避免瓶颈链路。然而，一旦可用带宽动态变化，提供最佳性能就变得困难。本文以当今最常用的云平台Amazon EC2/S3为研究对象，提出了两种高性能的组播算法。这些算法使得高效地将存储在Amazon S3中的大量数据传输到多个Amazon EC2节点成为可能。该算法的三个显著特点是(1)在没有网络拓扑信息的情况下在云上构建覆盖网络；(2)动态优化总吞吐量；(3)通过节点之间的协作来提高下载吞吐量。这两种算法在节点协作的方式上有所不同：第一种“非窃取”算法允许每个节点下载相等份额的所有数据，而第二种“窃取”算法使用工作窃取来对抗异类下载带宽的影响。因此，所有节点都可以快速从S3下载文件，即使在算法运行时网络性能发生变化也是如此。我们在EC2/S3上对我们的算法进行了测试，结果表明它们是可扩展的，并且一致地获得了高吞吐量。这两种算法都比让每个节点直接从S3下载所有数据要好得多。
Data-intensive parallel applications on clouds need to deploy large data sets from the cloud's storage facility to all compute nodes as fast as possible. Many multicast algorithms have been proposed for clusters and grid environments. The most common approach is to construct one or more spanning trees based on the network topology and network monitoring data in order to maximize available bandwidth and avoid bottleneck links. However, delivering optimal performance becomes difficult once the available bandwidth changes dynamically. In this paper, we focus on Amazon EC2/S3 (the most commonly used cloud platform today) and propose two high performance multicast algorithms. These algorithms make it possible to efficiently transfer large amounts of data stored in Amazon S3 to multiple Amazon EC2 nodes. The three salient features of our algorithms are (1) to construct an overlay network on clouds without network topology information, (2) to optimize the total throughput dynamically, and (3) to increase the download throughput by letting nodes cooperate with each other. The two algorithms differ in the way nodes cooperate: the first `non-steal' algorithm lets each node download an equal share of all data, while the second `steal' algorithm uses work stealing to counter the effect of heterogeneous download bandwidth. As a result, all nodes can download files from S3 quickly, even when the network performance changes while the algorithm is running. We evaluate our algorithms on EC2/S3, and show that they are scalable and consistently achieve high throughput. Both algorithms perform much better than having each node downloading all data directly from S3.