NetAgg: Using Middleboxes for Application-specific On-path Aggregation in Data Centres
NetAgg: Using Middleboxes for Application-specific On-path Aggregation in Data Centres
复制标题
DOI：
10.1145/2674005.2674996
复制
发表时间：
2014-12
期刊：
影响因子：
--
通讯作者：
Luo Mai;Lukas Rupprecht;Abdul Alim;Paolo Costa;Matteo Migliavacca;P. Pietzuch;A. Wolf
中科院分区：
文献类型：
--
作者：
Luo Mai;Lukas Rupprecht;Abdul Alim;Paolo Costa;Matteo Migliavacca;P. Pietzuch;A. Wolf
用于批处理（例如map/reduce框架）和在线服务（例如搜索引擎）的数据中心应用程序通过将数据和计算分布在许多服务器上来扩展。它们通常遵循分区/聚合模式：任务首先在本地处理数据的服务器上分区，然后聚合这些部分结果。然而，这个数据聚合步骤将性能瓶颈转移到了网络上，网络通常难以支持服务器之间多对少的高带宽流量。而不是在边缘服务器上执行数据聚合，我们表明，它可以更有效地完成沿着网络路径。我们描述了NETAGG，一个软件平台，支持路径聚合网络绑定分区/聚合应用程序。NETAGG采用了类似中间盒的设计，其中专用服务器（agg盒）通过高带宽链路连接到网络交换机。Agg盒执行由应用程序提供的聚合功能，这会使网络热点更加密集，因为只有一小部分传入流量在每一跳被转发。NETAGG只需要最小的应用程序更改：它使用边缘服务器上的shim层将应用程序流量透明地重定向到agg框。我们的实验结果表明，NETAGG大大提高了两个示例应用程序，Solr分布式搜索引擎和Hadoop批处理框架的吞吐量。其设计允许在现有数据中心进行增量部署，投资成本不高。
Data centre applications for batch processing (e.g. map/reduce frameworks) and online services (e.g. search engines) scale by distributing data and computation across many servers. They typically follow a partition/aggregation pattern: tasks are first partitioned across servers that process data locally, and then those partial results are aggregated. This data aggregation step, however, shifts the performance bottleneck to the network, which typically struggles to support many-to-few, high-bandwidth traffic between servers. Instead of performing data aggregation at edge servers, we show that it can be done more efficiently along network paths. We describe NETAGG, a software platform that supports on-path aggregation for network-bound partition/aggregation applications. NETAGG exploits a middlebox-like design, in which dedicated servers (agg boxes) are connected by high-bandwidth links to network switches. Agg boxes execute aggregation functions provided by applications, which alleviates network hotspots because only a fraction of the incoming traffic is forwarded at each hop. NETAGG requires only minimal application changes: it uses shim layers on edge servers to redirect application traffic transparently to the agg boxes. Our experimental results show that NETAGG improves substantially the throughput of two sample applications, the Solr distributed search engine and the Hadoop batch processing framework. Its design allows for incremental deployment in existing data centres and incurs only a modest investment cost.