ShmStreaming: A Shared Memory Approach for Improving Hadoop Streaming Performance
ShmStreaming: A Shared Memory Approach for Improving Hadoop Streaming Performance
复制标题
ShmStreaming：一种提高 Hadoop 流性能的共享内存方法
DOI：
10.1109/aina.2013.90
复制
发表时间：
2013
期刊：
影响因子：
--
通讯作者：
F. Tang and M. Guo
中科院分区：
文献类型：
--
作者：
L. Lai;J. Zhou;L. Zheng. H. Li;Y. Lu;F. Tang and M. Guo
Map-Reduce编程模型正在引起学术界和工业界对处理大数据的关注。Hadoop是该模型最流行的实现之一，已被广泛采用。为了支持用Java以外的语言编写的应用程序，Hadoop引入了一种流机制，允许它通过管道与外部程序通信。由于管道和上下文切换相关的额外开销，Hadoop流的性能明显比原生Hadoop作业差。我们提出了一种利用共享内存实现Hadoop流的机制ShmStreaming，以获得更好的性能。具体来说，ShmStreaming使用共享内存来实现连接Hadoop和外部程序的无锁FIFO队列。为了进一步减少上下文切换的数量，FIFO队列采用批处理技术，允许多个键值对一起处理。对于字数统计、grep和倒排索引的典型基准测试，实验结果表明，与原生Hadoop流实现相比，性能提高了20-30%。
The Map-Reduce programming model is now drawing both academic and industrial attentions for processing large data. Hadoop, one of the most popular implementations of the model, has been widely adopted. To support application programs written in languages other than Java, Hadoop introduces a streaming mechanism that allows it to communicate with external programs through pipes. Because of the added overhead associated with pipes and context switches, the performance of Hadoop streaming is significantly worse than native Hadoop jobs. We propose ShmStreaming, a mechanism that takes advantages of shared memory to realize Hadoop streaming for better performance. Specifically, ShmStreaming uses shared memory to implement a lockless FIFO queue that connects Hadoop and external programs. To further reduce the number of context switches, the FIFO queue adopts a batching technique to allow multiple key-value pairs to be processed together. For typical benchmarks of word count, grep and inverted index, experimental results show 20-30% performance improvement comparing to the native Hadoop streaming implementation.