Random access in nondelimited variable-length record collections for parallel reading with Hadoop
Random access in nondelimited variable-length record collections for parallel reading with Hadoop
复制标题
DOI：
10.23919/inm.2017.7987424
复制
发表时间：
2017-05
期刊：
影响因子：
--
通讯作者：
Jason W. Anderson;Christopher Gropp;Linh Ngo;A. Apon
中科院分区：
文献类型：
--
作者：
Jason W. Anderson;Christopher Gropp;Linh Ngo;A. Apon
用于存储网络数据包跟踪的行业标准数据包捕获（PCAP）格式由于缺少分隔符、索引或阻塞，通常只能以串行方式读取。这对大型网络的并行分析提出了挑战，其中数据包跟踪的大小可以是许多千兆字节。在这项工作中，我们提出了RAPCAP，一种新的方法随机访问到可变长度的记录集合，如PCAP通过识别一个记录边界内的一个小数目的字节的访问点。不像相关的启发式方法，可以限制可扩展性与非零概率的错误，新方法提供了一个正确的保证与格式良好的文件，不依赖于先验知识的内容。我们包括一个实际的实现的算法与Hadoop框架的扩展，和串行摄取的性能比较。最后，我们提出了一些类似的存储类型，可以利用RAPCAP的修改版本进行随机访问。
The industry standard Packet CAPture (PCAP) format for storing network packet traces is normally only readable in serial due to its lack of delimiters, indexing, or blocking. This presents a challenge for parallel analysis of large networks, where packet traces can be many gigabytes in size. In this work we present RAPCAP, a novel method for random access into variable-length record collections like PCAP by identifying a record boundary within a small number of bytes of the access point. Unlike related heuristic methods that can limit scalability with a nonzero probability of error, the new method offers a correctness guarantee with a well formed file and does not rely on prior knowledge of the contents. We include a practical implementation of the algorithm with an extension to the Hadoop framework, and a performance comparison to serial ingestion. Finally, we present a number of similar storage types that could utilize a modified version of RAPCAP for random access.