Hyperscaling Internet Graph Analysis with D4M on the MIT SuperCloud
Hyperscaling Internet Graph Analysis with D4M on the MIT SuperCloud
复制标题
在 MIT SuperCloud 上使用 D4M 进行超大规模互联网图分析
DOI：
10.1109/hpec.2018.8547552
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
A. Reuther
中科院分区：
文献类型：
--
作者：
V. Gadepally;J. Kepner;Lauren Milechin;W. Arcand;David Bestor;Bill Bergeron;C. Byun;M. Hubbell;Michael Houle;Michael Jones;P. Michaleas;J. Mullen;Andrew Prout;Antonio Rosa;Charles Yee;S. Samsi;A. Reuther
由于网络流量的数量和速度，检测网络流量中的异常行为是一个重大挑战。例如，10千兆以太网连接可以生成超过50 MB/s的数据包报头。对于全球网络提供商来说，这一挑战可能会放大许多数量级。新型计算机网络流量分析的开发需要：高级编程环境，大量的数据包捕获（PCAP）数据，以及用于“大规模”算法管道开发的各种数据产品。D4 M（动态分布式维度数据模型）结合了稀疏线性代数、关联数组、并行处理和分布式数据库（如SciDB和Apache Accumulo）的强大功能，提供了一个可扩展的数据和计算系统，解决了与网络分析开发相关的大数据问题。将D4 M与MIT SuperCloud众核处理器和并行存储系统相结合，使网络分析师能够在几分钟内交互式处理大量数据。为了展示这些功能，我们在D4 M中实现了一个代表性的分析管道，并使用MIT SuperCloud对96小时的千兆PCAP数据进行了基准测试。从解压缩原始文件到数据库摄取的整个管道在135行D4 M代码中实现，并实现了超过20，000的加速比。
Detecting anomalous behavior in network traffic is a major challenge due to the volume and velocity of network traffic. For example, a 10 Gigabit Ethernet connection can generate over 50 MB/s of packet headers. For global network providers, this challenge can be amplified by many orders of magnitude. Development of novel computer network traffic analytics requires: high level programming environments, massive amount of packet capture (PCAP) data, and diverse data products for “at scale” algorithm pipeline development. D4M (Dynamic Distributed Dimensional Data Model) combines the power of sparse linear algebra, associative arrays, parallel processing, and distributed databases (such as SciDB and Apache Accumulo) to provide a scalable data and computation system that addresses the big data problems associated with network analytics development. Combining D4M with the MIT SuperCloud manycore processors and parallel storage system enables network analysts to interactively process massive amounts of data in minutes. To demonstrate these capabilities, we have implemented a representative analytics pipeline in D4M and benchmarked it on 96 hours of Gigabit PCAP data with MIT SuperCloud. The entire pipeline from uncompressing the raw files to database ingest was implemented in 135 lines of D4M code and achieved speedups of over 20,000.