NeTS-NBD-SGER: Map/Reduce for Network Traffic Analysis (MR-Net Sger)

0823774这个项目探索了一类新的并行算法，可以处理非常大的网络数据集。目标是能够分析27亿次ping来绘制互联网地址空间，处理六个月的流量记录来了解长期流量趋势，并搜索一周。的数据包标头的价值，以追溯检测零日受损的机器。这些任务中的每一个都需要高效和经济地处理大小从50 GB到几TB的数据集。数据集大小的这一飞跃是100-1000倍或更多，需要与今天完全不同的处理网络数据的方式。的tcpdump和ethereal。最近的两个发展使这一飞跃成为可能。首先，Google已经证明了map/reduce抽象可以很容易地并行化，并且可以在数百台商用PC的集群上高效、经济地运行。它是他们的网络搜索引擎的基础，并促使至少一个开源实现。Map/Reduce是Google的计算引擎吗？的网络搜索引擎，越来越多地用于其他应用程序。Map/Reduce是处理大型网络数据集的关键。其次，最近的计划，如预测，使大量的网络数据集可用。PREDICT承诺提供来自几个大型ISP的数据包报头跟踪、地址空间扫描、网络流记录、暗地址空间流量和IP语音（VoIP）呼叫记录。在南加州大学，研究人员正在收集数据包报头跟踪和地址空间扫描。PREDICT是获取庞大网络数据集的关键。这项工作是作为SGER提出的，因为它既及时，在这一点上，高度投机。研究人员必须先确定map/reduce可以解决哪些网络问题，然后才能给出可信的完整建议。这个建议是关键，展示了巨大的网络数据集的map/reduce处理的潜在影响，以改变我们对互联网的理解。这项工作的智力价值是开发一个初步的理解，如何使用map/reduce风格处理网络数据集。哪些算法适用？哪些问题并行性好或差？需要什么样的计算集群？更一般地说，网络研究人员如何科普描述十亿用户互联网所需的千兆字节到兆字节的数据集？这项工作的更广泛的优点是，它将导致互联网面临的基本和实际问题的答案。考虑到这些数据集，问题包括：互联网是什么样子的？互联网地址空间的消耗率是多少？有多少互联网用户使用动态地址连接？如何有效应对入侵？新技术能否检测到低速率的拒绝服务攻击、受感染服务器中的垃圾邮件生成？可以追溯和检测僵尸网络控制网络？比这些具体问题更重要的是一个更广泛的问题，即如何理解十亿节点互联网的属性以及每天在其上流动的PB流量。

0823774This project explores a new class of parallel algorithms that process very large network datasets. The goal is to be able to analyze 2.7 billion pings to map the Internet address space, process six months of flow records to understand long-term traffic trends, and search a week?s worth of packet headers to retroactively detect zero-day compromised machines. Each of these tasks requires efficient and economical processing of datasets in sizes from 50GB to several terabytes. This leap in dataset sizes by a factor of 100-1000-fold or more requires fundamentally different ways of handling network data than today?s tcpdump and ethereal on a workstation. Two recent developments make this leap possible. First, Google has demonstrated that the map/reduce abstraction be easily parallelized and run efficiently and cost-effectively over clusters of hundreds of commodity PCs. It is the basis of their web search engine and has prompted at least one open source implementation. Map/reduce is the computation engine of Google?s web search engine, increasingly being used in other applications. Map/reduce is the key to processing huge network datasets. Second, recent programs such as PREDICT make massive network datasets available. PREDICT promises to make available packet header traces, address space scans, netflow records, dark address space traffic, and voice over IP (VoIP) call records from several large ISPs. At USC researchers are collecting packet header traces and address space scans. PREDICT is the key to obtaining huge network datasets. This work is proposes as a SGER because it is both timely and, at this point, highly speculative. The researchers must characterize what network problems can be solved by map/reduce before a full proposal will be credible. This proposal is the key to demonstrating the potential impact of for map/reduce processing of huge network datasets to change our understanding of the Internet.The intellectual merit of this work is to develop a preliminary understanding of how to use map/reduce style processing for network datasets. What algorithms are applicable? What problems parallelize well or poorly? What kind of compute clusters are needed? And more generally, how can networking researchers cope with gigabyte-to-terabyte datasets that are needed to describe a billion-user Internet?The broader merit of this work is that it will lead to answers of both fundamental and practical questions facing the Internet. Considering these datasets, questions include: What does the Internet look like? At what rate is the Internet address space being consumed? How many Internet users connect with dynamic addresses? How can one respond to intrusions effectively? Can new techniques detect low-rate denial-of-service attacks, spam generation in compromised servers? Can one traceback and detect botnets control networks? More important than these specific questions is the broader question of how can one understand properties in a billion-node Internet and the petabyte-per day of traffic that flows over it.