MRI: Development of a Shared Network Measurement Storage and Analysis Infrastructure

该项目开发了一个共享的网络测量分析和存储基础设施，称为数据传感器，旨在提供一个数据分析和管理工具的通用平台。该仪器作为一个研究平台，用于创建更大规模、可公开访问的测量分析和存储基础设施。从实际部署中收集和分析数据对网络社区以及由这些数据驱动的实验提出了严峻挑战。这项工作旨在通过从现成的组件构建共享基础设施来减少与管理研究人员所需的大量数据相关的大量管理时间和成本，从而促进以下研究工作：-创建互联网规模的取证分析体系结构，-理解和改进互联网路由，-设计和评估高度可用的网络体系结构，-评估新的数据传输体系结构，-在大型网络跟踪集合上测试蠕虫和入侵检测算法，以及-启用几个教育扩展项目。这些努力面临着数据管理、组织和分析的重大挑战，需要大量的硬件和软件基础设施来存储和分析TB级的网络测量数据。数据传感器包括数据库配置和设置、模式优化、数据组织和分类、数据分发、硬件和操作系统配置，以及创建代码以执行数据的基本过滤和处理。测量和分析基础设施将作为开发大规模可供公众访问的网络数据储存库的基本原型。

This project, developing a shared network measurement analysis and storage infrastructure called the Datapository, aims at providing a common platform of data analysis and management tools. The instrument serves as a research platform for creating a larger-scale, publicly accessible measurement analysis and storage infrastructure. Collection and analysis of data from real deployments critically challenges the network community, as well as experiments driven by such data. This work aims at reducing the substantial administration time and costs associated with management large amounts of data needed by researchers by building a shared infrastructure from off-the-shelf components, and consequently facilitating the following research efforts:-Creating Internet-scale forensic analysis architectures,-Understanding and improving Internet routing,-Designing and evaluating highly available network architectures,-Evaluating novel data transfer architectures,-Testing worm and intrusion detection algorithms on large network trace collections, and-Enabling several educational outreach projects.These efforts face a significant challenge of data management, organization, and analysis, requiring substantial hardware and software infrastructure to store and analyze terabytes of network measurement data. The Datapository includes database configuration and setup, schema optimization, data organization and classification, data distribution, hardware and operating system configuration, and the creation of a code to perform basic filtering and processing of the data. The measurement and analysis infrastructure will serve as the base prototype for the development of a large-scale publicly accessible network data repository.