II-NEW: Sharing The Perspective of Ten Million Peers

可靠的互联网研究需要全面的网络测量。这项工作的动机是观察，集体，在大规模的对等系统（P2P）的同行有一个独特的和有价值的角度对网络条件，一个今天？研究人员、网络运营商和用户的访问权限有限或根本没有。 一年多来，首席研究员（PI）已经提供了一个BitTorrent扩展，旨在减少P2P系统中跨ISP流量的已知问题。这种扩展通过使用基于重用来自商业CDN的网络信息的轻量级方法进行有偏见的对等选择来减少离网流量，并且已被证明在野外非常有效。除了偏向同行选择外，该扩展还允许订阅志愿者参与互联网的监测服务。该系统目前有26万多名志愿者用户（目前每天增长200人），分布在150多个国家和3000多个网络，已经是有史以来规模最大的监测服务。每天，不断增长的用户集合允许研究人员记录超过100，000，000个传输率样本，180万个跟踪路由，分布在18，000个AS上的超过350万个对等点。虽然显然有大量重要的研究问题可以基于这个数据集来回答，但收集、归档和分析这些数据量的任务是一个持续的挑战。在一天之内，我们必须处理超过7 GB的未压缩数据！为了应对这一挑战，PI将建立P2P监控系统，这是一个数据收集、存档和离线分析平台，用于被动收集参与主机收集的网络和应用程序级跟踪。最终目标是：（a）减少目前专门用于管理这一数据集的后勤问题的工作，以便集中精力探索这一数据集开辟的许多研究途径;（B）通过CAIDA等地位稳固的组织，使这一宝贵的数据集能够与更广泛的社区分享并长期存档。智力优势：该框架将促进广泛的研究活动，包括：（i）探索一种新的方法来设计和实现基于重用先前收集的测量的大规模分布式系统，（ii）理解完全依赖于从P2P系统被动收集的信息的网络预警系统的可行性，（iii）评估P2P用户形成的社交网络的演化和稳定性，（iv）探索通过全球数十万个Vantage点的贡献揭示更完整的互联网连接图的潜力，以及研究诊断和增强网络对自然灾害的复原力的方法。更广泛的影响：访问网络的终端主机视图是理解和描述底层网络以及满足新兴互联网服务和应用需求的关键。在这个项目下开发的框架将使研究人员从与收集和处理这些收集到的观点有关的后勤问题中解脱出来，使他们能够专注于探索它所开辟的一些有前途的研究路径。这个框架和数据集的可用性也将有助于通过学生培训和课程改进来显着提高西北大学的教育。高质量测量的可用性往往是大多数研究工作成功的关键。为了向更广泛的研究界开放健全的、基于测量的研究领域，我们将继续与CAIDA进行对话，探索促进该数据集共享并确保其长期存档的方法。

Sound Internet research requires comprehensive network measurements. This work is motivated by the observation that, collectively, peers in large-scale peer-to-peer systems (P2P) have a unique and valuable perspective on network conditions, one to which today?s researchers, network operators and users have limited or no access. For over a year the principal investigator (PI) has made available a BitTorrent extension aimed at reducing the known problem of cross-ISP traffic in P2P systems. This extension reduces off-network traffic through biased peer selection using a lightweight approach based on reusing network information from commercial CDNs, and it has been shown to be highly effective in the wild. In addition to biasing peer selection, the extension allows subscribing volunteers to participate in a monitoring service for the Internet. With over 260,000 volunteer users today (a current growth rate of 200 per day), and distributed in over 150 countries and more than 3000 networks, this system is already the largest-ever monitoring service. On a daily basis, the growing collection of users allow the researcher to record over 100, 000, 000 transfer-rate samples, 1.8 million traceroutes, to over 3.5 million peers spread over 18, 000 ASs. While there are clearly a large number of important research questions that could be answered based on this dataset, the task of collecting, archiving and analyzing this volume of data is a constant challenge. On a single day, we must handle over 7GB of uncompressed data! To address this challenge, the PI will build the P2P Monitoring System, a platform for data collection, archival and off-line analysis of the passively collected network- and application-level traces collected by the participating hosts. The end goals are (a) to reduce the effort currently dedicated to logistical issues related to the management of this data set in order to focus on exploring the many research paths opened by it and (b) to make possible the sharing and long-term archival of this valuable dataset with the wider community through well-established organizations such as CAIDA. Intellectual Merit: The framework will enable a wide range of research activities, including: (i) explore a new approach to the design and implementation of large-scale distributed systems based on the reuse of previously gathered measurements, (ii) understand the feasibility of a network early warning system that relies exclusively on information passively gathered from P2P systems, (iii) evaluate the evolution and stability of the social network formed by P2P users, (iv) explore the potential for revealing a more complete Internet connectivity graph through the contribution of hundreds of thousands of vantage points worldwide and (v) investigate approaches to diagnose and enhance the resilience of network to natural disasters. Broader Impact: Access to end-host views of the network is key to the understanding and characterization of the underlying network and to addressing the needs of new emergent Internet services and applications. The framework developed under this project will free researchers from logistical concerns related to the collection and processing of this collected view, allowing them instead to focus on exploring some of the promising research paths opened by it. The availability of this framework and dataset will also help to significantly enhance education at Northwestern through student training and curriculum enhancement. The availability of high-quality measurements is often key to the success of most research efforts. To open the field of sound, measurement-based research to the broader research community, we will continue an ongoing conversation with CAIDA exploring approaches to facilitate the sharing of this dataset and ensure its long-term archival.