II-NEW: Sharing The Perspective of Ten Million Peers

健全的互联网研究需要全面的网络测量。这项工作的动机是观察到，大规模对等系统(P2P)中的对等点对网络条件有独特而有价值的观点，而今天S的研究人员、网络运营商和用户只能有限地访问或没有访问权限。一年多来，首席调查者(PI)已经提供了一个BitTorrent扩展，旨在减少P2P系统中跨ISP流量的已知问题。该扩展通过使用基于重用来自商业CDN的网络信息的轻量级方法通过偏向对等点选择来减少离网流量，并且已被证明在野外是非常有效的。除了偏向同行选择外，该扩展还允许订阅志愿者参与互联网监控服务。该系统目前拥有超过26万名志愿者用户(目前每天增长200人)，分布在150多个国家和3000多个网络中，已经是有史以来规模最大的监测服务。每天，不断增长的用户集合允许研究人员记录超过1亿个传输速率样本，180万个跟踪路由，发送给分布在18,000个ASS上的超过350万个同龄人。虽然显然有大量重要的研究问题可以根据这个数据集得到回答，但收集、归档和分析这些数据量的任务是一个持续不断的挑战。在一天内，我们必须处理超过7 GB的未压缩数据！为了应对这一挑战，PI将建立P2P监控系统，这是一个对参与主机收集的被动收集的网络和应用程序级跟踪进行数据收集、存档和离线分析的平台。最终目标是：(A)减少目前专门用于与这一数据集管理有关的后勤问题的努力，以便集中精力探索它开辟的许多研究途径；(B)通过CAIDA等成熟的组织，与更广泛的社区共享和长期保存这一有价值的数据集。智力优势：该框架将有助于开展广泛的研究活动，包括：(I)探索基于先前收集的测量数据的重用来设计和实施大规模分布式系统的新方法；(Ii)了解完全依赖从P2P系统被动收集的信息的网络预警系统的可行性；(Iii)评估P2P用户形成的社交网络的演变和稳定性；(Iv)探索通过全球数十万个有利位置的贡献来揭示更完整的互联网连接图的可能性；以及(V)研究诊断和增强网络对自然灾害的复原力的方法。更广泛的影响：访问网络的终端主机视图对于了解和描述底层网络以及满足新的新兴互联网服务和应用的需求至关重要。在该项目下开发的框架将使研究人员从与收集和处理收集的观点相关的后勤问题中解脱出来，使他们能够专注于探索由该项目开辟的一些有前途的研究途径。这一框架和数据集的提供还将有助于通过学生培训和课程改进显著提高西北大学的教育水平。高质量测量的可用性往往是大多数研究工作成功的关键。为了向更广泛的研究界开放基于声音和测量的研究领域，我们将继续与CAIDA进行对话，探索促进该数据集共享并确保其长期存档的方法。

Sound Internet research requires comprehensive network measurements. This work is motivated by the observation that, collectively, peers in large-scale peer-to-peer systems (P2P) have a unique and valuable perspective on network conditions, one to which today?s researchers, network operators and users have limited or no access. For over a year the principal investigator (PI) has made available a BitTorrent extension aimed at reducing the known problem of cross-ISP traffic in P2P systems. This extension reduces off-network traffic through biased peer selection using a lightweight approach based on reusing network information from commercial CDNs, and it has been shown to be highly effective in the wild. In addition to biasing peer selection, the extension allows subscribing volunteers to participate in a monitoring service for the Internet. With over 260,000 volunteer users today (a current growth rate of 200 per day), and distributed in over 150 countries and more than 3000 networks, this system is already the largest-ever monitoring service. On a daily basis, the growing collection of users allow the researcher to record over 100, 000, 000 transfer-rate samples, 1.8 million traceroutes, to over 3.5 million peers spread over 18, 000 ASs. While there are clearly a large number of important research questions that could be answered based on this dataset, the task of collecting, archiving and analyzing this volume of data is a constant challenge. On a single day, we must handle over 7GB of uncompressed data! To address this challenge, the PI will build the P2P Monitoring System, a platform for data collection, archival and off-line analysis of the passively collected network- and application-level traces collected by the participating hosts. The end goals are (a) to reduce the effort currently dedicated to logistical issues related to the management of this data set in order to focus on exploring the many research paths opened by it and (b) to make possible the sharing and long-term archival of this valuable dataset with the wider community through well-established organizations such as CAIDA. Intellectual Merit: The framework will enable a wide range of research activities, including: (i) explore a new approach to the design and implementation of large-scale distributed systems based on the reuse of previously gathered measurements, (ii) understand the feasibility of a network early warning system that relies exclusively on information passively gathered from P2P systems, (iii) evaluate the evolution and stability of the social network formed by P2P users, (iv) explore the potential for revealing a more complete Internet connectivity graph through the contribution of hundreds of thousands of vantage points worldwide and (v) investigate approaches to diagnose and enhance the resilience of network to natural disasters. Broader Impact: Access to end-host views of the network is key to the understanding and characterization of the underlying network and to addressing the needs of new emergent Internet services and applications. The framework developed under this project will free researchers from logistical concerns related to the collection and processing of this collected view, allowing them instead to focus on exploring some of the promising research paths opened by it. The availability of this framework and dataset will also help to significantly enhance education at Northwestern through student training and curriculum enhancement. The availability of high-quality measurements is often key to the success of most research efforts. To open the field of sound, measurement-based research to the broader research community, we will continue an ongoing conversation with CAIDA exploring approaches to facilitate the sharing of this dataset and ensure its long-term archival.