ITR: Robust Large-Scale Distributed Systems

该项目是一个基于分布式哈希表（dht）的新型分散基础设施，它将支持新一代大规模分布式应用程序。我们构建的关键技术dht在面对故障、攻击和意外高负载时是健壮的。它们是可伸缩的，可以在不产生不必要开销的情况下实现大型系统。它们是自配置的，无需人工干预或监督即可自动合并新节点。它们通过提供干净灵活的接口简化了分布式编程。最后，它们提供了一个可供许多应用程序同时使用的共享基础设施。这里提倡的方法与集中式客户机-服务器和分布式应用程序的特定于应用程序的覆盖模型完全不同。这种新方法不仅会改变大规模分布式系统的构建方式，还可能对社会产生深远的影响。构建任何分布式系统的主要挑战在于处理安全性、健壮性、管理和可扩展性问题；今天，每个新系统都必须自己解决这些问题，需要大量的硬件投资和复杂的软件设计。共享的分布式基础设施将减轻单个应用程序的这些负担，从而大大降低大规模分布式服务的进入门槛。我们相信DHT是正确的启用基础设施是基于两个猜想：(1)具有应用程序独立的、不受约束的键和值的DHT提供了一个通用的接口，在这个接口上可以构建各种各样的分布式应用程序；(2)使用基于DHT的基础设施的分布式应用程序继承了基本的安全性、健壮性、易于操作和可扩展性。拟议研究的主要内容是对这两个猜想的探索。我们将研究第一个猜想，即通过构建各种基于DHT的系统，DHT抽象可以支持广泛的应用程序。我们最近的工作使用dht来支持各种应用，如分布式文件系统、多播覆盖网络、事件通知系统和分布式查询处理。dht通过提供通用的键/值命名而不是强加结构化的键（例如，DNS中的分层名称），简化了这些系统的结构。这些系统都是早期的原型，但它们表明dht对于分布式应用程序可能像普通散列表对于程序一样有用。第二个猜想依赖于从许多相互不信任的节点中创建健壮、安全且自组织的基础设施的技术。我们在稳健DHT设计方面的初步工作使我们相信这些技术是可以实现的。我们提出的大部分研究将致力于对这些技术的深入研究，其明确目标是为基础设施提供一个健全和连贯的设计。为了研究我们设计的真实行为，我们将创建一个大规模的开放测试平台，我们将为其分发基础设施软件、一些启用库和一些关键的引人注目的应用程序。除了对分布式应用程序的创建产生影响外，我们的研究计划还将在教育和推广方面受益。鉴于它们当前的重要性，分布式系统的安全性、健壮性和设计应该成为本科计算机科学教育的中心主题。为此，我们正在计划开设一门新的跨学科课程，以解决这些问题，并在本科课程的早期阶段将它们纳入更清晰的焦点。我们的测试平台和研究议程也是鼓励传统上没有参与网络和系统研究的组织参与的良好工具。参与试验台只需要很少的成本（一台PC和一个Internet连接）和最低水平的系统专业知识和监督。此外，由于材料与许多学生熟悉的P2P系统密切相关，该项目可能会吸引那些通常不会被这一领域的研究吸引的学生。基于这一前提，我们计划在非研究型本科院校为代表性不足的人群开展积极的外展计划。

This project is a novel decentralized infrastructure, based on distributed hash tables (DHTs), that will enable a new generation of large-scale distributed applications. The key technology on which we build, DHTs, are robust in the face of failures, attacks and unexpectedly high loads. They are scalable, achieving large system sizes without incurring undue overhead. They are self-configuring, automatically incorporating new nodes without manual intervention or oversight. They simplify distributed programming by providing a clean and flexible interface. And, finally, they provide a shared infrastructure simultaneously usable by many applications.The approach advocated here is a radical departure from both the centralized client-server and the application-specific overlay models of distributed applications. This new approach will not only change the way large-scale distributed systems are built, but could potentially have far reaching societal implications as well. The main challenge in building any distributed system lies in dealing with security, robustness, management, and scaling issues; today each new system must solve these problems for itself, requiring significant hardware investment and sophisticated software design. The shared distributed infrastructure will relieve individual applications of these burdens, thereby greatly reducing the barriers to entry for large-scale distributed services.Our belief that DHTs are the right enabling infrastructure is based on two conjectures: (1) a DHT with application-independent, unconstrained keys and values provides a general purpose interface upon which a wide variety of distributed applications can be built, and (2) distributed applications that make use of the DHT-based infrastructure inherit basic levels of security, robustness, ease of operation, and scaling. Much of the thrust of the proposed research is an exploration of these two conjectures.We will investigate the first conjecture, that the DHT abstraction can support a wide range of applications, by building a variety of DHT-based systems. Our recent work has used DHTs to support such varied applications as distributed file systems, multicast overlay networks, event notification systems, and distributed query processing. DHTs simplify the structure of these systems by providing general-purpose key/value naming rather than imposing structured keys (e.g., hierarchical names in DNS). These systems are early prototypes, but they suggest that DHTs may be as useful to distributed applications as ordinary hash tables are to programs.The second conjecture relies on techniques for creating robust, secure, and self-organizing infrastructures out of many mutually distrustful nodes. Our initial work on robust DHT designs gives us confidence that such techniques are within reach. The bulk of our proposed research will be devoted to the in-depth study of these techniques, with the express aim of producing a sound and coherent design for the infrastructure. To investigate the real-world behavior of our design, we will create a large-scale open testbed for which we will distribute our infrastructure software, some enabling libraries, and a few key compelling applications.In addition to its impact on the creation of distributed applications, our research program will have benefits in education and outreach. Given their current importance, security, robustness, and the design of distributed systems should become central topics in undergraduate computer science education. To this end, we are planning a new interdisciplinary course that will address these issues, and bring them into sharper focus early in the undergraduate course sequence.Our testbed and research agenda is also a good vehicle for encouraging the participation of organizations not traditionally involved in networking and systems research. Participation in the testbed requires little cost (a PC and an Internet connection) and minimal levels of systems expertise and over-sight. Moreover, because the material is closely related to the P2P systems with which many students are familiar, the project might appeal to students who would not normally be attracted to research in this area. Based on this premise, we plan an active outreach program to underrepresented populations at non-research undergraduate institutions.