ITR: Robust Large-Scale Distributed Systems

ITR：稳健的大型分布式系统

• 批准号: 0225660
• 负责人: Marinus Kaashoek
• 金额: --
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Cooperative Agreement
• 财政年份: 2002
• 资助国家: 美国
• 起止时间: 2002-10-01 至 2009-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0225660&HistoricalAwards=false
• 关键词: ITR; Robust; Large; Scale; Distributed

This project is a novel decentralized infrastructure, based on distributed hash tables (DHTs), that will enable a new generation of large-scale distributed applications. The key technology on which we build, DHTs, are robust in the face of failures, attacks and unexpectedly high loads. They are scalable, achieving large system sizes without incurring undue overhead. They are self-configuring, automatically incorporating new nodes without manual intervention or oversight. They simplify distributed programming by providing a clean and flexible interface. And, finally, they provide a shared infrastructure simultaneously usable by many applications.The approach advocated here is a radical departure from both the centralized client-server and the application-specific overlay models of distributed applications. This new approach will not only change the way large-scale distributed systems are built, but could potentially have far reaching societal implications as well. The main challenge in building any distributed system lies in dealing with security, robustness, management, and scaling issues; today each new system must solve these problems for itself, requiring significant hardware investment and sophisticated software design. The shared distributed infrastructure will relieve individual applications of these burdens, thereby greatly reducing the barriers to entry for large-scale distributed services.Our belief that DHTs are the right enabling infrastructure is based on two conjectures: (1) a DHT with application-independent, unconstrained keys and values provides a general purpose interface upon which a wide variety of distributed applications can be built, and (2) distributed applications that make use of the DHT-based infrastructure inherit basic levels of security, robustness, ease of operation, and scaling. Much of the thrust of the proposed research is an exploration of these two conjectures.We will investigate the first conjecture, that the DHT abstraction can support a wide range of applications, by building a variety of DHT-based systems. Our recent work has used DHTs to support such varied applications as distributed file systems, multicast overlay networks, event notification systems, and distributed query processing. DHTs simplify the structure of these systems by providing general-purpose key/value naming rather than imposing structured keys (e.g., hierarchical names in DNS). These systems are early prototypes, but they suggest that DHTs may be as useful to distributed applications as ordinary hash tables are to programs.The second conjecture relies on techniques for creating robust, secure, and self-organizing infrastructures out of many mutually distrustful nodes. Our initial work on robust DHT designs gives us confidence that such techniques are within reach. The bulk of our proposed research will be devoted to the in-depth study of these techniques, with the express aim of producing a sound and coherent design for the infrastructure. To investigate the real-world behavior of our design, we will create a large-scale open testbed for which we will distribute our infrastructure software, some enabling libraries, and a few key compelling applications.In addition to its impact on the creation of distributed applications, our research program will have benefits in education and outreach. Given their current importance, security, robustness, and the design of distributed systems should become central topics in undergraduate computer science education. To this end, we are planning a new interdisciplinary course that will address these issues, and bring them into sharper focus early in the undergraduate course sequence.Our testbed and research agenda is also a good vehicle for encouraging the participation of organizations not traditionally involved in networking and systems research. Participation in the testbed requires little cost (a PC and an Internet connection) and minimal levels of systems expertise and over-sight. Moreover, because the material is closely related to the P2P systems with which many students are familiar, the project might appeal to students who would not normally be attracted to research in this area. Based on this premise, we plan an active outreach program to underrepresented populations at non-research undergraduate institutions.

该项目是一个基于分布式哈希表(DHTs)的新型分布式基础设施，它将支持新一代大规模分布式应用。我们赖以构建的关键技术DHTs在面对故障、攻击和出人意料的高负载时是健壮的。它们是可扩展的，可以实现较大的系统规模，而不会产生不必要的开销。它们是自我配置的，无需手动干预或监督即可自动合并新节点。它们通过提供干净而灵活的界面简化了分布式编程。最后，它们提供了一个可同时供多个应用程序使用的共享基础设施。这里所倡导的方法与集中式客户端-服务器和特定于应用程序的分布式应用程序覆盖模型完全不同。这种新的方法不仅将改变大规模分布式系统的构建方式，而且还可能具有深远的社会影响。构建任何分布式系统的主要挑战在于处理安全性、健壮性、管理和可伸缩性问题；今天，每个新系统都必须自己解决这些问题，需要大量的硬件投资和复杂的软件设计。共享的分布式基础设施将减轻单个应用程序的负担，从而大大降低大规模分布式服务的进入门槛。我们相信分布式哈希表是正确的使能基础设施是基于两个猜想：(1)具有与应用程序无关的、不受约束的键和值的分布式哈希表提供了一个通用接口，可以在其上构建各种分布式应用程序；(2)利用基于分布式哈希表的基础设施的分布式应用程序继承了基本的安全性、健壮性、易操作性和可伸缩性。这项研究的主要目的是探索这两个猜想。我们将通过构建各种基于分布式哈希表的系统来调查第一个猜想，即分布式哈希表抽象可以支持广泛的应用。我们最近的工作使用DHTs来支持各种应用，如分布式文件系统、多播覆盖网络、事件通知系统和分布式查询处理。DHTs通过提供通用键/值命名而不是强加结构化键(例如，DNS中的分层名称)来简化这些系统的结构。这些系统是早期的原型，但它们表明DHTs对于分布式应用程序可能像普通哈希表对于程序一样有用。第二个猜想依赖于从许多相互不信任的节点创建健壮、安全和自组织基础设施的技术。我们在健壮的分布式哈希表设计方面的初步工作让我们相信，这样的技术是触手可及的。我们建议的大部分研究将致力于深入研究这些技术，明确的目标是为基础设施制定一个健全和连贯的设计。为了调查我们的设计在现实世界中的行为，我们将创建一个大规模的开放试验台，我们将在其中分发我们的基础设施软件、一些使能库和一些关键的引人注目的应用程序。除了它对创建分布式应用程序的影响外，我们的研究计划还将在教育和推广方面受益。鉴于它们目前的重要性，安全性、健壮性和分布式系统的设计应该成为本科计算机科学教育的中心话题。为此，我们正在计划一门新的跨学科课程来解决这些问题，并在本科课程序列的早期将它们放在更突出的位置。我们的试验床和研究议程也是鼓励传统上不参与网络和系统研究的组织参与的一个很好的工具。参与试验台只需要很少的成本(一台PC和互联网连接)，以及最低水平的系统专业知识和远见。此外，由于这些材料与许多学生熟悉的P2P系统密切相关，该项目可能会吸引通常不会对这一领域的研究感兴趣的学生。基于这一前提，我们计划对非研究型本科院校中代表性不足的人群开展积极的外展计划。