RI: Toward a Petabyte Storage Infrastructure

计算的新兴方向可能会大大增加存储需求。 存储的传统驱动因素-大型数据库、科学数据集以及一般的个人和机构信息需求-继续增长，而新的因素带来了更大、更复杂的需求。 这些新元素包括“收集的”材料，即，文本文档、图像和视频的数字库，以及来自与物理世界集成的广泛分布且多种多样的网络传感器的流数据。 随着CMOS技术的不断小型化，这最后一类可能会增加规模的速度类似于摩尔定律，而不是由人类的创造性活动的步伐的限制，它将进入更高层次的处理和storage.In预期这些发展，我们建议提供一套存储服务，有针对性地支持研究项目深刻的当前和未来的存储需求。 特别是，我们建议1。通过使用商品组件和无磁带（即，磁盘到磁盘）备份2.为科学集群计算设施提供快速磁盘缓存，以支持需要大量本地存储的大型计算3。提供几千个环境传感器，以支持收集和存储大型传感器衍生环境数据集的实验。对象存储实验是该项目的核心。 如今，为用户提供存储的成本主要是管理成本，如果任由其自行决定，情况只会变得更糟。 例如，IT支持组织报告说，存储硬件的成本仅占提供存储的满载成本的10-15%，而这主要是由备份成本决定的。 此外，用于可靠地分配字节的高端服务器比一般的低端服务器贵得多，我们的目标是提供一种通用的存储服务，提供有吸引力的“企业”质量，但成本与硬件的进步相适应，并且可以发展到潜在的更大规模。 我们计划通过将对象存储在多个廉价的服务器上来解决可用性和持久性问题，其中一些服务器在地理上分布，并且没有磁带或大量操作人员。 我们相信，最近在许多领域的发展，特别是版本控制、位置无关路由和拜占庭协议，将使这样的解决方案变得可行。我们计划使用通过这笔赠款开发的设施来支持与社会利益信息技术研究中心（CITRIS）有关的项目，一个大学-工业-国家的伙伴关系，旨在创造和利用信息技术来解决社会最关键的需求。 这些项目包括数字图书馆应用、分析交通视频以了解各种交通事件数据库的特征签名、集群计算应用（包括在自然图像中寻找物体之间的边界、探测来自天体物理点源的高能中微子以及自主飞行器支持问题）常春藤“智能建筑”测试环境（一项研究基础设施，包括校园内外多座建筑物内的固定和移动的微粒传感器网络，并试验建筑物的节能运行，“需求响应型”电力管理系统、消防安全和备灾以及结构完整性监测），以及电子文化地图集倡议，这是一个由近800个附属机构和120多个项目组成的全球合作项目，支持将联合收割机全球绘图、图像和文本结合在一起的应用程序，它为学者和其他用户提供了数字研究资源，用于视觉呈现来自多个学科的数据的复杂组合。

Emerging directions in computing are likely to dramatically increase storage demands. Traditional drivers for storage-large databases, scientific data sets and general personal and institutional information needs-continue to increase, while new elements impose even greater, more complex needs. Such new elements include "collected" materials, i.e., digital libraries of textual documents, images, and video, as well as streaming data from widely distributed and widely varied networked sensors integrated with the physical world. With the continued miniaturization of CMOS technology, this last category may increase in scale at a rate similar to Moore's law, rather than being limited by the pace of human creative activities, and it will feed into higher level processing and storage.In anticipation of these developments, we propose to provide a set of storage services targeted at supporting research projects with profound current and future storage needs. In particular, we propose1. experimenting with providing low-cost petabyte-scale generic "object storage", via the use of commodity components and tape-less (i.e., disk-to-disk) backup2. providing a fast disk cache for a scientific cluster computing facility, to support large computations that require significant local storage3. providing a few thousand environmental sensors to support experimenting with collecting and storing large sensor-derived environmental data sets.The object storage experiment is central to this project. The cost of providing storage to users today is dominated by administrative costs, and, left to its own devices, the situation will only grow worse. For example, IT support organizations report that the cost of storage hardware accounts for only 10-15% of the fully loaded cost of providing storage, which is dominated by the cost of backups. In addition, the higher-end servers used to dispense bytes reliably are much more expensive than generic low-end servers.Our goal is to provide a generic storage service that offers attractive "enterprise" qualities, but at a cost that tracks advances in hardware, and which can grow to a potentially much larger scale. We plan to address the issues of availability and durability by storing objects on multiple, inexpensive servers, some geographically distributed, and without tapes or extensive operations personnel. We believe that recent developments in a number of areas, particularly versioning, location-independent routing, and Byzantine Agreement, will make such a solution viable.Broader Impact: We plan to use the facilities developed through this grant to support projects affiliated with the Center for Information Technology Research in the Interest of Society (CITRIS), a university-industry-state partnership that aims to create and harness information technology to tackle society's most critical needs. Included among these projects are digital library applications, analyzing traffic videos to learn the characteristic signatures of various traffic incidents databases, cluster computing applications (including finding boundaries between objects in natural images, detecting high-energy neutrinos from astrophysical point sources, and autonomous aerial vehicles support problems), the Ivy "Smart Buildings" test environment (a research infrastructure comprising sensor networks of fixed and mobile motes in a number of buildings on and off campus, and experimenting with energy efficient building operation, "demand responsive" electricity management systems, fire safety and disaster preparedness, and structural integrity monitoring), and the Electronic Cultural Atlas Initiative, a global collaboration of nearly 800 affiliates and over 120 project supporting applications that combine global mapping, imagery, and texts, and which provides scholars and other users with digital research resources for the visual presentation of complex combinations of data from multiple disciplines.