Collaborative Research: Building the Community for the Open Storage Network

科学界正面临着一个重大挑战，即如何处理从各种规模的研究项目--从大型设施到小型研究实验室--产生的越来越多的开放科学数据。在过去的五年里，NSF已经资助了200多个高速连接到以10-100Gbps速度运行的互联网主干。该项目的目标是开发一个高性能分布式存储系统的原型模块，以扩展现有高速互连的可用性。该项目是一个潜在的国家级开放式科学数据存储基础设施的试点，在全面规模上可以为数百个站点和数百PB的数据提供服务。与这种分布式系统相关的许多技术已经存在；这个项目中的关键挑战是社会工程：如何设计一个足够简单但强大的存储节点，它可以很容易地复制，对大学和研究项目有吸引力，易于管理，并能够支持用于大规模科学分析的各种模式？许多大学都有数据密集型科学的几个必备组件--规模合理的计算集群，几PB的存储空间，甚至是高速连接--但执行数据密集型科学的分析非常痛苦和缓慢。数据永远不会在需要的时候出现，大型存储系统经常会出现故障，尽管有大量的RAID配置，而且以最高的网络速度将数据从一个磁盘移动到另一个磁盘仍然需要复杂的技能。该项目通过大数据中心提供了一个广泛的社区接受，一个独特的技能、设施和科学挑战的组合，以测试、评估和部署不同的硬件和软件组合，这些组合可以用于设计一个规模大得多的全国范围的系统。目标是为各种测试科学项目设计和运行详细的基准，这些项目需要数据传输、数据处理和海量计算的不同组合，并使用结果来设计和构建低成本、可扩展的千万亿级设备，包括廉价的硬件节点和简单的软件堆栈，可以在许多大学、超级计算机中心和大型NSF设施中复制。拟议中的系统可能会成为NSF在高端计算和快速网络方面现有投资的巨大乘数。它还可以加快全国数据存储标准化的步伐。公开的、开放的数据产品，经常在NSF提案的数据管理计划的末尾讨论，可以找到一个易于使用的家。各种教育项目可以简单地依靠强大的存储基础设施和简单的API，并为教育社区构建各种交付服务。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The scientific community is facing a major challenge dealing with the increasing amount of open scientific data emerging from research projects on all scales-- from large facilities to small research labs. Over the last five years the NSF has funded more than 200 high-speed connections to the Internet-2 backbone operating at 10-100Gbps speeds. The goal of this project is to develop a prototype module for a high performance distributed storage system that extends the usability of the existing high-speed interconnects. This project is a pilot for a potential national-scale storage infrastructure for open scientific data, which at full scale could serve hundred sites and many hundreds of Petabytes. Many of the technologies associated with such a distributed system already exist; the key challenge in this project is social engineering: how can one design a simple enough yet robust storage node that can be easily replicated, is attractive for universities and research projects to adopt, is easy to manage and can support the various patterns for large scale scientific analyses?Many universities have several of the necessary pieces for Data Intensive Science in place-- reasonably sized computing clusters, a few PB of storage and even a high-speed connection-- yet performing the analyses of data intensive science is very painful and slow. Data is never there when needed, large storage systems often fail despite having massive RAID configurations, and moving data from disk-to-disk at the full network speed still requires complex skills. The project offers a broad community buy-in through the Big Data Hubs, a unique combination of skills, facilities and science challenges to test, evaluate and deploy different hardware and software combinations that can be used in the design of a much larger, national-scale system. The goal is to design and run detailed benchmarks for various test science projects requiring different combinations of data transfer, data processing and massive compute, and use the results to design and build a low-cost, scalable petascale appliance including inexpensive hardware nodes and a simple software stack that can be replicated across many universities, supercomputer centers and large NSF facilities. The proposed system could become an enormous multiplier on the existing NSF investments in high end computing and fast networks. It could also accelerate the pace of standardization of data storage across the nation. The public, open data products, often discussed in the Data Management Plans at the end of NSF proposals could find an easy-to-use home. Various educational projects could simply rely upon a robust storage infrastructure with a simple API, and build a variety of delivery services for the educational community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.