Collaborative Research: Building the Community for the Open Storage Network

科学界正面临着一个重大挑战，即处理从各种规模的研究项目-从大型设施到小型研究实验室-产生的越来越多的公开科学数据。在过去的五年里，NSF已经资助了200多个高速连接到Internet-2骨干网，以10- 100 Gbps的速度运行。本项目的目标是开发一个高性能分布式存储系统的原型模块，扩展现有高速互连的可用性。该项目是一个潜在的国家级开放科学数据存储基础设施的试点项目，该基础设施全面可为数百个站点和数百PB提供服务。 与这种分布式系统相关的许多技术已经存在;该项目的关键挑战是社会工程：如何设计一个足够简单但强大的存储节点，可以很容易地复制，对大学和研究项目有吸引力，易于管理，可以支持大规模科学分析的各种模式？许多大学都有数据密集型科学的几个必要部分-合理大小的计算集群，几PB的存储甚至是高速连接-但执行数据密集型科学的分析是非常痛苦和缓慢的。数据在需要时永远不会出现，大型存储系统尽管有大规模的RAID配置，但经常会出现故障，并且以最快的网络速度将数据从磁盘移动到磁盘仍然需要复杂的技能。该项目通过大数据中心提供了广泛的社区购买，这是一种独特的技能，设施和科学挑战的组合，用于测试，评估和部署不同的硬件和软件组合，可用于设计更大的国家级系统。其目标是为需要不同的数据传输、数据处理和大规模计算组合的各种测试科学项目设计和运行详细的基准测试，并使用结果设计和构建一个低成本、可扩展的千万亿次设备，包括廉价的硬件节点和简单的软件堆栈，可以在许多大学、超级计算机中心和大型NSF设施中复制。拟议中的系统可以成为NSF在高端计算和快速网络方面现有投资的巨大倍增器。它还可以加快全国数据存储标准化的步伐。公共的、开放的数据产品，经常在NSF提案结尾的数据管理计划中讨论，可以找到一个易于使用的家。各种教育项目可以简单地依赖于具有简单API的强大存储基础架构，并为教育社区构建各种交付服务。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The scientific community is facing a major challenge dealing with the increasing amount of open scientific data emerging from research projects on all scales-- from large facilities to small research labs. Over the last five years the NSF has funded more than 200 high-speed connections to the Internet-2 backbone operating at 10-100Gbps speeds. The goal of this project is to develop a prototype module for a high performance distributed storage system that extends the usability of the existing high-speed interconnects. This project is a pilot for a potential national-scale storage infrastructure for open scientific data, which at full scale could serve hundred sites and many hundreds of Petabytes. Many of the technologies associated with such a distributed system already exist; the key challenge in this project is social engineering: how can one design a simple enough yet robust storage node that can be easily replicated, is attractive for universities and research projects to adopt, is easy to manage and can support the various patterns for large scale scientific analyses?Many universities have several of the necessary pieces for Data Intensive Science in place-- reasonably sized computing clusters, a few PB of storage and even a high-speed connection-- yet performing the analyses of data intensive science is very painful and slow. Data is never there when needed, large storage systems often fail despite having massive RAID configurations, and moving data from disk-to-disk at the full network speed still requires complex skills. The project offers a broad community buy-in through the Big Data Hubs, a unique combination of skills, facilities and science challenges to test, evaluate and deploy different hardware and software combinations that can be used in the design of a much larger, national-scale system. The goal is to design and run detailed benchmarks for various test science projects requiring different combinations of data transfer, data processing and massive compute, and use the results to design and build a low-cost, scalable petascale appliance including inexpensive hardware nodes and a simple software stack that can be replicated across many universities, supercomputer centers and large NSF facilities. The proposed system could become an enormous multiplier on the existing NSF investments in high end computing and fast networks. It could also accelerate the pace of standardization of data storage across the nation. The public, open data products, often discussed in the Data Management Plans at the end of NSF proposals could find an easy-to-use home. Various educational projects could simply rely upon a robust storage infrastructure with a simple API, and build a variety of delivery services for the educational community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.