Collaborative Research: SI2-SSI: Big Weather Web: A Common and Sustainable Big Data Infrastructure in Support of Weather Prediction Research and Education in Universities

合作研究：SI2-SSI：大天气网：支持大学天气预报研究和教育的通用且可持续的大数据基础设施

• 批准号: 1450195
• 负责人: Robert Fovell
• 金额: $ 24.61万
• 依托单位: SUNY at Albany
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-08-01 至 2020-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1450195&HistoricalAwards=false
• 关键词: Collaborative; Research; SI2; SSI; Big

Earth science communities need to rely on access to large and growing amounts of curated data to make progress in research and provide adequate education. Existing infrastructures pose significant barriers to this access, especially for small to mid-size research groups and primarily undergraduate institutions: cloud services disappear when funding runs out to pay for them and therefore do not provide the long-term availability required for curated data. Similarly, in-house IT infrastructure is maintenance-intensive and requires dedicated resources for which long-term funding is often unavailable. The goal of the Big Weather Web is to make in-house IT infrastructure affordable by combining the application of three recent technologies: virtualization, federated smart storage, and big data management. Virtualization allows push-button deployment and maintenance of complex systems, smart storage provides automatic, community-wide data availability guarantees, and big data management allows for easy curation of data and its products. The combination of these three technologies allows communities to create a standard community-specific computational environment and efficiently refine it with minimal repetition of work, introducing a high degree of reproducibility in research and education. This reproducibility accelerates learning and amplifies everyone?s contribution. Due to virtualization, it can easily take advantage of cloud services whenever they become available, and it can run on in-house IT infrastructure using significantly reduced maintenance resources. The Big Weather Web will be developed in the context of numerical weather prediction with the expectation that the resulting infrastructure can be easily adapted to other data-intensive scientific communities.The volume, variety, and velocity of scientific data generated is growing exponentially. Small to mid-size research groups and especially primarily undergraduate institutions (PUIs) do not have the resources to manage large amounts of data locally and share their data products globally at high availability. This lack of resources has a number of consequences in education and research that have been well-documented in recent EarthCube workshops: (1) data-intensive scientific results are not easily reproducible, whether in the context of research or education, (2) limited or non-existent availability of intermediate results causes a lot of unnecessary duplication of work and makes learning curves unnecessarily steep, and consequently (3) scientific communities of practice are falling behind technological innovations. This Big Weather Web project focuses on the numerical weather prediction community. Numerical weather models produce terabytes of output per day, comprising a wealth of information that can be used for research and education, but this amount of data is difficult to transfer, store, or analyze for most universities. The Big Weather Web addresses this situation with the design, implementation, and deployment of "nuclei," which are shared artifacts that enable reliable and efficient access and sharing of data, encode best practices, and are sustainably maintained and improved by the community. These nuclei use existing and well-established technologies, but the integration of these technologies will significantly reduce the resource burden mentioned above. Nucleus 1 is a large ensemble distributed over seven universities. Nucleus 2 is a common storage, linking, and cataloging methodology implemented as an appliance-like Data Investigation and Sharing Environment (DISE) that is extremely easy to maintain and that automatically ensures data availability and safety. Nucleus 3 is a versioned virtualization and container technology for easy deployment and reproducibility of computational environments. Together, these nuclei will advance discovery and understanding through sharing of data products and methods to replicate scientific results while promoting teaching, training, and learning by creating a shared environment for scientific communities of practice. These shared environments are particularly important for underrepresented groups who otherwise have limited access to knowledge that is primarily propagated by social means. Our approach is a significant step towards improving reproducibility in the complex computational environments found in many scientific communities.

地球科学界需要依靠获取大量且数量不断增加的精选数据来取得研究进展和提供适当的教育。现有的基础设施对这种访问构成了巨大的障碍，特别是对于中小型研究小组和主要是本科院校：当资金耗尽时，云服务就会消失，因此无法提供经过管理的数据所需的长期可用性。同样，内部IT基础设施是维护密集型的，需要专门的资源，而长期资金往往是不可用的。Big Weather Web的目标是通过结合三项最新技术的应用，使内部IT基础设施变得负担得起：虚拟化、联合智能存储和大数据管理。虚拟化允许按键部署和维护复杂系统，智能存储提供全社区范围的自动数据可用性保证，大数据管理允许轻松管理数据及其产品。这三种技术的结合使社区能够创建标准的社区特定的计算环境，并以最少的重复工作高效地改进它，在研究和教育中引入高度的重复性。这种可重复性加速了学习，放大了每个人对S的贡献。由于虚拟化，它可以在云服务可用时轻松利用它们，并且可以在内部IT基础设施上运行，使用的维护资源大大减少。大天气网将在数值天气预报的背景下开发，期望由此产生的基础设施可以很容易地适应其他数据密集型科学社区。产生的科学数据的数量、种类和速度呈指数级增长。中小型研究小组，特别是主要是本科院校(PUI)没有资源在本地管理大量数据，并在全球范围内以高可用性共享他们的数据产品。资源的缺乏在教育和研究方面造成了一些后果，在最近的EarthCube研讨会上得到了很好的记录：(1)数据密集的科学成果不容易复制，无论是在研究还是教育方面，(2)中间结果的可获得性有限或根本不存在，导致大量不必要的重复工作，使学习曲线变得不必要地陡峭，因此(3)科学实践界正在落后于技术创新。这个Big Weather Web项目的重点是数值天气预报社区。数值天气模型每天产生数TB的输出，其中包含大量可用于研究和教育的信息，但对于大多数大学来说，这些数据很难传输、存储或分析。Big Weather Web通过设计、实现和部署“核”来解决这种情况，“核”是共享的构件，支持可靠而高效的数据访问和共享，编码最佳实践，并由社区可持续地维护和改进。这些核使用现有和成熟的技术，但这些技术的整合将显著减少上述资源负担。原子核1是一个分布在七所大学的大型合奏。Negus 2是一种常见的存储、链接和编目方法，作为一个类似设备的数据调查和共享环境(DISE)实施，非常易于维护，并自动确保数据的可用性和安全性。Negus 3是一种版本化的虚拟化和容器技术，可轻松部署和重现计算环境。这些核心将通过共享数据产品和方法来促进发现和理解，以复制科学成果，同时通过为科学实践社区创造共享环境来促进教学、培训和学习。这些共享环境对代表性不足的群体尤其重要，否则他们获得主要通过社会手段传播的知识的机会有限。我们的方法是朝着提高在许多科学界发现的复杂计算环境中的重复性迈出了重要的一步。