SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data

许多地球物理模拟和几乎所有的气候模型都将其结果存档在数百到数千个文件中，存储空间从千兆字节到千兆字节不等。在地球物理学家能够解释这些数据之前，必须将其减小到可管理的大小(例如，通过对其求平均)和/或将其传输到指定位置(例如，台式计算机)。目前，研究人员经常通过互联网从多个地点复制原始数据，以开始他们的数据分析。该项目正在开发科学数据运算符(SDO)：这是一种高效的软件，允许研究人员以并行、远程的方式执行典型的数据简化和分析，而不会浪费时间和网络带宽。SDO软件结合了分布式和共享内存编程、客户端-服务器体系结构和开源开发技术。作为概念验证，将对国家边界内外的多个NCAR CCSM IPCC气候评估模拟(每个模拟约为1TB)进行分布式分析。该项目还为一名研究生开展分布式气候分析的论文研究提供了支持。在气候建模和分析之外，SDO将产生三个主要影响：(1)通过减少分析、发现和发布新结果的时间来增加大型地球物理数据集的价值；(2)揭示处理分布式地球物理数据时的任何关键带宽、I/O和客户端/服务器瓶颈；以及(3)以类似于地球物理领域的方式改进对不断增长的生物信息学数据集，特别是基因表达数据的分析。SDO是基于国际上成功的NetCDF运营商(NCO)软件的自由软件。包括该软件在内的项目成果将可通过项目网站查阅。

Many geophysical simulations and nearly all climate models archive their results in hundreds to thousands of files spanning gigabytes to terabytes of storage. Before the geophysicist can interpret these data, it must be reduced to a manageable size (e.g., by averaging it) and/or transported to a specified location (e.g., the desktop computer). Currently researchers often copy raw data over the Internet from multiple locations to commence their data analysis. This project is developing the Scientific Data Operators (SDO): efficient software that lets researchers perform typical data reduction and analysis in parallel, remotely, without wasting time and network bandwidth.The SDO software combines distributed and shared memory programming, client-server architecture, and Open Source development techniques. As proof-of-concept, a distributed analysis of multiple NCAR CCSM IPCC climate assessment simulations (each is about a terabyte) within and across national boundaries will be performed. This project also provides support for a graduate student to carry out dissertation research in distributed climate analysis. Outside climate modeling and analysis, SDO will have three main impacts: (1) increase the value of large geophysical datasets by decreasing the time to analyze, discover, and publish new results; (2) reveal any critical bandwidth, I/O, and client/server bottlenecks in processing distributed geophysical data; and (3) improve analysis of growing bioinformatics data sets, especially gene expression data, in ways similar to the geophysics domain. SDO is free software based on the internationally successful netCDF Operator (NCO) software. The project results including this software will be accessible via the project web site.