SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data

SEI(GEO)：针对兆级地球物理数据的高效分布式交互式和批量分析而优化的科学数据运算符

• 批准号: 0431203
• 负责人: Charles Zender
• 金额: $ 59.44万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2004
• 资助国家: 美国
• 起止时间: 2004-09-01 至 2008-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0431203&HistoricalAwards=false
• 关键词: SEI; GEO; Scientific; Data; Operators

Many geophysical simulations and nearly all climate models archive their results in hundreds to thousands of files spanning gigabytes to terabytes of storage. Before the geophysicist can interpret these data, it must be reduced to a manageable size (e.g., by averaging it) and/or transported to a specified location (e.g., the desktop computer). Currently researchers often copy raw data over the Internet from multiple locations to commence their data analysis. This project is developing the Scientific Data Operators (SDO): efficient software that lets researchers perform typical data reduction and analysis in parallel, remotely, without wasting time and network bandwidth.The SDO software combines distributed and shared memory programming, client-server architecture, and Open Source development techniques. As proof-of-concept, a distributed analysis of multiple NCAR CCSM IPCC climate assessment simulations (each is about a terabyte) within and across national boundaries will be performed. This project also provides support for a graduate student to carry out dissertation research in distributed climate analysis. Outside climate modeling and analysis, SDO will have three main impacts: (1) increase the value of large geophysical datasets by decreasing the time to analyze, discover, and publish new results; (2) reveal any critical bandwidth, I/O, and client/server bottlenecks in processing distributed geophysical data; and (3) improve analysis of growing bioinformatics data sets, especially gene expression data, in ways similar to the geophysics domain. SDO is free software based on the internationally successful netCDF Operator (NCO) software. The project results including this software will be accessible via the project web site.

许多地球物理模拟和几乎所有的气候模型都将其结果存档在数百到数千个文件中，存储空间从千兆字节到兆兆字节不等。在数据分析师能够解释这些数据之前，必须将其缩减到可管理的大小（例如，通过对其求平均）和/或传送到指定位置（例如，台式电脑）。目前，研究人员经常通过互联网从多个位置复制原始数据，以开始他们的数据分析。该项目正在开发科学数据操作员（SDO）：高效的软件，使研究人员能够并行地、远程地执行典型的数据简化和分析，而不浪费时间和网络带宽。SDO软件结合了分布式和共享内存编程、客户端-服务器架构和开源开发技术。作为概念验证，将对多个NCAR CCSM IPCC气候评估模拟（每个大约是一个TB）进行分布式分析。本计画亦提供一位研究生进行分布式气候分析之论文研究之支援。在气候建模和分析之外，SDO将产生三个主要影响：（1）通过减少分析、发现和发布新结果的时间来增加大型地球物理数据集的价值;（2）揭示处理分布式地球物理数据时的任何关键带宽、I/O和客户端/服务器瓶颈;以及（3）以类似于生物物理学领域的方式改进对不断增长的生物信息学数据集，特别是基因表达数据的分析。SDO是基于国际上成功的netCDF Operator（NCO）软件的自由软件。 包括该软件在内的项目成果可通过项目网站查阅。