SI2-SSE: Collaborative Research: Software Elements for Transfer and Analysis of Large-Scale Scientific Data

随着科学越来越多地由数据驱动，随着数据量和速度的增加，许多领域的科学进步只有在关键的“大数据”问题得到解决的情况下才是可行的--更重要的是，嵌入这些解决方案的软件工具对科学家来说是现成的。特别是，当前数据密集型科学研究工作面临的主要挑战是，尽管数据集的大小继续快速增长，但网络带宽、并行机的内存容量、内存访问速度和磁盘带宽都没有以同样的速度增长。该项目的目标是在俄亥俄州立大学最近的研究基础上，包括自动数据虚拟化、科学数据索引方法和基于位向量的新采样方法，全面开发、传播、部署和支持健壮的软件元素，以应对数据传输和分析中的挑战。俄亥俄州立大学已经开发的原型正在扩展成两个健壮的软件元素：软件元素1：网格部分文件传输协议：对网格文件传输协议的扩展：允许用户指定要传输的文件的子集的网格文件传输协议的扩展，以避免不必要的整个文件传输。软件元素2：用于NetCDF和HDF5的并行读取器，用于Paraview和VTK：用于NetCDF和HDF5的数据子集和采样工具，在I/O级和并行地执行数据选择和采样。该项目影响许多科学领域，例如，任何涉及大型(且不断增长的)数据集规模和数据传输和/或可视化需求的领域。该项目还有助于“大数据”方面的计算机科学研究，包括科学(基于数组的)数据库和可视化。另一项贡献将是为广大科学和工程研究界为大数据处理和分析做好准备。

As science has become increasingly data-driven, and as data volumes and velocity are increasing, scientific advance in many areas will only be feasible if critical `big-data' problems are addressed - and even more importantly, software tools embedding these solutions are readily available to the scientists. Particularly, the major challenge being faced by current data-intensive scientific research efforts is that while the dataset sizes continue to grow rapidly, neither among network bandwidths, memory capacity of parallel machines, memory access speeds, and disk bandwidths are increasing at the same rate.Building on top of recent research at Ohio State University, which includes work on automatic data virtualization, indexing methods for scientific data, and a novel bit-vectors based sampling method, the goal of this project is to fully develop, disseminate, deploy, and support robust software elements addressing challenges in data transfers and analysis. The prototypes that have been already developed at Ohio State are being extended into two robust software elements: Software Element 1: GridPFTP (Grid Partial-File Transport Protocol): An Extention to GridFTP: an extention of GridFTP that allows users to specify a subset of the file to be transferred, avoiding unnecessary transfer of the entire file.Software Element 2: Parallel Readers for NetCDF and HDF5 for Paraview and VTK: Data subsetting and sampling tools for NetCDF and HDF5 that perform data selection and sampling at the I/O level, and in parallel.This project impacts a number of scientific areas, i.e., any area that involves big (and growing) dataset sizes and need for data transfers and/or visualization. This project also contributes to computer science research in `big data', including scientific (array-based) databases, and visualization. Another contribution will be towards preparation of the broad science and engineering research community for big data handling and analytics.