SI2-SSE: Collaborative Research: Software Elements for Transfer and Analysis of Large-Scale Scientific Data

随着科学越来越受数据驱动，随着数据量和速度的增加，许多领域的科学进步只有在关键的“大数据”问题得到解决的情况下才是可行的-更重要的是，科学家可以随时获得嵌入这些解决方案的软件工具。特别是，当前数据密集型科学研究工作面临的主要挑战 尽管数据集大小继续快速增长，但无论是在网络带宽中， 并行机的存储容量， 存储器访问速度， 和磁盘带宽也在以同样的速度增长。在俄亥俄州州立大学最近的研究基础上，其中包括自动数据虚拟化，科学数据的索引方法， 和一种新颖的基于位向量的采样方法，该项目的目标是充分开发，传播，部署和支持强大的软件元素，以解决数据传输和分析中的挑战。 已经在俄亥俄州开发的原型正在扩展到两个强大的 软件元素：软件元素1： GridPFTP（网格部分文件传输协议）： GridFTP的扩展：GridFTP的扩展，允许用户指定要传输的文件的子集，避免不必要的整个文件的传输。软件元素2：用于Paraview和VTK的NetCDF和HDF 5的并行读取器：用于NetCDF和HDF 5的数据子集和采样工具，在I/O级别并行执行数据选择和采样。该项目影响许多科学领域， 也就是说，任何涉及 大（和不断增长的）数据集大小和数据传输和/或可视化的需求。 该项目还有助于“大数据”方面的计算机科学研究，包括科学（基于阵列的）数据库和可视化。 另一个贡献将是为大数据处理和分析准备广泛的科学和工程研究社区。

As science has become increasingly data-driven, and as data volumes and velocity are increasing, scientific advance in many areas will only be feasible if critical `big-data' problems are addressed - and even more importantly, software tools embedding these solutions are readily available to the scientists. Particularly, the major challenge being faced by current data-intensive scientific research efforts is that while the dataset sizes continue to grow rapidly, neither among network bandwidths, memory capacity of parallel machines, memory access speeds, and disk bandwidths are increasing at the same rate.Building on top of recent research at Ohio State University, which includes work on automatic data virtualization, indexing methods for scientific data, and a novel bit-vectors based sampling method, the goal of this project is to fully develop, disseminate, deploy, and support robust software elements addressing challenges in data transfers and analysis. The prototypes that have been already developed at Ohio State are being extended into two robust software elements: Software Element 1: GridPFTP (Grid Partial-File Transport Protocol): An Extention to GridFTP: an extention of GridFTP that allows users to specify a subset of the file to be transferred, avoiding unnecessary transfer of the entire file.Software Element 2: Parallel Readers for NetCDF and HDF5 for Paraview and VTK: Data subsetting and sampling tools for NetCDF and HDF5 that perform data selection and sampling at the I/O level, and in parallel.This project impacts a number of scientific areas, i.e., any area that involves big (and growing) dataset sizes and need for data transfers and/or visualization. This project also contributes to computer science research in `big data', including scientific (array-based) databases, and visualization. Another contribution will be towards preparation of the broad science and engineering research community for big data handling and analytics.