BIGDATA: Small: DCM: Collaborative Research: An efficient, versatile, scalable, and portable storage system for scientific data containers

科学大数据集变得太大太复杂，无法装入RAM，迫使科学应用程序执行大量缓慢的磁盘和网络I/O。这种增长还使科学数据更容易因崩溃和人为错误而损坏。该项目将使用算法、数据库和存储研究的最新成果来提高标准科学数据格式的性能和可靠性。这将使科学研究更便宜、更快、更可靠、更具重复性。分层数据格式(HDF5)标准是科学数据的容器格式。它允许科学家在HDF5文件中定义和存储复杂的数据结构。不幸的是，当前标准迫使用户将所有数据对象及其元数据属性存储在一个大型物理文件中；这种混合阻碍了特定于元数据的优化。目前的存储还使用了可伸缩性很差的数据结构来处理大数据。最后，当前型号缺乏快照支持，这对于错误恢复非常重要。新的HDF5版本允许用户创建更多功能的存储插件，以控制每个对象和属性的存储策略。该项目正在开发对HDF5中的快照的支持，设计新的数据结构和算法以将现代存储设备上的HDF5数据访问扩展到BigData。该项目正在设计几个新的HDF5驱动程序：将对象映射到Linux文件系统；将对象存储在数据库中；以及访问远程Web服务器上的数据对象。这些改进是使用BigData的大规模可视化应用程序进行评估的，这些应用程序源于真实世界的科学计算。

Scientific Bigdata sets are becoming too large and complex to fit in RAM, forcing scientific applications to perform a lot of slow disk and network I/O. This growth also makes scientific data more vulnerable to corruptions due to crashes and human errors. This project will use recent results from algorithms, database, and storage research to improve the performance and reliability of standard scientific data formats. This will make scientific research cheaper, faster, more reliable, and more reproducible.The Hierarchical Data Format (HDF5) standard is a container format for scientific data. It allows scientists to define and store complex data structures inside HDF5 files. Unfortunately, the current standard forces users to store all data objects and their meta-data properties inside one large physical file; this mix hinders meta-data-specific optimizations. The current storage also uses data-structures that scale poorly for large data. Lastly, the current model lacks snapshot support, important for recovery from errors.A new HDF5 release allows users to create more versatile storage plugins to control storage policies on each object and attribute. This project is developing support for snapshots in HDF5, designing new data structures and algorithms to scale HDF5 data access on modern storage devices to Bigdata. The project is designing several new HDF5 drivers: mapping objects to a Linux file system; storing objects in a database; and accessing data objects on remote Web servers. These improvements are evaluated using large-scale visualization applications with Bigdata, stemming from real-world scientific computations.