BIGDATA: Small: DCM: Collaborative Research: An efficient, versatile, scalable, and portable storage system for scientific data containers

大数据：小型：DCM：协作研究：用于科学数据容器的高效、多功能、可扩展和便携式存储系统

• 批准号: 1251137
• 负责人: Erez Zadok
• 金额: $ 44.43万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-07-01 至 2017-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1251137&HistoricalAwards=false
• 关键词: BIGDATA; Small; DCM; Collaborative; Research

Scientific Bigdata sets are becoming too large and complex to fit in RAM, forcing scientific applications to perform a lot of slow disk and network I/O. This growth also makes scientific data more vulnerable to corruptions due to crashes and human errors. This project will use recent results from algorithms, database, and storage research to improve the performance and reliability of standard scientific data formats. This will make scientific research cheaper, faster, more reliable, and more reproducible.The Hierarchical Data Format (HDF5) standard is a container format for scientific data. It allows scientists to define and store complex data structures inside HDF5 files. Unfortunately, the current standard forces users to store all data objects and their meta-data properties inside one large physical file; this mix hinders meta-data-specific optimizations. The current storage also uses data-structures that scale poorly for large data. Lastly, the current model lacks snapshot support, important for recovery from errors.A new HDF5 release allows users to create more versatile storage plugins to control storage policies on each object and attribute. This project is developing support for snapshots in HDF5, designing new data structures and algorithms to scale HDF5 data access on modern storage devices to Bigdata. The project is designing several new HDF5 drivers: mapping objects to a Linux file system; storing objects in a database; and accessing data objects on remote Web servers. These improvements are evaluated using large-scale visualization applications with Bigdata, stemming from real-world scientific computations.

科学大数据集变得过于庞大和复杂，无法容纳在RAM中，迫使科学应用程序执行大量缓慢的磁盘和网络I/O。 这种增长也使科学数据更容易受到崩溃和人为错误的影响。 该项目将利用算法、数据库和存储研究的最新成果来提高标准科学数据格式的性能和可靠性。 这将使科学研究更便宜，更快，更可靠，更可重复。分层数据格式（HDF 5）标准是科学数据的容器格式。 它允许科学家在HDF 5文件中定义和存储复杂的数据结构。 不幸的是，当前的标准强制用户将所有数据对象及其元数据属性存储在一个大的物理文件中;这种混合阻碍了特定于元数据的优化。 当前的存储还使用了对于大数据来说伸缩性很差的数据结构。最后，当前模型缺乏快照支持，这对于从错误中恢复很重要。新的HDF 5版本允许用户创建更通用的存储插件来控制每个对象和属性的存储策略。 该项目正在开发对HDF 5中快照的支持，设计新的数据结构和算法，以将现代存储设备上的HDF 5数据访问扩展到Bigdata。该项目正在设计几个新的HDF 5驱动程序：将对象映射到Linux文件系统;将对象存储在数据库中;以及访问远程Web服务器上的数据对象。 这些改进是使用Bigdata的大规模可视化应用程序进行评估的，这些应用程序源自真实世界的科学计算。