HECURA: Microdata Storage Systems for High-End Computing

本研究计划旨在了解和开发微数据存储系统，这是一项在许多应用战神都需要的技术，包括基因组处理和雷达知识形成。 微数据存储系统被设计为能够很好地处理小文件（微文件）和大文件（宏文件）。今天的文件系统被优化为阅读和写入大块数据，但它们在处理大量微数据时表现不佳。研究集中在三种有前途的技术上： - 微数据存储结构，例如缓冲存储库B树，它可以将微文件的插入和范围查询的性能提高到传统B树的数量级，同时仍然保持宏文件的高性能。 - 缓存无关数据结构，它提供文件组织的被动自调优，实际上可能优于磁盘文件系统的调优缓存感知数据结构。 - 基于虚拟内存的事务性内存，允许程序员以简单的方式实现复杂的文件结构，同时提供无锁编程和自动崩溃恢复。研究人员使用基准测试，如DARPA HPC SSCA#3基准测试（他们开发的仅I/O版本），以评估微数据存储系统对高端计算的影响。 研究人员还在开发关于微数据存储系统的课程材料，这些材料将在麻省理工学院开放课程计划http://ocw.mit.edu下免费提供。

This research project is aimed at understanding and developing microdata storage systems, a technology which is needed for many application ares, including genome processing and radar knowledge formation. Microdata storage systems are designed to perform well for small files (microfiles), as well as for large files (macrofiles).Today's filesystems are optimized for reading and writing data in large blocks, but they perform poorly when dealing with large volumes of microdata.The research focuses on three promising technologies: - Microdata storage structures, such as buffered repository B-trees, which can improve the performance of insertions and range queries of microfiles by orders of magnitude over traditional B-trees, while still preserving high performance on macrofiles. - Cache-oblivious data structures, which provide passive self-tuning of the file organization and may actually outperform tuned cache-aware data structures for disk file systems. - Virtual-memory-based transactional memory, which allows programmers to implement complex file structures in a straightforward manner, while providing lock-free programming and automatic crash recovery.The investigators employ benchmarks, such as the DARPA HPC SSCA#3 benchmark (an I/O-only version of which they developed), to evaluate the impact of microdata storage systems on high-end computing. The investigators are also developing course materials on microdata storage systems which will be made freely available under the MIT OpenCourseware initiative http://ocw.mit.edu.