CSR: Small: Collaborative Research:System Research on Persistent High-Dimensional Data Access and Its Application to Semiclassical Molecular Dynamics Simulation

高维数据出现在许多科学研究中。对驻留在磁盘上的大型高维数据集的高频访问对高访问效率提出了I/O和算法挑战。这一方案的研究人员面临着来自半经典分子动力学研究的高维数据，这些数据在模拟过程中需要以非常高的频率重复访问，并且在一些模拟研究中，数据集可能远远大于存储容量。该项目通过提出一个用于快速访问驻留在存储中的高维数据的中间件系统来应对大型高维数据集的挑战，该中间件系统具有用于数据组织、索引、搜索和I/O延迟隐藏的易于使用的API。为了能够以较低的存储空间实现快速搜索，研究了各种索引结构，并提出了一种新的局部性保留树(LPT)。此外，该项目在动力学研究中利用特定的数据访问模式，通过与I/O-S的计算重叠来隐藏I/O延迟，这是通过一小组易于使用的编程构造实现的，这些编程构造用于正在开发的非阻塞读/写操作，以进一步减少数据访问时间。开发的非阻塞API与硬盘驱动器(HDD)、混合固态驱动器(SSD)和HDD兼容。此外，开发了一个基于SSD的I/O加速器，并将其与非阻塞API集成。该项目解决了访问大型高维数据集的重要而具有挑战性的问题，其应用超出了半经典分子动力学模拟的范围。调查人员将利用研究努力来激励和教育学生。为了扩大对项目参与者的影响，将利用从研究中获得的知识、见解和材料来加强课程，并仔细改编为课程材料和课程项目。

High dimensional data arise in many scientific studies. High-frequency accesses of large disk-resident high dimensional datasets pose both I/O and algorithmic challenges for high access efficiency. Investigators of this proposal are faced with high dimensional data from semiclassical molecular dynamics studies that need to be accessed repeatedly at a very high frequency during the simulation, and in some simulation studies the datasets can be far larger than the memory capacity. The project addresses the challenge of large high-dimensional datasets by proposing a middleware system for fast access of storage-resident high dimensional data, with easy-to-use APIs for data organizing, indexing, search, and I/O latency hiding. In order to enable fast searches with low memory footprint, various indexing structures are explored and a new locality preserving tree (LPT) is proposed. Furthermore, the project exploits specific data access patterns in dynamics studies for I/O latency hiding through overlapping of computations with I/O?s, enabled by a small set of easy-to-use programming constructs for non-blocking read/write operations that are under development for further reduction of data access time. The developed non-blocking APIs are compatible with hard disk drives (HDDs) and hybrid solid state drives (SSDs) and HDDs. In addition, an SSD-based I/O accelerator is developed and integrated with non-blocking APIs.This project tackles important and challenging problems for accessing large high-dimensional data sets, which has applications beyond semiclassical molecular dynamics simulations. The investigators will leverage the research efforts to motivate and educate students. To broaden the impact beyond project participants, courses will be enhanced with knowledge, insights, and materials obtained from the research, with careful adaptation as course materials and course projects.