CRII: ACI: Accelerating In-Situ Scientific Data Analysis Using Software-Defined Storage Resource Enclaves

数据密集型知识发现需要科学应用程序与分析和可视化代码同时运行，在现场执行以及时检查输出和知识提取。因此，用于大型科学数据分析的输入/输出（I/O）管道可能既长又复杂，因为它们包含跨高性能计算系统的不同I/O堆栈层的许多分析“阶段”。任何I/O层的性能限制都可能导致I/O瓶颈，从而导致比预期更长的端到端I/O延迟。在这个项目中，PI的目标是在系统级别实现一种称为软件定义存储资源enclave （SIREN）的新型数据管理基础设施，以强制执行端到端策略，从而决定I/O管道的性能。该项目中开发的技术的交叉性质可以帮助大型科学数据分析充分利用超级计算机上内存和存储设备的全部功能。该项目将促进温哥华华盛顿州立大学研究生水平数据密集型计算课程的发展，并有助于本科生、女性和代表性不足的学生的教育。因此，本研究与美国国家科学基金会推动科学进步，促进国家繁荣和福利的使命是一致的。该项目的技术目标有三个方面。首先，SIREN旨在允许管理员设置enclave的分配，以管理属于同一I/O管道的一组应用程序。其次，它打算同时考虑存储设备的特征（例如，ssd的读写容量差异和磁盘对数据位置的性能敏感性），在多个I/O堆栈层执行I/O策略（例如，比例共享），以实现最佳性能。第三，PI旨在解决特定于存储的实现问题，包括用户友好界面的设计、enclave命名和解析、元数据管理、故障处理和准入控制。SIREN的引入可以从根本上改变超级计算机上广泛使用的数据分期服务的执行模型。它还有助于理解数据暂存期间在外部I/O干扰下I/O管道的性能特征。

Data intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes, executing in situ for timely output inspection and knowledge extraction. Consequently, the Input/Output (I/O) pipelines for large scientific data analysis can be long and complex because they comprise many "stages" of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer can cause an I/O bottleneck resulting in longer than expected end-to-end I/O latency. In this project, PI aims to implement a novel data management infrastructure called Software-defined Storage Resource Enclaves (SIREN) at system levels to enforce end-to-end policies that dictate an I/O pipeline's performance. The cross-cutting nature of the technologies developed in the project can help large scientific data analytics leverage the full capability of memory and storage devices on supercomputers. The project will facilitate the development of a graduate level data-intensive computing course at Washington State University Vancouver, and contribute to the education of undergraduate, female, and under-representative students. Therefore, this research aligns with the NSF mission to promote the progress of science and to advance the national prosperity and welfare.The technical objectives of the project are three-fold. First, SIREN aims to allow administrators to set allocations for enclaves to manage a group of applications that belong to the same I/O pipeline. Second, it intends to enforce I/O policies (e.g., proportional sharing) at more than one layer of I/O stacks simultaneously considering characteristics of storage devices (e.g., disparity of read/write capacity for SSDs and performance sensitivity to data locality for disks) to achieve optimal performance. Third, PI aims to solve storage-specific implementation issues, including design of user-friendly interfaces, enclave naming and resolution, metadata management, failure handling, and admission control. The introduction of SIREN can fundamentally change the execution model of data staging services widely used on supercomputers. It will also contribute to the understanding of performance characteristics of I/O pipelines under external I/O interference during data staging.