CRII: ACI: Accelerating In-Situ Scientific Data Analysis Using Software-Defined Storage Resource Enclaves

数据密集型知识发现要求科学应用程序与分析和可视化代码同时运行，并在现场执行，以及时进行输出检查和知识提取。因此，用于大型科学数据分析的输入/输出(I/O)管道可能又长又复杂，因为它们包含跨越高性能计算系统I/O堆栈的不同层的许多分析“阶段”。任何I/O层的性能限制都可能导致I/O瓶颈，从而导致比预期更长的端到端I/O延迟。在该项目中，PI的目标是在系统级别实施一种名为软件定义存储资源包围区(SIREN)的新型数据管理基础设施，以实施决定I/O管道性能的端到端策略。该项目开发的技术的交叉性质可以帮助大型科学数据分析利用超级计算机上内存和存储设备的全部能力。该项目将促进温哥华华盛顿州立大学研究生水平的数据密集型计算课程的发展，并有助于本科生、女性学生和代表性不足学生的教育。因此，这项研究与NSF促进科学进步、促进国家繁荣和福祉的使命是一致的。该项目的技术目标有三个。首先，SIREN旨在允许管理员为Enclaves设置分配，以管理属于同一I/O管道的一组应用程序。其次，它打算同时考虑存储设备的特性(例如，SSD的读/写容量差异和磁盘对数据局部性的性能敏感性)，在多层I/O堆栈上强制执行I/O策略(例如，按比例共享)，以实现最佳性能。第三，PI旨在解决特定于存储的实施问题，包括设计用户友好的界面、Enclave命名和解析、元数据管理、故障处理和准入控制。SIREN的引入可以从根本上改变超级计算机上广泛使用的数据暂存服务的执行模式。这也将有助于理解数据转移过程中I/O管道在外部I/O干扰下的性能特征。

Data intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes, executing in situ for timely output inspection and knowledge extraction. Consequently, the Input/Output (I/O) pipelines for large scientific data analysis can be long and complex because they comprise many "stages" of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer can cause an I/O bottleneck resulting in longer than expected end-to-end I/O latency. In this project, PI aims to implement a novel data management infrastructure called Software-defined Storage Resource Enclaves (SIREN) at system levels to enforce end-to-end policies that dictate an I/O pipeline's performance. The cross-cutting nature of the technologies developed in the project can help large scientific data analytics leverage the full capability of memory and storage devices on supercomputers. The project will facilitate the development of a graduate level data-intensive computing course at Washington State University Vancouver, and contribute to the education of undergraduate, female, and under-representative students. Therefore, this research aligns with the NSF mission to promote the progress of science and to advance the national prosperity and welfare.The technical objectives of the project are three-fold. First, SIREN aims to allow administrators to set allocations for enclaves to manage a group of applications that belong to the same I/O pipeline. Second, it intends to enforce I/O policies (e.g., proportional sharing) at more than one layer of I/O stacks simultaneously considering characteristics of storage devices (e.g., disparity of read/write capacity for SSDs and performance sensitivity to data locality for disks) to achieve optimal performance. Third, PI aims to solve storage-specific implementation issues, including design of user-friendly interfaces, enclave naming and resolution, metadata management, failure handling, and admission control. The introduction of SIREN can fundamentally change the execution model of data staging services widely used on supercomputers. It will also contribute to the understanding of performance characteristics of I/O pipelines under external I/O interference during data staging.