CRII: ACI: Accelerating In-Situ Scientific Data Analysis Using Software-Defined Storage Resource Enclaves

数据密集型知识发现需要科学应用程序与分析和可视化代码同时运行，在原位执行以及时进行输出检查和知识提取。因此，用于大型科学数据分析的输入/输出（I/O）流水线可能是长且复杂的，因为它们包括跨高性能计算系统的I/O栈的不同层的分析的许多“阶段”。任何I/O层的性能限制都可能导致I/O瓶颈，从而导致比预期更长的端到端I/O延迟。在这个项目中，PI的目标是在系统级别实现一种称为软件定义的存储资源飞地（SIREN）的新型数据管理基础设施，以强制执行决定I/O管道性能的端到端策略。该项目中开发的技术的跨领域性质可以帮助大型科学数据分析利用超级计算机上内存和存储设备的全部功能。该项目将促进华盛顿州立大学温哥华研究生级数据密集型计算课程的开发，并为本科生、女生和代表性不足的学生的教育做出贡献。 因此，本研究符合美国国家科学基金会的使命，以促进科学的进步和促进国家的繁荣和福利。首先，SIREN旨在允许管理员为飞地设置分配，以管理属于同一I/O管道的一组应用程序。其次，它打算强制执行I/O策略（例如，比例共享）同时考虑存储设备的特性（例如，SSD的读/写容量差异以及磁盘对数据局部性的性能敏感度），以实现最佳性能。第三，PI旨在解决特定于存储的实现问题，包括用户友好界面的设计、安全区命名和解析、元数据管理、故障处理和准入控制。SIREN的引入可以从根本上改变在超级计算机上广泛使用的数据分段服务的执行模式。这也将有助于理解在数据分级期间外部I/O干扰下的I/O管道的性能特性。

Data intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes, executing in situ for timely output inspection and knowledge extraction. Consequently, the Input/Output (I/O) pipelines for large scientific data analysis can be long and complex because they comprise many "stages" of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer can cause an I/O bottleneck resulting in longer than expected end-to-end I/O latency. In this project, PI aims to implement a novel data management infrastructure called Software-defined Storage Resource Enclaves (SIREN) at system levels to enforce end-to-end policies that dictate an I/O pipeline's performance. The cross-cutting nature of the technologies developed in the project can help large scientific data analytics leverage the full capability of memory and storage devices on supercomputers. The project will facilitate the development of a graduate level data-intensive computing course at Washington State University Vancouver, and contribute to the education of undergraduate, female, and under-representative students. Therefore, this research aligns with the NSF mission to promote the progress of science and to advance the national prosperity and welfare.The technical objectives of the project are three-fold. First, SIREN aims to allow administrators to set allocations for enclaves to manage a group of applications that belong to the same I/O pipeline. Second, it intends to enforce I/O policies (e.g., proportional sharing) at more than one layer of I/O stacks simultaneously considering characteristics of storage devices (e.g., disparity of read/write capacity for SSDs and performance sensitivity to data locality for disks) to achieve optimal performance. Third, PI aims to solve storage-specific implementation issues, including design of user-friendly interfaces, enclave naming and resolution, metadata management, failure handling, and admission control. The introduction of SIREN can fundamentally change the execution model of data staging services widely used on supercomputers. It will also contribute to the understanding of performance characteristics of I/O pipelines under external I/O interference during data staging.