CSR: Small: IRIS: A unified data access framework for the merging of compute-centric and data-centric storage

CSR：小型：IRIS：用于合并以计算为中心和以数据为中心的存储的统一数据访问框架

• 批准号: 1814872
• 负责人: Xian-He Sun
• 金额: $ 50万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1814872&HistoricalAwards=false
• 关键词: CSR; Small; IRIS; unified; data

As the boundary between High Performance Computing (HPC) and High-Performance Data Analytics (HPDA) continue to blur, the conventional compute-centric HPC and the newly emerged data-driven Big Data application are converging. HPC becomes more data-intensive. In the meantime, HPDA requires more computing power. MapReduce and Spark software environments are developed and are popular for HPDA. However, these software frameworks are not designed for HPC and not compatible with HPC storage subsystems. In this research, the design and development of a unified data access framework, named IRIS, is proposed for the integration of compute-centric and data-centric storage solutions. The intellectual merit of this research is three-fold. 1) Mapping of incompatible structures: Mapping a file to key-value pairs and vice versa efficiently is a challenging task. 2) Maintaining metadata information: Since IRIS is a unified storage layer, it needs to maintain compatibility with legacy codes. IRIS will address this challenge with tunable consistency, which need to be carefully studied for implementation and design choices. 3) Minimizing overhead and memory footprint of IRIS solutions: Mapping of incompatible structures can cause excessive memory usage; this will be addressed in this research.This project is expected to have significant impact, including bridging data generation and data analysis processes; promoting collaboration between the model simulation and data analysis communities; and building a foundation element for next generation integrated storage systems. This research will create advanced solutions and technologies that will have direct impact on improving the efficiency of data access and management at scale. Since Big Data is a national strategic infrastructure for science, engineering, and industry, this research will advance a broad range of fields. It aims to make significant progress toward a unified storage access system. All data generated from this project will be stored in an electronic format and will be preserved on the server machines at the Illinois Institute of Technology (IIT): http://cs.iit.edu/~scs/. The server machines have hot-copy backup disks for backing up the primary copy of all data. A secondary copy of all data will be kept on the server machines in the computer science department at IIT per semester basis. The data will be transferred to new storage devices every 2 years. All data will be retained within 3 years of the project completion date.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着高性能计算（HPC）和高性能数据分析（HPDA）之间的界限不断模糊，传统的以计算为中心的HPC和新兴的数据驱动的大数据应用正在融合。HPC变得更加数据密集。与此同时，HPDA需要更多的计算能力。MapReduce和Spark软件环境被开发出来，并在HPDA中很受欢迎。但是，这些软件框架不是为HPC设计的，并且与HPC存储子系统不兼容。在这项研究中，设计和开发一个统一的数据访问框架，命名为IRIS，提出了以计算为中心的存储解决方案和以数据为中心的集成。 这项研究的学术价值有三个方面。1)不兼容结构的映射：有效地将文件映射到键值对，反之亦然是一项具有挑战性的任务。2)维护元数据信息：由于IRIS是一个统一的存储层，它需要保持与遗留代码的兼容性。IRIS将通过可调的一致性来应对这一挑战，这需要仔细研究实现和设计选择。3)最大限度地减少IRIS解决方案的开销和内存占用：不兼容结构的映射可能会导致过多的内存使用，这将在本研究中解决。本项目预计将产生重大影响，包括桥接数据生成和数据分析过程;促进模型模拟和数据分析社区之间的合作;以及为下一代集成存储系统构建基础元素。这项研究将创建先进的解决方案和技术，对提高大规模数据访问和管理的效率产生直接影响。由于大数据是科学，工程和工业的国家战略基础设施，这项研究将推动广泛的领域。它旨在朝着统一存储访问系统的方向取得重大进展。本项目产生的所有数据将以电子格式存储，并保存在伊利诺伊理工学院（IIT）的服务器上：http://cs.iit.edu/~scs/。服务器计算机具有热拷贝备份磁盘，用于备份所有数据的主副本。所有数据的第二份副本将保存在IIT计算机科学部门的服务器上，每学期一次。数据将每两年转移到新的存储设备。所有数据将在项目完成后的3年内保留。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

LabStor: A Modular and Extensible Platform for Developing High-Performance, Customized I/O Stacks in Userspace

• DOI: 10.1109/sc41404.2022.00028
• 发表时间: 2022-11
• 期刊: SC22: International Conference for High Performance Computing, Networking, Storage and Analysis
• 作者: Luke Logan;Jaime Cernuda Garcia;J. Lofstead;Xian-He Sun;Anthony Kougkas

HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder

• DOI: 10.1109/cluster48925.2021.00064
• 发表时间: 2021-09
• 期刊: 2021 IEEE International Conference on Cluster Computing (CLUSTER)
• 作者: Jaime Cernuda Garcia;H. Devarajan;Luke Logan;Keith Bateman;N. Rajesh;Jie Ye;Anthony Kougkas;Xian-He Sun

Apollo:: An ML-assisted Real-Time Storage Resource Observer

• DOI: 10.1145/3431379.3460640
• 发表时间: 2020-06
• 期刊: Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing
• 作者: N. Rajesh;H. Devarajan;Jaime Cernuda Garcia;Keith Bateman;Luke Logan;Jie Ye;Anthony Kougkas

DLIO: A Data-Centric Benchmark for Scientific Deep Learning Applications

• DOI: 10.1109/ccgrid51090.2021.00018
• 发表时间: 2021-05
• 期刊: 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
• 作者: H. Devarajan;Huihuo Zheng;Anthony Kougkas;Xian-He Sun;V. Vishwanath

Stimulus: Accelerate Data Management for Scientific AI applications in HPC

• DOI: 10.1109/ccgrid54584.2022.00020
• 发表时间: 2022-05
• 期刊: 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
• 作者: H. Devarajan;Anthony Kougkas;Huihuo Zheng;V. Vishwanath;Xian-He Sun