CSR: Small: IRIS: A unified data access framework for the merging of compute-centric and data-centric storage

随着高性能计算(HPC)和高性能数据分析(HPDA)之间的界限不断模糊，传统的以计算为中心的HPC和新出现的数据驱动的大数据应用程序正在融合。HPC变得更加数据密集型。与此同时，HPDA需要更强的计算能力。开发了MapReduce和Spark软件环境，并在HPDA上流行。然而，这些软件框架不是为HPC设计的，与HPC存储子系统不兼容。在本研究中，为整合以计算为中心的存储解决方案和以数据为中心的存储解决方案，设计并开发了一个统一的数据访问框架IRIS。这项研究的学术价值有三个方面。1)不兼容结构的映射：高效地将文件映射到键-值对或反之亦然是一项具有挑战性的任务。2)维护元数据信息：由于IRIS是统一存储层，需要保持与遗留代码的兼容性。IRIS将以可调的一致性来应对这一挑战，这需要在实现和设计选择方面进行仔细研究。3)最大限度地减少IRIS解决方案的开销和内存占用：映射不兼容的结构可能会导致过多的内存使用；这一问题将在本研究中解决。该项目预计将产生重大影响，包括桥接数据生成和数据分析流程；促进模型模拟和数据分析社区之间的协作；以及为下一代集成存储系统构建基础元素。这项研究将创建先进的解决方案和技术，这些解决方案和技术将对提高大规模数据访问和管理的效率产生直接影响。由于大数据是科学、工程和工业的国家战略基础设施，因此这项研究将推动广泛领域的发展。它的目标是在统一存储访问系统方面取得重大进展。该项目产生的所有数据将以电子格式存储，并将保存在伊利诺伊理工学院的服务器机器上：http://cs.iit.edu/~scs/.服务器机器具有热拷贝备份磁盘，用于备份所有数据的主副本。所有数据的次要副本将以每学期IIT为基础保存在计算机科学系的服务器机器上。数据将每两年传输到新的存储设备。所有数据将在项目完成之日起3年内保留。这一奖励反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

As the boundary between High Performance Computing (HPC) and High-Performance Data Analytics (HPDA) continue to blur, the conventional compute-centric HPC and the newly emerged data-driven Big Data application are converging. HPC becomes more data-intensive. In the meantime, HPDA requires more computing power. MapReduce and Spark software environments are developed and are popular for HPDA. However, these software frameworks are not designed for HPC and not compatible with HPC storage subsystems. In this research, the design and development of a unified data access framework, named IRIS, is proposed for the integration of compute-centric and data-centric storage solutions. The intellectual merit of this research is three-fold. 1) Mapping of incompatible structures: Mapping a file to key-value pairs and vice versa efficiently is a challenging task. 2) Maintaining metadata information: Since IRIS is a unified storage layer, it needs to maintain compatibility with legacy codes. IRIS will address this challenge with tunable consistency, which need to be carefully studied for implementation and design choices. 3) Minimizing overhead and memory footprint of IRIS solutions: Mapping of incompatible structures can cause excessive memory usage; this will be addressed in this research.This project is expected to have significant impact, including bridging data generation and data analysis processes; promoting collaboration between the model simulation and data analysis communities; and building a foundation element for next generation integrated storage systems. This research will create advanced solutions and technologies that will have direct impact on improving the efficiency of data access and management at scale. Since Big Data is a national strategic infrastructure for science, engineering, and industry, this research will advance a broad range of fields. It aims to make significant progress toward a unified storage access system. All data generated from this project will be stored in an electronic format and will be preserved on the server machines at the Illinois Institute of Technology (IIT): http://cs.iit.edu/~scs/. The server machines have hot-copy backup disks for backing up the primary copy of all data. A secondary copy of all data will be kept on the server machines in the computer science department at IIT per semester basis. The data will be transferred to new storage devices every 2 years. All data will be retained within 3 years of the project completion date.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.