CAREER: Data-Intensive HPC Analytics: A Systems Approach Through Extended Interfaces, Data Restructuring and Data-centric Scheduling

随着新兴的e-Science应用的出现，今天的科学研究越来越依赖于对PB以上大小的大型数据集进行PB级以上的计算。代表包括分析和模拟驱动的应用，如人类视觉模拟、天体物理数据分析、地震建模、使用系综运行的气候建模等。在上述许多领域，科学家正在处理大量数据并对其进行分析，以探索新的概念和想法。这些应用程序构成了数据密集型HPC分析，它位于当前HPC和数据密集型可扩展计算(DISK)的交叉点上。当HPC系统使用传统配置来支持数据密集型HPC分析时，数据将从大型远程存储系统复制到无盘计算节点进行处理。来回复制数据是一个既昂贵又耗时的过程。这些数据密集型应用程序不需要计算密集型资源，而是需要具有本地存储能力的中等计算能力的计算机，以便可以就地处理数据。这种配置的一个这样的例子是Hadoop框架。然而，为了使Hadoop成为一个有效的HPC工具，目前该框架中存在一些必须克服的限制。研究人员正在利用Hadoop框架在HPC中处理大量模式化数据。这项研究计划包括三个推进。它正在开发MapReduceAPI，以支持更广泛的I/O访问模式、各种数据重组方案以提高这些访问模式的I/O性能，以及考虑到多个块位置和网络上的数据传输延迟的高效调度方案。这项研究纳入了几项教育活动，例如开发数据密集型高性能计算课程。

With the advent of emerging e-Science applications, today's scientific research increasingly relies on petascale-and-beyond computing over large data sets with petabyte-and-beyond sizes. Representatives include analytics- and simulation- driven applications such as the human vision simulation, astrophysics data analysis, earthquake modeling, climate modeling using ensemble runs, etc. In many of the above-mentioned fields, scientists are dealing with large amounts of data and analyzing them to explore new concepts and ideas. These applications make up data-intensive HPC analytics, which lies at the intersection of current HPC and Data-Intensive Scalable Computing (DISC).When HPC systems use traditional configurations to support data-intensive HPC analytics, data is copied from a large remote storage system to diskless compute nodes for processing. Copying data back and forth is an expensive and time consuming process. These data-intensive applications do not require compute intensive resources, but rather moderate compute power machines with the capability of local storage so that data can be processed in-place. One such example of this configuration is the Hadoop framework. However, there are currently limitations in this framework which must be overcome in order to make Hadoop an effective HPC tool. The investigator is leveraging the Hadoop framework to process large amount of patterned data in HPC. This research program includes three thrusts. It is developing the MapReduce API to support a wider range of I/O access patterns, various data restructuring schemes to improve I/O performance for these access patterns, and an efficient scheduling scheme considering multiple chunk locations and data transfer latencies over the network. The research is integrated into several educational activities, such as the development of data-intensive HPC curricula.