XPS: SDA: Collaborative Research: A Scalable and Distributed System Framework for Compute-Intensive and Data-Parallel Applications

传统的高性能计算（HPC）应用是计算密集型的，而最近的高性能计算应用需要更多的数据密集型分析和可视化来提取知识。在许多情况下，这些应用程序执行与过去相同的计算算法（例如，并行搜索或并行渲染），但现在必须为更大的数据集这样做。例如，生命科学，以及科学可视化的交叉领域，构成了一个新兴的HPC应用类别，它不仅执行复杂的计算，而且还摄取大量的数据。在当今的计算平台上运行这些新的HPC数据并行应用程序带来了新的挑战，并需要额外的功能。然而，今天的HPC平台仍然采用以计算为中心的模式，并不能很好地应对这些新的挑战。这种模型往往将大量的数据转移到各种并行计算过程中。因此，完成I/O的漫长CPU等待时间和巨大的数据移动开销成为高性能和可伸缩性的主要障碍。该项目包括创建一个可扩展的跨层软件框架，使计算密集型和数据密集型并行HPC应用程序能够在分布式文件系统上运行。该框架由两个相互交织的研究任务组成：(1)一个自适应的、数据位置感知的中间件系统，它通过监控物理数据位置来动态调度计算进程以访问本地数据；(2)一个框架，它从并行应用程序中捕获计算和数据I/O处理关系，并协调相应进程和I/O执行的调度，以实现最大的并行效率。该项目的成功通过消除科学应用中CPU等待时间和频繁访问数据的网络传输，提高了高性能计算资源的生产力和投资回报。一个开源的、可持续的、可重用的软件框架被交付，以加速诸如生物信息学、气候、高能物理、宇宙学、天体物理学和色动力学等领域的发现和创新过程。这两个提议机构，弗吉尼亚理工大学和中佛罗里达大学，以及他们合作的美国能源部国家实验室的协同作用，将在学生的研究生教育中催化新的和有益的观点，并为21世纪的高性能计算劳动力做好准备。

Whereas traditional high-performance computing (HPC) applications are computationally intensive, recent HPC applications require more data-intensive analysis and visualization to extract knowledge. In many cases, these applications execute the same computational algorithm as in the past (e.g., parallel search or parallel rendering) but now must do so for significantly larger data sets. For example, the life sciences, along with the cross-cutting area of scientific visualization, constitute an emerging category of HPC applications that not only perform sophisticated calculations but also ingest a sea of data. Running these new HPC data-parallel applications on today's computing platforms imposes new challenges and demands additional functionality.However, today's HPC platforms still adopt a compute-centric model and do not handle these new challenges well. Such a model often moves a large amount of data to various parallel computational processes. Consequently, long CPU wait times for I/O to complete and enormous data-movement overhead become major stumbling blocks to high performance and scalability. This project encompasses the creation of a scalable cross-layer software framework to enable both computationally intensive and data-intensive parallel HPC applications to run on distributed file systems. This framework consists of two interwoven research tasks: (1) an adaptive, data locality-aware, middleware system that dynamically schedules compute processes to access local data by monitoring physical data locations and (2) a framework that captures the computation and data I/O processing relationship from parallel applications and coordinates the scheduling of the corresponding process and I/O execution for maximum parallel efficiency. The success of this project contributes enhanced productivity and return on investment on HPC resources via the elimination of both CPU wait time and network transfer of frequently accessed data in scientific applications. An open-source, sustainable, and reusable software framework is delivered to speed-up the discovery and innovation process in areas such as bioinformatics, climate, high-energy physics, cosmology, astrophysics, and chromodynamics. The synergy in the two proposing institutions, Virginia Tech and the University of Central Florida, and their collaborating DOE national laboratories, will catalyze new and beneficial perspectives in the graduate education of students and prepare a 21st-century workforce in HPC.