Collaborative Research: SHF:SMALL: Compile-Parallelize-Schedule-Retarget-Repeat (EASER) Paradigm for Dealing with Extreme Heterogeneity

计算的异构性是指在一个计算系统甚至集群的一个节点内存在多种设备。许多技术趋势使得高性能计算（HPC）中的高度异构性不可避免，从而导致沿着沿着许多方向的研究。 传统的调度问题，这是指采取一组要执行的程序，并将它们映射到可用的资源，变得更加复杂，在这种异构性的存在，因为编译器也需要与编译器进行交互。该项目的目标是考虑这些发展的应用程序执行的新范例，并在开发执行时间，编译，并行化和调度的预测进行研究。 传统上，决定（可能是手动）如何并行化应用程序，编译和集群级调度是顺序和独立完成的。研究人员认为，当人们试图优化多租户异构集群时，他们的孤立处理是不可接受的。相反，研究人员设想了一个可以称为EASER的要求-- compilE-pArallelize-Schedule-rEtarget-Repeat。为了详细说明这一愿景，在EASER范式中，编译器首先将核心函数映射到特定设备，生成输入到并行化方法选择模块的执行时间预测，并一起生成最终的可执行文件。随后，该二进制文件被提交给调度程序，调度程序评估作业队列并可能建议替代配置/设备。如果是，则将调用重定向模块，从而导致上述步骤的潜在重复。 该项目在执行新兴机器学习（ML）工作负载的集群环境中开发，支持和评估EASER框架。本文提出了以下几个方面的研究内容：1）基于操作符驱动的性能预测--提出了一种新的策略，该策略包括一个预测SIMD/VLIW性能的通用模型和一个基于操作符分类的开发存储器层次性能模型的方法。2)集成作业调度和并行化策略选择--在性能预测模型的基础上，通过引入参数化和增量式并行化策略选择方法，并积极减少调度方法中的搜索空间，将这两个（传统上独立的）模块集成起来。3)重定向编译器-通过将优化分类为依赖于架构或独立，将开发用于ML工作负载的重定向编译器。 该项目还将对教育和人力资源开发作出若干贡献。两位研究人员将在计算机系统和机器学习的交叉点上介绍课程（材料），引起人们对计算机系统教育中ML相关工作量的关注。每所大学的大部分资金将用于支持博士学位。学生在他们的研究，谁将接受培训，跨传统（子）领域的工作。两位研究人员都坚定地致力于增加计算领域的多样性，并在其研究项目中监督代表性不足的群体成员方面有着良好的记录。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Heterogeneity in computing refers to having a variety of devices present within one computing system or even within one node of a cluster. A number of technological trends are making a high degree of heterogeneity inevitable in High Performance Computing (HPC), leading to research along many directions. The traditional scheduling problem, which refers to taking a set of programs to be executed and mapping them to the available resources, becomes more complicated in the presence of such heterogeneity, as the schedulers need to interact with the compiler also. The goal of this project is to consider new paradigms for application execution in view of these developments and conduct research in developing predictions of execution times, compilation, parallelization, and scheduling. Traditionally, deciding (likely manually) how an application is to be parallelized, compilation, and cluster-level scheduling are done sequentially and independently. The investigators posit that their isolated treatment is not going to be acceptable when one tries to optimize for multi-tenant heterogeneous clusters. Instead, the investigators envision a requirement that can be referred to as EASER -- compilE-pArallelize-Schedule-rEtarget-Repeat. To elaborate on the vision, in the EASER paradigm the compiler first maps the core functions to a specific device, generating predictions of execution time that are input to the parallelization approach selection module, and together they produce a final executable. Subsequently, this binary is presented to the scheduler, which assesses the job queue and might suggest alternative configuration(s)/device(s). If so, a retargeting module is to be invoked, leading to a potential repetition of the above steps. This project develops, supports, and evaluates the EASER framework in the context of a cluster that executes emerging machine learning (ML) workloads. Research is proposed in the following areas: 1) Compiler-Driven Performance Prediction -- It includes a novel strategy that comprises a general model for predicting SIMD/VLIW performance and an operator classification based approach to developing a memory hierarchy performance model. 2) Integrated Job Scheduling and Parallelization Strategy Selection -- Building on the performance prediction models, these two (conventionally independent) modules are integrated, by including parameterized and incremental parallelization strategy selection methods and aggressively reducing the search space in scheduling methods. 3) Retargeting Compiler -- By classifying optimizations as either architecture-dependent or independent, a retargeting compiler for ML workloads will be developed. This project will also make several contributions to education and human resource development. Both investigators will be introducing course(s) (material) at the intersection of computer systems and machine learning, bringing attention to ML-related workloads in computer systems education. A majority of funds at each University will be used to support Ph.D. students in their research, who will be trained to work across traditional (sub-) areas. Both investigators are strongly committed to increasing diversity in computing fields and have a strong record of supervising members of underrepresented groups in their research programs. Building on their Universities' existing connections, they will be further working on improving diversity at all levels.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.