Collaborative Research: SHF:SMALL: Compile-Parallelize-Schedule-Retarget-Repeat (EASER) Paradigm for Dealing with Extreme Heterogeneity

计算中的异构性是指在一个计算系统中甚至在集群的一个节点中存在各种设备。许多技术趋势使得高性能计算(HPC)中的高度异构性不可避免，这导致了许多方向的研究。传统的调度问题是指获取一组要执行的程序并将它们映射到可用资源，在存在这种异构性的情况下变得更加复杂，因为调度器也需要与编译器交互。该项目的目标是根据这些发展考虑应用程序执行的新范例，并在开发执行时间预测、编译、并行化和调度方面进行研究。传统上，决定(可能是手动)如何并行化应用程序、编译和集群级调度是按顺序独立完成的。研究人员假设，当试图针对多租户异类集群进行优化时，他们的孤立处理将是不可接受的。相反，研究人员设想了一个可以称为Easer--编译-并行化-调度-重定目标-重复的需求。为了详细说明这一愿景，在EASER范例中，编译器首先将核心函数映射到特定设备，生成输入到并行化方法选择模块的执行时间预测，然后它们一起产生最终的可执行文件。随后，此二进制文件被提供给调度程序，调度程序评估作业队列，并可能建议替代配置(S)/设备(S)。如果是，则调用重定目标模块，导致可能重复上述步骤。该项目在执行新兴机器学习(ML)工作负载的集群环境中开发、支持和评估Easer框架。1)编译器驱动的性能预测--它包括一个新的策略，包括一个预测SIMD/VLIW性能的通用模型和一个基于操作符分类的内存层次性能模型的开发方法。2)集成作业调度和并行化策略选择--在性能预测模型的基础上，通过引入参数化和增量式并行化策略选择方法，并在调度方法中积极缩小搜索空间，将这两个(传统上独立的)模块集成在一起。3)重定目标编译器--通过将优化分为依赖于体系结构的优化和独立的优化，将开发一个用于ML工作负载的重定目标编译器。该项目还将对教育和人力资源开发作出若干贡献。两位研究人员将在计算机系统和机器学习的交叉点介绍课程(S)(材料)，以引起人们对计算机系统教育中与ML相关的工作量的关注。每所大学的大部分资金将用于支持博士生的研究，他们将接受跨传统(子)领域的工作培训。这两位研究人员都坚定地致力于增加计算领域的多样性，并在他们的研究项目中监督代表不足的群体的成员方面有着良好的记录。在各自大学现有关系的基础上，他们将进一步致力于提高各个层面的多样性。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Heterogeneity in computing refers to having a variety of devices present within one computing system or even within one node of a cluster. A number of technological trends are making a high degree of heterogeneity inevitable in High Performance Computing (HPC), leading to research along many directions. The traditional scheduling problem, which refers to taking a set of programs to be executed and mapping them to the available resources, becomes more complicated in the presence of such heterogeneity, as the schedulers need to interact with the compiler also. The goal of this project is to consider new paradigms for application execution in view of these developments and conduct research in developing predictions of execution times, compilation, parallelization, and scheduling. Traditionally, deciding (likely manually) how an application is to be parallelized, compilation, and cluster-level scheduling are done sequentially and independently. The investigators posit that their isolated treatment is not going to be acceptable when one tries to optimize for multi-tenant heterogeneous clusters. Instead, the investigators envision a requirement that can be referred to as EASER -- compilE-pArallelize-Schedule-rEtarget-Repeat. To elaborate on the vision, in the EASER paradigm the compiler first maps the core functions to a specific device, generating predictions of execution time that are input to the parallelization approach selection module, and together they produce a final executable. Subsequently, this binary is presented to the scheduler, which assesses the job queue and might suggest alternative configuration(s)/device(s). If so, a retargeting module is to be invoked, leading to a potential repetition of the above steps. This project develops, supports, and evaluates the EASER framework in the context of a cluster that executes emerging machine learning (ML) workloads. Research is proposed in the following areas: 1) Compiler-Driven Performance Prediction -- It includes a novel strategy that comprises a general model for predicting SIMD/VLIW performance and an operator classification based approach to developing a memory hierarchy performance model. 2) Integrated Job Scheduling and Parallelization Strategy Selection -- Building on the performance prediction models, these two (conventionally independent) modules are integrated, by including parameterized and incremental parallelization strategy selection methods and aggressively reducing the search space in scheduling methods. 3) Retargeting Compiler -- By classifying optimizations as either architecture-dependent or independent, a retargeting compiler for ML workloads will be developed. This project will also make several contributions to education and human resource development. Both investigators will be introducing course(s) (material) at the intersection of computer systems and machine learning, bringing attention to ML-related workloads in computer systems education. A majority of funds at each University will be used to support Ph.D. students in their research, who will be trained to work across traditional (sub-) areas. Both investigators are strongly committed to increasing diversity in computing fields and have a strong record of supervising members of underrepresented groups in their research programs. Building on their Universities' existing connections, they will be further working on improving diversity at all levels.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.