Collaborative Research: OAC Core: Enabling Extremely Fine-grained Parallelism on Modern Many-core Architectures

计算机系统正变得越来越复杂：具有多核处理器和通用图形处理器的多插槽系统有可能在节点级别满足要求苛刻的应用程序的需求。由于硬件对芯片上数千个计算单元的并行度增加了几个数量级，因此可编程性和效率通常不容易找到。任务并行是一种重要的并行类型，它将计算分解成一组相互依赖的任务，这些任务可以在不同的计算单元上并发执行。为了实现强大的可伸缩性和高水平的有效并行性，当今的并行语言越来越需要支持过度分解(比核心多得多的任务)，以提高性能，隐藏阻塞操作造成的延迟，并以其他方式实现最大加速比。通过在现代和未来硬件中看到的不断增长的范围内实现对细粒度并行的有效支持，预计并行程序员的生产率将得到提高。趋势显示，有证据表明，大多数TOP500高性能计算系统可能会使用这项工作直接针对的硬件。该项目旨在开展一个影响广泛的分布式并行编程教育项目，鼓励学生根据现实世界的挑战进行实习，并为将技术从研究转移到开源项目铺平道路。特别强调让妇女和代表性不足的少数群体参与进来。这方面的教育将为科学家和工程师在并行计算方面的流畅性创造一个新的、更容易获得的基础。这项工作探索了新的数据结构和算法，允许在亚微秒级实现细粒度并行的可伸缩运行时和执行模型。PI在语言和运行时级别的初步工作为实现这一目标提供了一条途径。该项目的目标是：1)统一运行时使能任务粒度按周期测量：设计、分析和实现用于在不同节点硬件上进行高效细粒度计算的构建块；2)在一系列计算机架构上的真实并行系统和应用内核的环境中评估这些构建块的性能；3)测量运行时对基准内核和真实应用程序的性能和可扩展性影响；以及4)通过新的并行计算课程材料将这项研究与从本科生到研究生的教育计划相结合。这项高风险/高回报的研究旨在对每一种规模的并行机编程的简易性和效率进行革命性的改进。其贡献在于实现了高效、隐式并行的高级语言，这些语言针对单节点部署进行了优化，具有多核体系结构，以支持按周期测量的细粒度并行，从而实现了全新的多任务计算应用程序。数据流体系结构使隐式并行性易于使用编程模型处理，其影响可以与MATLAB、R和Python相媲美，此外，相同的代码还可以在分布式系统或大型HPC系统中运行。因此，科学家将能够编写一次程序，在任何合适的规模上运行它，并使其无缝地使用硬件的每个组件的最合适的粒度。这项工作在数据流体系结构方面的创新将广泛适用于许多现有的并行编程系统，如OpenMP、SWIFT/Parsl和CUDA/OpenCL，在执行细粒度并行方面的效率和在可能的情况下增加对隐式并行的支持。目标硬件包括Intel/AMD x86、ThunderX/2 ARM、IBM Power9和NVIDIA/AMD GPU。该奖项反映了NSF的法定使命，通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Computer systems are becoming increasingly complex: multisocket systems with many-core processors and general graphic processors have the potential to address the needs of demanding applications at the node level. Programmability and efficiency are often not easy to find together due to the hardware growing several orders of magnitude in degree of parallelism to thousands of computing units on a chip. Task parallelism is an important type of parallelism in which computation is broken down into a set of inter-dependent tasks which can be executed concurrently on various computing units. To achieve strong scaling and high levels of effective parallelism, there is a growing need in today's parallel languages with supporting over-decomposition (many more tasks than cores) in order to improve performance, hide latency caused by blocking operations, and otherwise achieve maximum speedup. By enabling the efficient support of fine-grained parallelism across the growing range of scales seen in modern and future hardware, it is expected that the productivity of parallel programmers will be enhanced. Trends show evidence that most of the Top500 high-performance computing systems will likely employ hardware that this work directly targets. The project aims to conduct a high-impact education program in distributed parallel programming with broad reach, encouraging student internships grounded in real-world challenges, and paving the way for technology transfer from research to open-source projects. Special emphasis is placed on engaging women and underrepresented minorities. This education facet will create a new and more accessible foundation for fluency in parallel computing for scientists and engineers.This work explores novel data-structures and algorithms that allow for scalable runtime and execution models for fine-grained parallelism at sub-microsecond timescales. Preliminary work by the PIs at the language and runtime levels suggests a path to achieving this. The project objectives are: 1) unifying runtime enabling task granularities measured in cycles: design, analysis, and implementation of building blocks for efficient fine-grained computing on diverse node hardware; 2) evaluating performance of these building blocks in the context of real parallel systems and application kernels on a range of computer architectures; 3) measuring performance and scalability impact of runtime on benchmark kernels and real applications; and 4) integrating this research with education programs from undergraduate to graduate levels through new course material on parallel computing. This high-risk/high-reward research is geared towards yielding transformative improvements in the ease and efficiency of programming parallel machines at every scale. The contributions lie in the realization of productive, implicitly parallel high-level languages optimized for single node deployments with many-core architectures to support fine-grained parallelism measured in cycles, enabling an entirely new class of many-task computing applications. The dataflow architecture makes implicit parallelism tractable with a programming model whose impact could rival that of MATLAB, R, and Python, with the added benefit that the same code could also run in a distributed system or large-scale HPC systems. Thus, the scientist would be able to write a program once, run it at any suitable scale, and have it seamlessly use the most appropriate granularity for each component of the hardware. This work’s innovations in dataflow architecture will be broadly applicable to a number of existing parallel programming systems such as OpenMP, Swift/Parsl, and CUDA/OpenCL, in terms of both efficiency in executing fine grained parallelism and adding support for implicit parallelism where possible. Target hardware includes Intel/AMD x86, ThunderX/2 ARM, IBM Power9, and NVIDIA/AMD GPUs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.