CSR: Small: A Fine-Grained Hierarchical Memory Management System for Applications with Dynamic Memory Demand on GPUs

图形处理单元（GPU）由于其提供高吞吐量、高效计算的能力而已经成为现代高端计算机平台的重要组件。因此，GPU利用率对许多应用程序的性能至关重要。尽管在GPU编程方面进行了十多年的研究，但对具有动态内存需求的一类重要应用程序的支持并不好。本研究旨在简化动态内存应用程序在GPU上的编程并提高其性能。该项目的成果将为具有复杂和动态内存使用的各种应用程序释放GPU的力量，包括生物信息学，科学计算和机器学习。此外，该项目还将为高性能计算课程的开发做出贡献，并为该机构中代表性不足的学生提供研究机会。在GPU上支持动态内存应用程序主要面临三个挑战。首先，这些应用程序中的动态数据大小需要在多个存储器级别上存储数据。跨存储器层次结构的数据同步和同步可能是复杂且耗时的。第二，动态内存消耗使得内存分配变得重要，因为在不同的内存级别上分配给每个数据对象的内存量会显著影响性能。第三，这些应用程序中的许多程序具有复杂和倾斜的存储器访问模式，这使得难以确定存储器层次结构中的最佳数据放置。本项目旨在通过在GPU上引入一个细粒度、层次化的内存管理系统来解决上述挑战。该系统提供了一种新的编程接口，用于管理GPU内存层次结构中的多维张量，这将有助于动态内存应用程序的开发。我们的系统还提供自动内存预分配和内存访问优化，从而简化了这些应用程序的性能优化。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Graphics Processing Units (GPUs) have become an important component of modern high end computer platforms due to their ability to deliver high throughput, efficient computation. GPU utilization is therefore crucial to the performance of many applications. Despite over a decade of study in GPU programming, an important class of applications with dynamic memory demand is not well supported. This research aims to simplify the programming and improve the performance of dynamic-memory applications on GPUs. The outcome of this project will unlock the power of GPUs for a wide variety of applications with complex and dynamic memory usage, including bioinformatics, scientific computing, and machine learning. Additionally, this project will contribute to the development of high-performance computing courses and provide research opportunities to underrepresented students at the institution.There are mainly three challenges for supporting dynamic-memory applications on GPUs. First, the dynamic data sizes in these applications require data storage at multiple memory levels. Accessing and synchronizing data across the memory hierarchy can be complicated and time-consuming. Second, the dynamic memory consumption makes memory allocation nontrivial, as the amount of memory allocated to each data object at different memory levels can significantly affect performance. Third, many of these applications have complex and skewed memory access patterns, which makes it difficult to determine the optimal data placement in the memory hierarchy. This project aims to address the above challenges by introducing a fine-grained, hierarchical memory management system on GPU. The system provides a novel programming interface for managing multi-dimensional tensors in the GPU memory hierarchy, which will facilitate the development of dynamic-memory applications. Our system also provides automatic memory pre-allocation and memory access optimizations, which simplify performance optimization for these applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.