CSR: Small: A Fine-Grained Hierarchical Memory Management System for Applications with Dynamic Memory Demand on GPUs

图形处理单元（gpu）由于能够提供高吞吐量、高效的计算，已经成为现代高端计算机平台的重要组成部分。因此，GPU利用率对许多应用程序的性能至关重要。尽管在GPU编程方面进行了十多年的研究，但一类具有动态内存需求的重要应用程序并没有得到很好的支持。本研究旨在简化动态记忆体在gpu上的程式设计，并改善其效能。该项目的成果将为各种复杂和动态内存使用的应用解锁gpu的功能，包括生物信息学，科学计算和机器学习。此外，该项目将有助于开发高性能计算课程，并为该机构中代表性不足的学生提供研究机会。在gpu上支持动态内存应用程序主要有三个挑战。首先，这些应用程序中的动态数据大小需要在多个内存级别存储数据。在内存层次结构中访问和同步数据既复杂又耗时。其次，动态内存消耗使得内存分配非常重要，因为在不同内存级别上分配给每个数据对象的内存量会显著影响性能。第三，许多这些应用程序具有复杂和倾斜的内存访问模式，这使得很难确定内存层次结构中的最佳数据位置。本项目旨在通过在GPU上引入细粒度、分层的内存管理系统来解决上述挑战。该系统提供了一种新颖的编程接口，用于管理GPU内存层次中的多维张量，这将有利于动态内存应用的开发。我们的系统还提供自动内存预分配和内存访问优化，这简化了这些应用程序的性能优化。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Graphics Processing Units (GPUs) have become an important component of modern high end computer platforms due to their ability to deliver high throughput, efficient computation. GPU utilization is therefore crucial to the performance of many applications. Despite over a decade of study in GPU programming, an important class of applications with dynamic memory demand is not well supported. This research aims to simplify the programming and improve the performance of dynamic-memory applications on GPUs. The outcome of this project will unlock the power of GPUs for a wide variety of applications with complex and dynamic memory usage, including bioinformatics, scientific computing, and machine learning. Additionally, this project will contribute to the development of high-performance computing courses and provide research opportunities to underrepresented students at the institution.There are mainly three challenges for supporting dynamic-memory applications on GPUs. First, the dynamic data sizes in these applications require data storage at multiple memory levels. Accessing and synchronizing data across the memory hierarchy can be complicated and time-consuming. Second, the dynamic memory consumption makes memory allocation nontrivial, as the amount of memory allocated to each data object at different memory levels can significantly affect performance. Third, many of these applications have complex and skewed memory access patterns, which makes it difficult to determine the optimal data placement in the memory hierarchy. This project aims to address the above challenges by introducing a fine-grained, hierarchical memory management system on GPU. The system provides a novel programming interface for managing multi-dimensional tensors in the GPU memory hierarchy, which will facilitate the development of dynamic-memory applications. Our system also provides automatic memory pre-allocation and memory access optimizations, which simplify performance optimization for these applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.