FoMR: DeepFetch: Compact Deep Learning based Prefetcher on Configurable Hardware

快速计算机处理器、张量处理单元、硬件加速器和异构体系结构使计算能力得以大规模加速，但内存速度却没有跟上。因此，内存性能已成为许多依赖大量内存访问的应用程序的瓶颈。一些新兴的存储技术，如3d堆叠动态随机存取存储器（3D-DRAM）和非易失性存储器，试图从硬件的角度解决内存瓶颈问题，但需要在带宽、功耗、延迟和成本之间进行权衡。该项目将开发一种基于机器学习的方法，自动学习访问模式，可用于最佳预取数据，而不是重新设计现有算法以适应特定的内存技术。具体来说，高度紧凑的长短期内存（LSTM）模型将被用作预取器的核心，用于预测内存访问。通过新颖的模型压缩技术、分层内存建模和专用硬件，该项目将克服充分利用机器学习和新兴硬件来改进预取的障碍。该项目的成功完成将提高应用程序的内存性能，包括信号处理、计算机视觉和语言处理。在硬件上实现一个实用的基于LSTM的预取器需要处理一些挑战，这些挑战将在这一努力中得到解决：(i)训练一个小模型（以实现快速推理），具有大的跟踪，在预测多个应用程序的内存访问时高度准确；（ii）模型压缩，保证实时推理；（iii）在线按需重新训练模型以学习特定于应用程序的模型，这需要使用少量数据进行快速学习；（iv）基于预取“什么”、“何时”和“何处”模型的预测和不确定性，实时做出预取决策，这也需要对目标内存层次结构进行仔细建模；（vi）基于预测，实时决定重新排序数据（动态数据布局）是否可以改善延迟，使未来的预取更有效；（vii）在有限的可用可配置硬件上映射预测和决策框架，以确保使用小面积/功率的低延迟训练和高吞吐量预取。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Fast computer processors, tensor processing units, hardware accelerators, and heterogeneous architectures have enabled large-scale speed-ups in computational power, but memory speeds have not kept pace at the same time. Memory performance therefore has become the bottleneck in many applications that rely on heavy memory access. Several emerging memory technologies such 3D-Stacked Dynamic Random Access Memory (3D-DRAM) and non-volatile memory attempt to address memory bottleneck issues from a hardware perspective, but with a tradeoff among bandwidth, power, latency, and cost. Rather than redesigning existing algorithms to suit specific memory technology, this project will develop a Machine Learning-based approach that automatically learns access patterns which may be used to optimally prefetch data. Specifically, highly compact Long short-term memory (LSTM) models will be used as the centerpiece of the prefetcher for predicting memory accesses. Through novel model compression techniques, hierarchical memory modeling and dedicated hardware, this project will overcome barriers of fully exploiting machine learning and emerging hardware to improve prefetching. Successful completion of this project will lead to improved memory performance for applications, including signal processing, computer vision, and language processing.A practical LSTM based prefetcher implementation on hardware requires dealing with certain challenges that will be addressed in this endeavor: (i) training a small model (to enable fast inference) with large traces that is highly accurate in predicting memory accesses for multiple applications; (ii) model compression to ensure real-time inference; (iii) retraining the model online on-demand to learn application specific models, which would require fast learning with small amount of data; (iv) making prefetching decisions in real-time based on the prediction and uncertainty of the model ''what'', ''when'', and ''where'' to prefetch, which also requires careful modeling of the target memory hierarchy; (vi) based on the predictions, deciding in real-time if reordering data (dynamic data layout) can improve the latency, making future prefetches more effective; (vii) mapping the framework of predictions and decision making on limited available configurable hardware in - ensuring low latency training and high-throughput prefetching utilizing small area/power.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.