FoMR: DeepFetch: Compact Deep Learning based Prefetcher on Configurable Hardware

快速的计算机处理器、张量处理单元、硬件加速器和异构架构已经实现了计算能力的大规模加速，但内存速度却没有跟上。因此，内存性能已成为许多依赖大量内存访问的应用程序的瓶颈。 3D堆叠动态随机存取存储器（3D-DRAM）和非易失性存储器等多种新兴存储器技术试图从硬件角度解决存储器瓶颈问题，但需要在带宽、功耗、延迟和成本之间进行权衡。该项目将开发一种基于机器学习的方法，自动学习可用于最佳预取数据的访问模式，而不是重新设计现有算法以适应特定的内存技术。具体来说，高度紧凑的长短期记忆（LSTM）模型将用作预取器的核心，用于预测内存访问。通过新颖的模型压缩技术、分层内存建模和专用硬件，该项目将克服充分利用机器学习和新兴硬件来改进预取的障碍。该项目的成功完成将提高应用程序的内存性能，包括信号处理、计算机视觉和语言处理。在硬件上实现基于 LSTM 的实用预取器需要应对本次工作中将解决的某些挑战：(i) 训练一个具有大轨迹的小型模型（以实现快速推理），该模型在预测多个应用程序的内存访问方面高度准确； (ii) 模型压缩以确保实时推理； (iii) 按需在线重新训练模型以学习特定于应用的模型，这需要使用少量数据进行快速学习； (iv) 根据模型“什么”、“何时”和“何处”预取的预测和不确定性实时做出预取决策，这也需要对目标内存层次结构进行仔细建模； (vi) 根据预测，实时决定重新排序数据（动态数据布局）是否可以改善延迟，使未来的预取更加有效； (vii) 在有限的可用可配置硬件上绘制预测和决策框架，确保利用小面积/功率进行低延迟训练和高吞吐量预取。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

Fast computer processors, tensor processing units, hardware accelerators, and heterogeneous architectures have enabled large-scale speed-ups in computational power, but memory speeds have not kept pace at the same time. Memory performance therefore has become the bottleneck in many applications that rely on heavy memory access. Several emerging memory technologies such 3D-Stacked Dynamic Random Access Memory (3D-DRAM) and non-volatile memory attempt to address memory bottleneck issues from a hardware perspective, but with a tradeoff among bandwidth, power, latency, and cost. Rather than redesigning existing algorithms to suit specific memory technology, this project will develop a Machine Learning-based approach that automatically learns access patterns which may be used to optimally prefetch data. Specifically, highly compact Long short-term memory (LSTM) models will be used as the centerpiece of the prefetcher for predicting memory accesses. Through novel model compression techniques, hierarchical memory modeling and dedicated hardware, this project will overcome barriers of fully exploiting machine learning and emerging hardware to improve prefetching. Successful completion of this project will lead to improved memory performance for applications, including signal processing, computer vision, and language processing.A practical LSTM based prefetcher implementation on hardware requires dealing with certain challenges that will be addressed in this endeavor: (i) training a small model (to enable fast inference) with large traces that is highly accurate in predicting memory accesses for multiple applications; (ii) model compression to ensure real-time inference; (iii) retraining the model online on-demand to learn application specific models, which would require fast learning with small amount of data; (iv) making prefetching decisions in real-time based on the prediction and uncertainty of the model ''what'', ''when'', and ''where'' to prefetch, which also requires careful modeling of the target memory hierarchy; (vi) based on the predictions, deciding in real-time if reordering data (dynamic data layout) can improve the latency, making future prefetches more effective; (vii) mapping the framework of predictions and decision making on limited available configurable hardware in - ensuring low latency training and high-throughput prefetching utilizing small area/power.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.