Collaborative Research: Hardware-Aware Matrix Computations for Deep Learning Applications

协作研究：深度学习应用的硬件感知矩阵计算

• 批准号: 2247014
• 负责人: Atri Rudra
• 金额: $ 37.7万
• 依托单位: SUNY at Buffalo
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-05-01 至 2026-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2247014&HistoricalAwards=false
• 关键词: Collaborative; Research; Hardware; Aware; Matrix

Deep Learning (DL) systems these days are ubiquitous, and arguably affect our everyday lives more than any other computational system. Recently, such deep models (e.g., GPT-3) have increasingly become large and unwieldy with a large computational footprint. Given the ever increasing computational requirements, it has become nearly impossible to make progress on cutting edge research in learning such DL models outside of a few large technological companies. This project will explore principled ways to create DL systems that are as expressive as the large deep models but at a fraction of the computational cost. On the practical front these improvements are expected to expand the possibility of creating such powerful DL models to larger parts of society. On the educational front, this project will train undergraduate (UG) researchers and will integrate responsible computing into UG curriculum.This project will study how one can use structured matrices in concert with modern hardware constraints to achieve similar performance as these really large models but at a fraction of size and computational cost. Specifically, the investigators focus on the following two thrusts: (i) Design the ‘holy grail’ of structured matrices that satisfy all properties that are desirable in DL applications (including having an efficient projection problem as well as having efficient parallel and/or hardware friendly learning algorithms); and (ii) Thinking of new applications that our new theory can unlock. This DL lens exposes new problems to consider when studying structured matrices. In turn, the new family of structured matrices studied in this project will not only have immediate practical applications but will also unlock new twists on classical theoretical problems in matrix computations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习（DL）系统如今无处不在，可以说比任何其他计算系统都更能影响我们的日常生活。最近，这种深度模型（例如，GPT-3）变得越来越庞大和笨重，并且占用了大量的计算空间。考虑到不断增长的计算需求，除了几家大型科技公司之外，在学习这种深度学习模型的前沿研究方面几乎不可能取得进展。该项目将探索有原则的方法来创建与大型深度模型一样具有表现力的深度学习系统，但计算成本只是其中的一小部分。在实践方面，这些改进有望将创建如此强大的深度学习模型的可能性扩展到更大的社会领域。在教育方面，该项目将培养本科生（UG）研究人员，并将负责任的计算纳入UG课程。该项目将研究如何在现代硬件约束下使用结构化矩阵，以获得与这些真正的大型模型相似的性能，但尺寸和计算成本只是一小部分。具体来说，研究人员专注于以下两个重点：(i)设计结构化矩阵的“圣杯”，满足深度学习应用中所需的所有属性（包括具有高效的投影问题以及具有高效的并行和/或硬件友好的学习算法）；（ii）思考我们的新理论可以解锁的新应用。在研究结构化矩阵时，DL透镜暴露了需要考虑的新问题。反过来，在这个项目中研究的新的结构化矩阵族不仅有直接的实际应用，而且还将解开矩阵计算中经典理论问题的新转折。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Zoology: Measuring and Improving Recall in Efficient Language Models

动物学：测量和提高高效语言模型的召回率

• 发表时间: 2024
• 期刊: Proceedings of 12th International Conference on Learning Representations (ICLR
• 作者: Arora, Simran;Eyuboglu, Sabri;Timalsina, Aman;Johnson, Isys;Poli, Michael;Zou, James;Rudra, Atri;Ré, Christopher
• 通讯作者: Ré, Christopher

Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture

Monarch Mixer：基于简单次二次 GEMM 的架构

• 发表时间: 2023
• 期刊: Proceedings of the 36th Neural Information Processing Systems Conference (NeurIPS
• 作者: Fu, Daniel Y.;Arora, Simran;Grogan, Jessica;Johnson, Isys;Eyuboglu, Sabri;Thomas, Armin W.;Spector, Benjamin;Poli, Michael;Rudra, Atri;Ré, Christopher
• 通讯作者: Ré, Christopher

Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions

• DOI: 10.48550/arxiv.2310.18780
• 发表时间: 2023-10
• 期刊: ArXiv
• 作者: Stefano Massaroli;Michael Poli;Daniel Y. Fu;Hermann Kumbong;Rom N. Parnichkun;Aman Timalsina;David W. Romero;Quinn McIntyre;Beidi Chen;A. Rudra;Ce Zhang;Christopher Ré;Stefano Ermon;Y. Bengio