Collaborative Research: Hardware-Aware Matrix Computations for Deep Learning Applications

如今深度学习(DL)系统无处不在，可以说比任何其他计算系统对我们日常生活的影响都更大。最近，这样的深层模型(例如GPT-3)已经变得越来越大和笨拙，占用了大量的计算空间。考虑到不断增长的计算要求，在少数几家大型科技公司之外学习此类DL模型的尖端研究几乎不可能取得进展。这个项目将探索有原则的方法来创建与大型深度模型一样有表现力但计算成本很小的数字图书馆系统。在实践方面，这些改进预计将扩大为社会更大部分创造如此强大的数字图书馆模型的可能性。在教育方面，该项目将培训本科生(UG)研究人员，并将负责任的计算整合到UG课程中。该项目将研究如何使用结构化矩阵和现代硬件约束来获得与这些真正大型模型相似的性能，但体积和计算成本却很小。具体地说，研究人员专注于以下两个方面：(I)设计满足DL应用程序所需的所有属性的结构化矩阵的“圣杯”(包括具有高效的投影问题以及高效的并行和/或硬件友好的学习算法)；以及(Ii)考虑我们的新理论可以解锁的新应用程序。当研究结构化矩阵时，这一DL透镜暴露了需要考虑的新问题。反过来，在这个项目中研究的新的结构化矩阵家族不仅将立即具有实际应用，而且还将揭开矩阵计算中经典理论问题的新转折。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep Learning (DL) systems these days are ubiquitous, and arguably affect our everyday lives more than any other computational system. Recently, such deep models (e.g., GPT-3) have increasingly become large and unwieldy with a large computational footprint. Given the ever increasing computational requirements, it has become nearly impossible to make progress on cutting edge research in learning such DL models outside of a few large technological companies. This project will explore principled ways to create DL systems that are as expressive as the large deep models but at a fraction of the computational cost. On the practical front these improvements are expected to expand the possibility of creating such powerful DL models to larger parts of society. On the educational front, this project will train undergraduate (UG) researchers and will integrate responsible computing into UG curriculum.This project will study how one can use structured matrices in concert with modern hardware constraints to achieve similar performance as these really large models but at a fraction of size and computational cost. Specifically, the investigators focus on the following two thrusts: (i) Design the ‘holy grail’ of structured matrices that satisfy all properties that are desirable in DL applications (including having an efficient projection problem as well as having efficient parallel and/or hardware friendly learning algorithms); and (ii) Thinking of new applications that our new theory can unlock. This DL lens exposes new problems to consider when studying structured matrices. In turn, the new family of structured matrices studied in this project will not only have immediate practical applications but will also unlock new twists on classical theoretical problems in matrix computations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.