Collaborative Research: SCALE MoDL: Representation Theoretic Foundations of Deep Learning

合作研究：SCALE MoDL：深度学习的表示理论基础

• 批准号: 2134274
• 负责人: Qi Yu
• 金额: $ 30万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134274&HistoricalAwards=false
• 关键词: Collaborative; Research; SCALE; MoDL; Representation

In the past decade, deep learning has had transformative impacts across society. However, progress has often relied on heuristic methods, massive data, and great computing power. This comes with limited theoretical understanding and has at times given rise to failures of generalization and vulnerable performance in extreme scenarios. This project will address these limitations by developing strong theoretical foundations for deep learning using representation theory, which is the mathematical study of symmetry. Symmetry plays a key role in human reasoning. Greater understanding of the role symmetry plays in deep learning will unlock a variety of improved models. These include models that can learn from scientific knowledge and not just raw data, models with trustable, guaranteed performance, and models that can recombine patterns they have already learned — as humans do easily — to generalize to new situations more rapidly. An explicit goal of this project is to broaden research into why deep learning works. To this end, the investigators will integrate the research into education and establish a mentorship program for high school students from groups underrepresented in science.The goal of the research is to understand the role of representation theory in enabling efficient optimization and improved generalization of deep learning even in domains with approximate or unknown symmetry. This project pursues three lines of research that will broaden the impact of representation theory in deep learning beyond strict inductive biases. The first is the trade-off between the degree of symmetry in the model and the degree of symmetry in the domain. This line of research will study networks that combine equivariant and non-equivariant features. The second line of research will examine learning symmetry directly from data to improve generalization in domains without known symmetries. The third aim is to develop a theoretical basis for deep learning using quiver representations. This perspective reveals the symmetry of the structure of deep-learning models themselves, through their parameter spaces, even when the domains have no obvious symmetry.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在过去的十年中，深度学习对整个社会产生了变革性的影响。 然而，进步往往依赖于启发式方法、海量数据和强大的计算能力。 这伴随着有限的理论理解，有时会导致泛化失败和极端情况下的脆弱性能。该项目将通过使用表示论（对称性的数学研究）为深度学习奠定坚实的理论基础，从而解决这些局限性。 对称性在人类推理中起着关键作用。 更好地理解对称性在深度学习中的作用将解锁各种改进的模型。其中包括可以从科学知识而不仅仅是原始数据中学习的模型、具有可靠、有保证性能的模型，以及可以像人类一样轻松地重新组合已经学到的模式以更快地推广到新情况的模型。 该项目的一个明确目标是扩大对深度学习为何有效的研究。 为此，研究人员将把研究融入教育中，并为科学领域代表性不足的群体的高中生建立一个导师计划。该研究的目标是了解表示论在实现深度学习的高效优化和改进泛化方面的作用，即使是在具有近似或未知对称性的领域。 该项目进行三方面的研究，这些研究将扩大表示理论在深度学习中的影响，超越严格的归纳偏差。 首先是模型的对称度和域的对称度之间的权衡。 这一系列研究将研究结合等变和非等变特征的网络。 第二条研究将检查直接从数据中学习对称性，以提高未知对称性领域的泛化能力。 第三个目标是为使用箭袋表示的深度学习奠定理论基础。这种观点通过参数空间揭示了深度学习模型本身结构的对称性，即使领域没有明显的对称性。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Probabilistic Symmetry for Multi-Agent Dynamics

• 发表时间: 2022-05
• 作者: Sophia Sun;R. Walters;Jinxi Li;Rose Yu

Automatic Symmetry Discovery with Lie Algebra Convolutional Network

• 发表时间: 2021-09
• 作者: Nima Dehmamy;R. Walters;Yanchen Liu-;Dashun Wang;Rose Yu

Approximately Equivariant Networks for Imperfectly Symmetric Dynamics

• 发表时间: 2022-01
• 期刊: ArXiv
• 作者: Rui Wang;R. Walters;Rose Yu

Symmetry Teleportation for Accelerated Optimization

用于加速优化的对称隐形传态

• 发表时间: 2022
• 期刊: Advances in neural information processing systems
• 作者: Bo Zhao;Nima Dehmamy;Robin Walters;Rose Yu
• 通讯作者: Rose Yu