Representation Learning via Variational Mean Field Theory

深度学习在许多科学和工程应用中取得了巨大的成功。它以计算上易于处理的方式使用数据驱动的方法彻底改变了基于模型的经典方法。 表示学习从下游任务（例如分类、回归和强化学习）所需的原始数据中提取有用信息，是深度学习最重要的方向之一。尽管表征学习在图像和信号处理、语音和物体识别、自然语言处理、化学和药物发现等领域取得了成功，但对表征学习的理论理解还远远不能令人满意。该项目旨在使用平均场博弈（MFG）理论对表示学习（特别是深度生成模型和图表示学习）进行新的理解。该研究不仅旨在提供对生成模型和图表示的理论理解，而且还将作为将深度表示学习从黑盒方法转变为可解释且值得信赖的方法的关键步骤。该项目将使学术界、政府实验室和工业界的研究人员受益，并将为本科生和研究生提供应用数学、工程和数据科学方面的跨学科培训。与 MIT-IBM Watson AI 实验室的合作提供互补的技术技能和行业角度，将增加本科生和研究生的职业机会。该项目的目标是：i) 将 MFG 中的理论和分析工具与深度生成模型和图形表示学习联系起来，ii) 通过双层优化的视角，探索 MFG 引导的表示学习中新的数据驱动架构设计。第一个目标是将实际的归一化流理解为变分 MFG，其中 MFG 中的可逆粒子轨迹可以自然地视为归一化流中的生成和归一化方向。第二个目标是提出一种基于 MFG 的图表示学习新框架。这种使用图结构数据进行建模的新方法克服了从图同构角度研究的消息传递框架的局限性，从而产生了训练速度更快且更具可扩展性的新网络架构。为了补充 MFG 中基于专家的依赖项和架构选择，将研究一种新的双层优化方法，以共同学习前两个目标中的模型依赖项、架构和参数。研究团队的互补专业知识正在通过基于 MFG、图形和优化的方法来丰富深度学习的理论基础。该研究议程预计将促进表示学习、图形学习、双层优化、信号处理和控制理论交叉领域的多学科努力。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has had tremendous success in many science and engineering applications. It revolutionizes classical model-based methods using data-driven approaches in a computationally tractable way. Representation learning, which extracts useful information from raw data needed for downstream tasks (for example, classification, regression, and reinforcement learning), is one of the most important directions of deep learning. Despite its success in areas such as image and signal processing, speech and object recognition, natural language processing, chemistry and drug discovery, theoretical understanding of representation learning is far from satisfactory. This project aims at new understanding of representation learning (in particular, deep generative models and graph representation learning) using mean-field game (MFG) theory. The research is intended not only to provide a theoretical understanding of generative models and graph representations but also to serve as a key step in transforming deep representation learning from a black-box approach to an explainable and trustworthy method. The project will benefit researchers in academia, government labs, and industry and will provide interdisciplinary training in applied mathematics, engineering, and data science to undergraduate and graduate students. Collaboration with the MIT-IBM Watson AI Lab, which offers complementary technical skills and industrial angles, will enhance career opportunities for undergraduate and graduate students. This project aims at i) bridging theoretical and analytical tools in MFG with deep generative models and graph representation learning, and ii) exploring new data-driven architecture designs in MFG-guided representation learning through the lens of bi-level optimization. This first objective is to understand the practical normalizing flows as the variational MFG, where reversible particle trajectories in MFG can be naturally viewed as the generative and normalizing directions in normalizing flows. The second objective is to propose a new framework for graph representation learning based on MFG. This new way of modeling with graph-structured data overcomes the limitation of the message passing framework studied from the graph isomorphism angle, leading to new network architectures that are faster and more scalable to train. To complement the expert-based choice of the dependencies and architectures in MFG, a new bi-level optimization approach will be investigated to jointly learn model dependences, architectures, and parameters in the first two objectives. The complementary expertise of the research team is being leveraged to enrich the theoretical foundations of deep learning through MFG-, graph-, and optimization-based approaches. This research agenda is expected to foster multidisciplinary efforts at the intersection of representation learning, graph learning, bi-level optimization, signal processing, and control theory.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.