Representation Learning via Variational Mean Field Theory

深度学习在许多科学和工程应用中取得了巨大的成功。它以一种易于计算的方式使用数据驱动的方法彻底改变了传统的基于模型的方法。表征学习是深度学习的一个重要方向，它从下游任务(如分类、回归、强化学习等)所需的原始数据中提取有用的信息。尽管它在图像和信号处理、语音和物体识别、自然语言处理、化学和药物发现等领域取得了成功，但对表征学习的理论理解还远远不能令人满意。本项目旨在用平均场博弈(MFG)理论对表征学习(特别是深度生成模型和图形表征学习)有新的理解。本研究不仅旨在为生成模型和图形表示提供理论上的理解，而且也是将深度表示学习从黑盒方法转变为可解释和可信的方法的关键一步。该项目将使学术界、政府实验室和工业领域的研究人员受益，并将为本科生和研究生提供应用数学、工程和数据科学方面的跨学科培训。与麻省理工学院-IBM沃森人工智能实验室的合作，提供互补的技术技能和行业角度，将增加本科生和研究生的就业机会。该项目的目标是i)将MFG中的理论和分析工具与深度生成模型和图形表示学习联系起来，以及ii)通过双层优化的镜头在MFG引导的表示学习中探索新的数据驱动的体系结构设计。第一个目标是将实际的正规化流动理解为变分MFG，其中MFG中的可逆粒子轨迹可以自然地视为正规化流动的生成和正规化方向。第二个目标是提出一个新的基于MFG的图表示学习框架。这种用图结构数据建模的新方法克服了从图同构的角度研究消息传递框架的局限性，导致了新的网络结构，其训练速度更快，可扩展性更强。为了补充MFG中基于专家的依赖关系和体系结构的选择，将研究一种新的双层优化方法来联合学习前两个目标中的模型依赖关系、体系结构和参数。研究团队的互补专业知识正被用来通过基于MFG、图表和优化的方法来丰富深度学习的理论基础。这一研究议程预计将促进在表示学习、图形学习、双层优化、信号处理和控制理论的交叉领域的多学科努力。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has had tremendous success in many science and engineering applications. It revolutionizes classical model-based methods using data-driven approaches in a computationally tractable way. Representation learning, which extracts useful information from raw data needed for downstream tasks (for example, classification, regression, and reinforcement learning), is one of the most important directions of deep learning. Despite its success in areas such as image and signal processing, speech and object recognition, natural language processing, chemistry and drug discovery, theoretical understanding of representation learning is far from satisfactory. This project aims at new understanding of representation learning (in particular, deep generative models and graph representation learning) using mean-field game (MFG) theory. The research is intended not only to provide a theoretical understanding of generative models and graph representations but also to serve as a key step in transforming deep representation learning from a black-box approach to an explainable and trustworthy method. The project will benefit researchers in academia, government labs, and industry and will provide interdisciplinary training in applied mathematics, engineering, and data science to undergraduate and graduate students. Collaboration with the MIT-IBM Watson AI Lab, which offers complementary technical skills and industrial angles, will enhance career opportunities for undergraduate and graduate students. This project aims at i) bridging theoretical and analytical tools in MFG with deep generative models and graph representation learning, and ii) exploring new data-driven architecture designs in MFG-guided representation learning through the lens of bi-level optimization. This first objective is to understand the practical normalizing flows as the variational MFG, where reversible particle trajectories in MFG can be naturally viewed as the generative and normalizing directions in normalizing flows. The second objective is to propose a new framework for graph representation learning based on MFG. This new way of modeling with graph-structured data overcomes the limitation of the message passing framework studied from the graph isomorphism angle, leading to new network architectures that are faster and more scalable to train. To complement the expert-based choice of the dependencies and architectures in MFG, a new bi-level optimization approach will be investigated to jointly learn model dependences, architectures, and parameters in the first two objectives. The complementary expertise of the research team is being leveraged to enrich the theoretical foundations of deep learning through MFG-, graph-, and optimization-based approaches. This research agenda is expected to foster multidisciplinary efforts at the intersection of representation learning, graph learning, bi-level optimization, signal processing, and control theory.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.