Representation Learning via Variational Mean Field Theory

深度学习在许多科学和工程应用中取得了巨大的成功。它以一种计算易于处理的方式使用数据驱动的方法，彻底改变了传统的基于模型的方法。表示学习从下游任务（例如分类、回归和强化学习）所需的原始数据中提取有用信息，是深度学习最重要的方向之一。尽管表征学习在图像和信号处理、语音和对象识别、自然语言处理、化学和药物发现等领域取得了成功，但对表征学习的理论理解还远远不能令人满意。该项目旨在利用平均场博弈（MFG）理论对表示学习（特别是深度生成模型和图表示学习）进行新的理解。该研究的目的不仅是提供对生成模型和图表示的理论理解，而且是将深度表示学习从黑盒方法转变为可解释和可信方法的关键一步。该项目将使学术界、政府实验室和工业界的研究人员受益，并将为本科生和研究生提供应用数学、工程和数据科学方面的跨学科培训。与麻省理工学院- ibm沃森人工智能实验室的合作，提供了互补的技术技能和工业角度，将增加本科生和研究生的就业机会。该项目旨在i)将MFG中的理论和分析工具与深度生成模型和图表示学习连接起来，ii)通过双级优化的视角探索MFG指导下表示学习中新的数据驱动架构设计。第一个目标是将实际的归一化流理解为变分MFG，其中MFG中的可逆粒子轨迹可以自然地视为归一化流中的生成和归一化方向。第二个目标是提出一种新的基于MFG的图表示学习框架。这种利用图结构数据建模的新方法克服了从图同构角度研究的消息传递框架的局限性，从而产生了更快、更易于训练的新网络体系结构。为了补充MFG中基于专家的依赖关系和体系结构选择，将研究一种新的双层优化方法，以联合学习前两个目标中的模型依赖关系、体系结构和参数。利用研究团队的互补专业知识，通过基于MFG、图和优化的方法丰富深度学习的理论基础。本研究议程旨在促进表征学习、图学习、双级优化、信号处理和控制理论交叉领域的多学科努力。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has had tremendous success in many science and engineering applications. It revolutionizes classical model-based methods using data-driven approaches in a computationally tractable way. Representation learning, which extracts useful information from raw data needed for downstream tasks (for example, classification, regression, and reinforcement learning), is one of the most important directions of deep learning. Despite its success in areas such as image and signal processing, speech and object recognition, natural language processing, chemistry and drug discovery, theoretical understanding of representation learning is far from satisfactory. This project aims at new understanding of representation learning (in particular, deep generative models and graph representation learning) using mean-field game (MFG) theory. The research is intended not only to provide a theoretical understanding of generative models and graph representations but also to serve as a key step in transforming deep representation learning from a black-box approach to an explainable and trustworthy method. The project will benefit researchers in academia, government labs, and industry and will provide interdisciplinary training in applied mathematics, engineering, and data science to undergraduate and graduate students. Collaboration with the MIT-IBM Watson AI Lab, which offers complementary technical skills and industrial angles, will enhance career opportunities for undergraduate and graduate students. This project aims at i) bridging theoretical and analytical tools in MFG with deep generative models and graph representation learning, and ii) exploring new data-driven architecture designs in MFG-guided representation learning through the lens of bi-level optimization. This first objective is to understand the practical normalizing flows as the variational MFG, where reversible particle trajectories in MFG can be naturally viewed as the generative and normalizing directions in normalizing flows. The second objective is to propose a new framework for graph representation learning based on MFG. This new way of modeling with graph-structured data overcomes the limitation of the message passing framework studied from the graph isomorphism angle, leading to new network architectures that are faster and more scalable to train. To complement the expert-based choice of the dependencies and architectures in MFG, a new bi-level optimization approach will be investigated to jointly learn model dependences, architectures, and parameters in the first two objectives. The complementary expertise of the research team is being leveraged to enrich the theoretical foundations of deep learning through MFG-, graph-, and optimization-based approaches. This research agenda is expected to foster multidisciplinary efforts at the intersection of representation learning, graph learning, bi-level optimization, signal processing, and control theory.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.