CAREER: Learning Structured Representations with Deep Probabilistic Programs

编程语言可以在机器学习研究的民主化中发挥决定性作用。在深度学习中，编程框架使得以模块化的方式定义神经网络成为可能，甚至成为常规。这导致了研究的爆炸式增长，在计算机视觉，自然语言处理和强化学习方面取得了突破。拟议的工作将开发深度概率编程语言，该语言训练神经网络在基于仿真的模型中执行推理。这些语言将通过开发模型来帮助社区应对人工智能研究中的新挑战，这些模型将归纳偏见纳入不确定性的推理，并改善有限数据的泛化。在物理科学的应用中，归纳偏差可以包含我们对问题领域的物理知识。更一般地说，概率程序帮助我们表示模型结构，例如推理动作如何影响场景中的对象。所提出的工作地址的技术挑战是扩展概率程序中的推理方法。为此，研究人员将开发一种推理编程语言，允许用户优化特定模型的推理方法。推理方法是根据观测数据对程序中未知变量的后验分布进行推理。随机变分方法通过训练接受数据作为输入并返回变量分布的神经网络来近似后验。这种策略在未知变量采用非结构化向量形式的简单模型中效果很好。然而，在具有更复杂结构的模型中，有效的推理通常需要关于条件独立性的推理。这对于以编程方式指定的模型来说是具有挑战性的，其中关于模型结构的推理需要程序分析。为了应对这一挑战，研究人员将开发一种基于两种结构的推理语言。第一种是模型组合子，它定义了一种一阶语言，用于以允许我们推理条件独立性的方式编写黑盒程序。第二个是推理组合器，它可以用于将按构造重要性校正的采样操作应用于模型的特定组件。模型和推理组合器将允许用户为特定模型开发正确和有效的随机变分方法。除了开发这些基本的抽象并证明其正确性外，研究人员还将展示这些方法在应用于少数拍摄深度生成模型和结构化能量模型中的实用性。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Programming languages can play a decisive role in democratizing machine learning research. In deep learning, programming frameworks have made it possible – and even routine – to define neural networks in a modular manner. This has led to an explosion of research, with breakthroughs in computer vision, natural language processing, and reinforcement learning. The proposed work will develop deep probabilistic programming languages, which train neural networks to perform inference in simulation-based models. These languages will help the community address emerging challenges in artificial intelligence research by developing models that incorporate inductive biases to reason about uncertainty and improve generalization from limited data. In applications in the physical sciences, inductive biases can incorporate our physical knowledge of a problem domain. More generally, probabilistic programs help us represent model structure, for example to reason about how actions affect objects in a scene. The technical challenge that the proposed work addresses is scaling up methods for inference in probabilistic programs. To do so, the investigators will develop a language for inference programming, which will allow users to optimize the inference approach for a specific model. Inference methods reason about the posterior distribution over unknown variables in a program in light of observed data. Stochastic variational methods approximate the posterior by training a neural network that accepts data as input and returns a distribution over variables. This strategy works well in simple models in which unknown variables take the form of an unstructured vector. However, in models with more complex structure, efficient inference often requires reasoning about conditional independence. This is challenging for programmatically specified models, where reasoning about model structure requires program analysis. To address this challenge, the investigators will develop an inference language based on two constructs. The first are model combinators, which define a first-order language for composing black-box programs in a manner that allows us to reason about conditional independence. The second are inference combinators, which may be used to apply correct-by-construction importance sampling operations to specific components of the model. Together, model and inference combinators will allow users to develop correct and efficient stochastic variational methods for specific models. In addition to developing these fundamental abstractions and proving their correctness, the investigators will demonstrate the utility of these methods in applications to few-shot deep generative models, and structured energy-based models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.