Generative Modeling with Short Run Computing

使用短期计算的生成建模

• 批准号: 2015577
• 负责人: Yingnian Wu
• 金额: $ 20万
• 依托单位: University of California-Los Angeles
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2015577&HistoricalAwards=false
• 关键词: Generative; Modeling; Short; Run; Computing

In our daily lives, we constantly receive a large amount of sensory data in the form of images, texts, and speech, yet we can effortlessly make sense of the data by learning, recognizing, and understanding the patterns and meanings in the data. How this is done by the brain is still largely a mystery, and this is a central problem in machine learning and artificial intelligence, which has a vast scope of applications and is transforming our lives. One way to make sense of sensory data is to construct models to generate them, by assuming that the data are generated by some relatively simple hidden factors or causes. Such models are called generative models. To make sense of the data is to infer the hidden causes that generate the input data, and this can be accomplished by short-run computing dynamics. The main goal of this project is to develop such generative models and the associated short-run computing dynamics. The project has the potential to lead to new learning techniques that can be useful in applications such as computer vision. The PI will also train graduate students supported by this grant and further enhance the graduate and undergraduate courses taught by the PI. Generative models unify supervised, unsupervised, and semi-supervised learning in a principled likelihood-based framework. While supervised learning has met tremendous successes in recent years, unsupervised and semi-supervised learning remains a challenge. The bottleneck for likelihood-based learning of generative models is the intractable computation of expectations which usually requires expensive Markov chain Monte Carlo (MCMC) sampling, whose convergence can be problematic. The main motivation of the research is to get around this bottleneck. The following are specific aims of the proposed research: (1) Variational optimization of short-run MCMC dynamics for the sampling computations in likelihood-based learning of generative models, by combining the advantages of MCMC and variational inference. (2) Developing biologically plausible generative models with multiple layers of hidden variables, and the associated short-run inference and synthesis dynamics that can account for feedbacks and inhibitions between hidden variables.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在我们的日常生活中，我们不断收到大量图像、文本和语音形式的感官数据，但我们可以通过学习、识别和理解数据中的模式和含义，轻松地理解这些数据。大脑是如何做到这一点在很大程度上仍然是一个谜，这是机器学习和人工智能的核心问题，它具有广泛的应用范围，正在改变我们的生活。理解感官数据的一种方法是构建模型来生成它们，假设数据是由一些相对简单的隐藏因素或原因生成的。这样的模型称为生成模型。理解数据就是推断生成输入数据的隐藏原因，这可以通过短期计算动态来完成。该项目的主要目标是开发此类生成模型以及相关的短期计算动态。该项目有潜力带来新的学习技术，可用于计算机视觉等应用。 PI还将在这笔资助的支持下培养研究生，并进一步加强PI教授的研究生和本科生课程。 生成模型将监督、无监督和半监督学习统一在一个原则性的基于可能性的框架中。尽管监督学习近年来取得了巨大成功，但无监督和半监督学习仍然是一个挑战。基于似然的生成模型学习的瓶颈是难以处理的期望计算，这通常需要昂贵的马尔可夫链蒙特卡罗（MCMC）采样，其收敛可能存在问题。这项研究的主要动机是为了绕过这个瓶颈。本研究的具体目标如下：（1）结合 MCMC 和变分推理的优点，对生成模型的基于似然学习中的采样计算进行短期 MCMC 动力学的变分优化。 (2) 开发具有多层隐藏变量的生物学上合理的生成模型，以及相关的短期推理和综合动态，可以解释隐藏变量之间的反馈和抑制。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Molecule Design by Latent Space Energy-Based Modeling and Gradual Distribution Shifting

• DOI: 10.48550/arxiv.2306.14902
• 发表时间: 2023-06
• 作者: Deqian Kong;Bo Pang;Tian Han;Y. Wu

Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering

• DOI: 10.1109/cvpr52688.2022.00501
• 发表时间: 2022-06
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Feng Gao;Q. Ping;G. Thattai;Aishwarya N. Reganti;Yingting Wu;Premkumar Natarajan

Learning Joint Latent Space EBM Prior Model for Multi-layer Generator

• DOI: 10.1109/cvpr52729.2023.00351
• 发表时间: 2023-06
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Jiali Cui;Y. Wu;Tian Han

Latent Diffusion Energy-Based Model for Interpretable Text Modeling

• DOI: 10.48550/arxiv.2206.05895
• 发表时间: 2022-06
• 作者: Peiyu Yu;Sirui Xie;Xiaojian Ma;Baoxiong Jia;Bo Pang;Ruigi Gao;Yixin Zhu;Song-Chun Zhu;Y. Wu

Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification

• DOI: 10.1109/cvpr46437.2021.01473
• 发表时间: 2020-04
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Jianwen Xie;Yifei Xu;Zilong Zheng;Song-Chun Zhu;Y. Wu