RI: Small: New Directions in Probabilistic Deep Learning: Exponential Families, Bayesian Nonparametrics and Empirical Bayes

深度学习（DL）为现代机器学习（ML）提供了一个强大的范例，在自然语言处理、计算机视觉和机器人等一系列领域都有应用。深度学习已经被证明是强大的，因为它可以捕获输入和输出之间的复杂关系，并且它具有分析大量数据集的有效算法。但是DL也有局限性。目前，深度学习经常提供“黑盒”预测。黑箱表示结果不涉及明确的假设或结果的确定性程度。要在重要的应用程序中使用ML，了解这些方法基于哪些假设是至关重要的。其次，基本的深度学习方法提供点预测，但不提供它们的不确定性。为了将机器学习安全地部署在关键的决策系统中，机器学习方法必须提供关于其预测可靠性的校准测量。最后，所有这些问题都意味着深度学习不能提供容易解释的预测。可解释性对于理解机器学习如何犯错、在需要问责制的高风险环境中部署机器学习以及在科学理解中使用机器学习预测非常重要。这个跨学科项目通过使用严格的概率机器学习方法和应用贝叶斯统计来形成可解释的深度学习模型来解决这些问题。这些模型将基于明确陈述的假设，并为其预测提供校准后的不确定性。本研究旨在解决深度学习中的开放问题，为其经验证明的一些想法提供数学清晰度，并将其扩展到概率建模，以广泛应用于天文学，计算社会科学的语言建模和电子医疗记录。该项目将通过两个主题的研究，将深度学习中的现代思想应用于复杂数据集的现代概率模型。第一个主题发展了概率深度学习的基础，阐明了深度神经网络模型如何从指数族和广义线性模型等经典思想中汲取灵感，并将深度学习扩展到无限深度的贝叶斯非参数模型。第二个主题是发展经验贝叶斯表示学习。表示学习是深度学习的基石，它是关于寻找高维数据的低维描述。但从统计学的角度来看，问题在于许多表示都不能准确地捕捉数据的分布。这个项目将探索经验贝叶斯的强大概念，一个融合了频率论和贝叶斯思想的经典统计思想，如何为定义好的表征提供一个自然的框架。通过新的理论、算法和软件，该项目将显著扩展深度学习的能力。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning (DL) provides a powerful paradigm for modern machine learning (ML), with applications in a range of areas, such as natural language processing, computer vision and robotics. DL has proven powerful because it can capture complex relationships between input and output, and it enjoys efficient algorithms for analyzing massive datasets. But DL also has limitations. Currently, DL often provides ‘black box’ predictions. Black box indicates that the results do not involve clearly articulated assumptions or the degree of certainty in the results. To use ML in important applications, it is crucial to know from which assumptions the methods are based. Second, basic DL methods provide point predictions, but do not provide uncertainty about them. For ML to be safely deployed in critical decision-making systems, ML methods must provide calibrated measurement about the reliability of its predictions. Finally, all of these issues mean that DL does not provide easily interpretable predictions. Interpretability is important for understanding how ML makes mistakes, for deploying ML in high-stakes settings that require accountability, and when using ML predictions in the service of scientific understanding. This interdisciplinary project addresses these issues by using the rigorous methodology of probabilistic ML and applied Bayesian statistics to form interpretable DL models. Models that will be based on clearly stated assumptions and provide calibrated uncertainty about their predictions. This research aims to solve open problems in DL, provide mathematical clarity to some of its empirically proven ideas, and expand its reach to probabilistic modeling for broad applications in astronomy, language modeling for the computational social sciences, and electronic healthcare records.The project will adapt modern ideas in DL for modern probabilistic models of complex datasets through research in two topics. The first topic develops the foundations of probabilistic deep learning, clarifying how deep neural network models draw from classical ideas like exponential families and generalized linear models, and expanding DL to Bayesian nonparametric models of infinite depth. The second topic develops empirical Bayes representation learning. Representation learning, a cornerstone of DL, is about finding low-dimensional descriptions of high-dimensional data. But from a statistical perspective, the problem is that many representations can accurately capture the distribution of the data. This project will explore how the powerful concept of empirical Bayes, a classical statistical idea that blends frequentist and Bayesian thinking, provides a natural framework for defining good representations. Through new theory, algorithms, and software, this project will significantly expand the capabilities of DL.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.