EAGER: Search-Accelerated Markov Chain Monte Carlo Algorithms for Bayesian Neural Networks and Trillion-Dimensional Problems

使用深度神经网络的机器学习出现在每个科学领域。在天文学中，机器学习（ML）的应用非常广泛。 高级ML方法涉及大量可调变量，参数范围从数百万到数万亿。目前用于调整这些参数的方法可以找到一个最佳拟合模型，但无法产生与观察结果匹配的ML模型范围。了解给定方法的不确定性范围在科学中具有根本的重要性。更广泛地说，在科学之外的许多领域，了解不确定性的范围很重要（例如，无人驾驶汽车、医疗和军事应用）。该提案引入了一种新的算法，将探索万亿维模型及更高维度的不确定性（包括当前大型神经网络的维度）。对于这样的万亿维参数空间，该方法将比以前最好的方法快500倍。这种方法可以彻底改变科学和商业领域的机器学习。如果能够证明所有合理的神经网络都能给出类似的结果，将直接解决目前的鲁棒性和可重复性问题，并严格量化模型的不确定性。该提案还涉及与退伍军人从军队过渡到大学的直接更广泛的影响工作。该提案将推进采样算法性能的前沿，具有典型的O（log D）或更好的维度缩放，并提供有关高维后验分布几何形状的新知识，包括深度神经网络的几何形状。提高采样算法性能将允许在天文学中解决广泛的问题，否则这些问题将在计算上难以处理，特别是重建问题（例如，用于宇宙学或混合源重建的场级推断）和高维建模（例如，同时模拟多个星系的物理性质的联合分布，给出它们的光度、空间和红移分布）。推进对高维后验分布几何结构的理解将能够开发更优化的探索和采样算法，特别是对于高级神经网络，进一步减少鲁棒后验分布的障碍。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning with deep neural networks occurs in every scientific field. In astronomy, machine learning (ML) applications are wide-ranging. Advanced ML methods involve large numbers of tunable variables, in the range of millions to trillions of parameters. Current methods for tuning these parameters can find a single best-fitting model but are unable to produce the range of ML models that match observations. Knowing the range of uncertainties for a given method is of fundamental importance in science. More broadly, there are many areas beyond science where knowing the range of uncertainties is important (e.g., driverless cars, medical, and military applications). This proposal introduces a new algorithm that will explore uncertainties for trillion-dimensional models and beyond (encompassing the dimensionality of current large neural networks). For such trillion-dimensional parameter spaces, the method would be five hundred times faster than the best previous approaches. This method could revolutionize machine learning across scientific and commercial fields. The ability to show that all reasonable neural networks give similar results would directly address present problems of robustness and reproducibility, as well as rigorously quantify model uncertainties. This proposal also involves direct broader impact work with veterans transitioning from the military to college. This proposal would advance the frontiers of sampling algorithm performance, with typical O(log D) or better scaling with dimensionality, as well as provide new knowledge about the geometry of high-dimensional posterior distributions, including those for deep neural networks. Advancing sampling algorithm performance would allow a broad range of problems to be addressed in astronomy that would otherwise be computationally intractable, especially reconstruction problems (e.g., field-level inference for cosmology or blended source reconstruction) and high-dimensional modeling (e.g., modeling the joint distribution of physical properties of multiple galaxies simultaneously, given their luminosity, spatial, and redshift distributions). Advancing understanding of the geometry of high-dimensional posterior distributions would enable the development of more optimized algorithms for exploration and sampling, particularly for advanced neural networks, further reducing the barriers to robust posterior distributions. The results of this proposal will be released as an open-source implementation of the algorithm as well as an open-access accompanying paper.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.