Scalable Algorithms for Bayesian On-Line Learning with Large-Scale Dynamic Data

贝叶斯方法为评估大数据机器学习中的模型不确定性提供了一种原则性的方法，这对可信人工智能(AI)的发展至关重要。然而，缺乏高效的蒙特卡罗算法极大地阻碍了贝叶斯方法在大数据时代的应用。与频域方法相比，贝叶斯方法通常要慢得多。为了解决这一困难，在最近的文献中已经开发了各种可伸缩的蒙特卡罗算法。然而，这些算法只能应用于静态数据；它们都不能直接应用于动态数据。以数据科学为核心的许多问题，如自然语言处理、自动驾驶和天气预报，都面临着动态数据的挑战。传统的粒子滤波或顺序蒙特卡罗算法缺乏处理大规模动态数据所需的可扩展性。通过在朗之万动力学框架下重新定义集合卡尔曼滤波(EnKF)，该项目提出了朗之万化的EnKF作为一种通用的可扩展的随机梯度序贯蒙特卡罗算法，用于大规模动态数据的贝叶斯在线学习。朗之万化的EnKF改进了对一大类数据同化问题的不确定性量化，推动了可信人工智能的发展。该项目的成功完成将产生一套可扩展的、理论上严格的贝叶斯在线学习算法，这将为数据驱动技术的发展提供重大好处。研究结果将通过合作、出版物和会议演示向感兴趣的社区传播。该项目还将通过研究生的直接参与和将研究成果纳入本科生和研究生课程，对教育产生重大影响。尽管EnKF在处理海洋学、水库建模和天气预报中遇到的复杂动态数据方面取得了极大的成功，但它并不收敛到正确的滤波分布，除了大集合极限中的线性系统。朗之万化的EnKF解决了这个问题；它在数据同化中收敛到正确的过滤分布，从而能够量化基本动力系统的不确定性。朗之万化的EnKF也可以用于大规模统计数据的贝叶斯学习，通过使用朗之万动力学和次抽样技术将贝叶斯反问题重新表述为状态空间模型。朗之万化EnKF的不同变体将被开发，以将其应用扩展到非高斯数据和不完整数据。总体来说，该项目将为大数据的贝叶斯分析提供一个完整的治疗方案。朗之化的EnKF可以应用于各种数据场景下的大数据问题：动态数据和静态数据，高斯数据和非高斯数据，完整数据和不完整数据，只要数据的分类方式不同。朗之万化的EnKF背后的统计理论将被严格研究。将进行令人兴奋的科学应用，包括语言建模和动态网络分析。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Bayesian methods provide a principled way for assessing model uncertainty in machine learning of big data, which is critical to the development of trustworthy artificial intelligence (AI). However, the lack of efficient Monte Carlo algorithms has drastically hindered applications of Bayesian methods in the big data era. Compared to frequentist methods, Bayesian methods are often much slower. To tackle this difficulty, a variety of scalable Monte Carlo algorithms have been developed in the recent literature. However, these algorithms can only be applied to static data; none of them can be directly applied to dynamic data. Many of the problems centering data science, such as natural language processing, autonomous car driving and weather forecasting, are facing challenges of dynamic data. The traditional particle filters or sequential Monte Carlo algorithms lack the scalability necessary for dealing with large-scale dynamic data. By reformulating the ensemble Kalman filter (EnKF) under the framework of Langevin dynamics, this project proposes Langevinized EnKF as a general and scalable stochastic gradient sequential Monte Carlo algorithm for Bayesian on-line learning with large-scale dynamic data. The Langevinized EnKF improves uncertainty quantification for a wide class of data assimilation problems, advancing the development of trustworthy AI. Successful completion of this project will generate a set of scalable and theoretically rigorous algorithms for Bayesian on-line learning, which can provide significant benefits to the development of data driven technologies. The research results will be disseminated to communities of interest via collaborations, publications, and conference presentations. The project will also have significant impacts on education through direct involvement of graduate students and incorporation of the research results into undergraduate and graduate courses. Although the EnKF has been extremely successful in dealing with complex dynamic data encountered in oceanography, reservoir modeling and weather forecasting, it does not converge to the right filtering distribution except for linear systems in the large ensemble limit. The Langevinized EnKF resolves this issue; it converges to the right filtering distribution in data assimilation and is thus able to quantify uncertainty of the underlying dynamic system. The Langevinized EnKF can also be used for Bayesian learning with large-scale statistic data by reformulating the Bayesian inverse problem as a state-space model with Langevin dynamics and the subsampling technique. Different variants of the Langevinized EnKF will be developed to extend its applications to non-Gaussian data and incomplete data. As the whole, this project will provide a complete treatment for Bayesian analysis of big data. The Langevinized EnKF can be applied to big data problems in various data scenarios: dynamic data and static data, Gaussian data and non-Gaussian data, and complete data and incomplete data, provided the data is classified in different ways. Statistical theory underlying the Langevinized EnKF will be rigorously studied. Exciting scientific applications, including language modeling and dynamic network analysis, will be conducted.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.