RI: Small: Efficient Bayesian Learning from Stochastic Gradients

RI：小：从随机梯度中进行高效贝叶斯学习

• 批准号: 1216045
• 负责人: Max Welling
• 金额: $ 50万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-09-01 至 2016-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1216045&HistoricalAwards=false
• 关键词: RI; Small; Efficient; Bayesian; Learning

The total volume of data was estimated to be 0.8 Zettabytes in 2009 (1 Zettabyte = 1 trillion gigabytes) and predicted to grow to a staggering 35 Zettabytes in 2020, doubling every two years. Therefore, one of the primary challenges for machine learning is to develop statistically principled methods that will scale up to very large datasets. Moreover, we would like to (efficiently) learn highly complex models without the worry of overfitting and with confidence levels on our predictions. While Bayesian methods satisfy these latter desiderata, the current state-of-the-art inference procedures based on Markov Chain Monte Carlo (MCMC) posterior sampling do not meet the "big-data" challenge.We propose a new family of MCMC procedures that typically requires only a few hundred data-cases per update. These "stochastic gradient MCMC samplers" inherit the efficiencies of stochastic approximation methods, but will asymptotically sample from the correct posterior distribution. This endows this family of methods with an "anytime" property, namely that one can sample cheaply from a rough approximation of the posterior but can obtain more accurate samples in exchange for more computation.We believe this new class of methods will for the first time unlock the full strength of Bayesian methods for very large datasets. Due to their highly practical nature, the techniques developed under this grant are likely to gain widespread acceptance across a broad spectrum of academic disciplines as well as in industry. To expedite the transfer process we will publish open source software on our webpages and collaborate with a company (ID Analytics) to work on realistic, large scale inference problems. Two students at the University of California, Irvine (UCI) will be employed on this grant who will collaborate with a number of students and postdocs in the UK (University of Oxford and University of Bristol). UCI and UK students will also be exchanged for a few weeks a year to cross-fertilize research and to gain international experience. Research results from this grant will be integrated into artificial intelligence and machine learning courses at UCI through class projects.

2009年的数据总量估计为0.8 Zettabyte （1 Zettabyte = 1万亿千兆字节），预计到2020年将增长到惊人的35 Zettabytes，每两年翻一番。因此，机器学习的主要挑战之一是开发统计学上有原则的方法，这些方法将扩展到非常大的数据集。此外，我们希望（有效地）学习高度复杂的模型，而不用担心过度拟合，并且对我们的预测有信心。虽然贝叶斯方法满足后一种需求，但目前基于马尔可夫链蒙特卡罗（MCMC）后验抽样的最先进的推理程序不能满足“大数据”的挑战。我们提出了一组新的MCMC过程，每次更新通常只需要几百个数据用例。这些“随机梯度MCMC采样器”继承了随机逼近方法的有效性，但将逐渐从正确的后验分布中采样。这使得这类方法具有“随时随地”的特性，即可以从后验的粗略近似值中便宜地进行采样，但可以获得更准确的样本，以换取更多的计算。我们相信这类新方法将首次在非常大的数据集上释放贝叶斯方法的全部力量。由于其高度实用的性质，在这项资助下开发的技术可能会在广泛的学术学科和工业领域获得广泛的认可。为了加快传输过程，我们将在我们的网页上发布开源软件，并与一家公司（ID Analytics）合作，解决现实的大规模推理问题。加州大学欧文分校（UCI）的两名学生将被聘用，他们将与英国（牛津大学和布里斯托尔大学）的一些学生和博士后合作。UCI和英国的学生每年还将交换几周的时间，以进行交叉研究并获得国际经验。该基金的研究成果将通过课堂项目整合到UCI的人工智能和机器学习课程中。