RI: Small: Efficient Bayesian Learning from Stochastic Gradients

2009年的数据总量估计为0.8 ZB(1 ZB=1万亿GB)，预计到2020年将增长到惊人的35 ZB，每两年翻一番。因此，机器学习的主要挑战之一是开发能够扩展到非常大的数据集的统计原理方法。此外，我们希望(有效地)学习高度复杂的模型，而不必担心过度拟合，并对我们的预测保持一定的信心。虽然贝叶斯方法满足了这些需求，但现有的基于马尔可夫链蒙特卡罗(MCMC)后验抽样的推理方法并不能满足大数据的挑战。这些“随机梯度MCMC采样器”继承了随机逼近方法的效率，但将从正确的后验分布进行渐近采样。这赋予了这类方法“随时”的性质，即人们可以从粗略的后验近似中廉价地采样，但可以获得更准确的样本，以换取更多的计算。我们相信，这类新的方法将首次充分发挥贝叶斯方法在非常大的数据集上的全部优势。由于其高度实用的性质，在这笔赠款下开发的技术可能会在广泛的学术学科和工业中得到广泛接受。为了加快转移过程，我们将在我们的网页上发布开源软件，并与一家公司(ID Analytics)合作解决现实的、大规模的推理问题。加州大学欧文分校(UCI)的两名学生将受雇于这笔助学金，他们将与英国的一些学生和博士后(牛津大学和布里斯托尔大学)合作。UCI和英国的学生每年还将有几周的时间进行交流，相互促进研究并获得国际经验。这笔拨款的研究成果将通过课堂项目整合到UCI的人工智能和机器学习课程中。

The total volume of data was estimated to be 0.8 Zettabytes in 2009 (1 Zettabyte = 1 trillion gigabytes) and predicted to grow to a staggering 35 Zettabytes in 2020, doubling every two years. Therefore, one of the primary challenges for machine learning is to develop statistically principled methods that will scale up to very large datasets. Moreover, we would like to (efficiently) learn highly complex models without the worry of overfitting and with confidence levels on our predictions. While Bayesian methods satisfy these latter desiderata, the current state-of-the-art inference procedures based on Markov Chain Monte Carlo (MCMC) posterior sampling do not meet the "big-data" challenge.We propose a new family of MCMC procedures that typically requires only a few hundred data-cases per update. These "stochastic gradient MCMC samplers" inherit the efficiencies of stochastic approximation methods, but will asymptotically sample from the correct posterior distribution. This endows this family of methods with an "anytime" property, namely that one can sample cheaply from a rough approximation of the posterior but can obtain more accurate samples in exchange for more computation.We believe this new class of methods will for the first time unlock the full strength of Bayesian methods for very large datasets. Due to their highly practical nature, the techniques developed under this grant are likely to gain widespread acceptance across a broad spectrum of academic disciplines as well as in industry. To expedite the transfer process we will publish open source software on our webpages and collaborate with a company (ID Analytics) to work on realistic, large scale inference problems. Two students at the University of California, Irvine (UCI) will be employed on this grant who will collaborate with a number of students and postdocs in the UK (University of Oxford and University of Bristol). UCI and UK students will also be exchanged for a few weeks a year to cross-fertilize research and to gain international experience. Research results from this grant will be integrated into artificial intelligence and machine learning courses at UCI through class projects.