RI: Small: Efficient Bayesian Learning from Stochastic Gradients

据估计，2009年的数据总量为0.8泽字节（1泽字节= 1万亿千兆字节），预计到2020年将增长到惊人的35泽字节，每两年翻一番。因此，机器学习的主要挑战之一是开发统计原则方法，这些方法将扩展到非常大的数据集。此外，我们希望（有效地）学习高度复杂的模型，而不必担心过度拟合，并且对我们的预测具有信心。虽然贝叶斯方法满足这些后者desiderata，目前国家的最先进的推理程序的基础上马尔可夫链蒙特卡罗（MCMC）后验抽样不符合“大数据”的挑战。我们提出了一个新的家庭的MCMC程序，通常只需要几百个数据的情况下，每次更新。这些“随机梯度MCMC采样器”继承了随机近似方法的效率，但将从正确的后验分布渐近采样。这就赋予了这类方法一个“随时”的属性，即人们可以从后验的粗略近似中廉价地采样，但可以获得更准确的样本以换取更多的计算。我们相信这类新的方法将首次释放贝叶斯方法在非常大的数据集上的全部力量。由于其高度实用的性质，在此资助下开发的技术可能会在广泛的学科和行业中获得广泛的认可。为了加快转移过程，我们将在我们的网页上发布开源软件，并与一家公司（ID Analytics）合作，研究现实的大规模推理问题。加州大学欧文分校（UCI）的两名学生将利用这笔赠款与英国（牛津大学和布里斯托大学）的一些学生和博士后合作。UCI和英国学生每年也将交换几周，以交叉研究并获得国际经验。这笔赠款的研究成果将通过课堂项目整合到UCI的人工智能和机器学习课程中。

The total volume of data was estimated to be 0.8 Zettabytes in 2009 (1 Zettabyte = 1 trillion gigabytes) and predicted to grow to a staggering 35 Zettabytes in 2020, doubling every two years. Therefore, one of the primary challenges for machine learning is to develop statistically principled methods that will scale up to very large datasets. Moreover, we would like to (efficiently) learn highly complex models without the worry of overfitting and with confidence levels on our predictions. While Bayesian methods satisfy these latter desiderata, the current state-of-the-art inference procedures based on Markov Chain Monte Carlo (MCMC) posterior sampling do not meet the "big-data" challenge.We propose a new family of MCMC procedures that typically requires only a few hundred data-cases per update. These "stochastic gradient MCMC samplers" inherit the efficiencies of stochastic approximation methods, but will asymptotically sample from the correct posterior distribution. This endows this family of methods with an "anytime" property, namely that one can sample cheaply from a rough approximation of the posterior but can obtain more accurate samples in exchange for more computation.We believe this new class of methods will for the first time unlock the full strength of Bayesian methods for very large datasets. Due to their highly practical nature, the techniques developed under this grant are likely to gain widespread acceptance across a broad spectrum of academic disciplines as well as in industry. To expedite the transfer process we will publish open source software on our webpages and collaborate with a company (ID Analytics) to work on realistic, large scale inference problems. Two students at the University of California, Irvine (UCI) will be employed on this grant who will collaborate with a number of students and postdocs in the UK (University of Oxford and University of Bristol). UCI and UK students will also be exchanged for a few weeks a year to cross-fertilize research and to gain international experience. Research results from this grant will be integrated into artificial intelligence and machine learning courses at UCI through class projects.