Distributed MCMC Inference in Dirichlet Process Mixture Models Using Julia
Distributed MCMC Inference in Dirichlet Process Mixture Models Using Julia
复制标题
DOI：
10.1109/ccgrid.2019.00066
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
Or Dinari;A. Yu;O. Freifeld;John W. Fisher III
中科院分区：
文献类型：
--
作者：
Or Dinari;A. Yu;O. Freifeld;John W. Fisher III
由于大型数据集的可用性越来越高，对通用并行分析工具的需求变得越来越大。在无监督学习中，贝叶斯非参数混合模型，例如狄利克雷过程混合模型（DPMM），提供了一种原则性的贝叶斯方法来适应模型的复杂性。然而，尽管有潜力，DPMM还没有成为一个流行的工具。这部分是由于缺乏友好的软件工具，可以有效地处理大型数据集。在这里，我们将展示如何使用Julia，可以实现高效和易于修改的实施DPMM中的分布式推理。特别是，我们展示了最近的并行MCMC推理算法-最初在C++中实现的单个多核机器-可以有效地分布在多个多核机器使用分布式内存模型。这导致了加速，消除了内存和存储限制，并让我们从更大的数据集和更高的维度学习DPMM。事实证明，即使在一台机器上，Julia实现也比原来的C++实现更优雅地处理更高的维度（至少对于高斯）。最后，我们使用所提出的实现来学习图像补丁的模型，并将学习到的模型应用于图像去噪。虽然我们推测，高度优化的分布式实现，比如C++，可能比Julia中提出的实现更快，但从我们作为机器学习研究人员（而不是HPC研究人员）的角度来看，由于易于开发和抽象级别，后者也提供了实用和货币价值。我们的代码可在https://github.com/dinarior/dpmm subclusters.jl上公开获取
Due to the increasing availability of large data sets, the need for general-purpose massively-parallel analysis tools become ever greater. In unsupervised learning, Bayesian nonparametric mixture models, exemplified by the Dirichlet-Process Mixture Model (DPMM), provide a principled Bayesian approach to adapt model complexity to the data. Despite their potential, however, DPMMs have yet to become a popular tool. This is partly due to the lack of friendly software tools that can handle large datasets efficiently. Here we show how, using Julia, one can achieve efficient and easily-modifiable implementation of distributed inference in DPMMs. Particularly, we show how a recent parallel MCMC inference algorithm - originally implemented in C++ for a single multi-core machine - can be distributed efficiently across multiple multi-core machines using a distributed-memory model. This leads to speedups, alleviates memory and storage limitations, and lets us learn DPMMs from significantly larger datasets and of higher dimensionality. It also turned out that even on a single machine the proposed Julia implementation handles higher dimensions more gracefully (at least for Gaussians) than the original C++ implementation. Finally, we use the proposed implementation to learn a model of image patches and apply the learned model for image denoising. While we speculate that a highly-optimized distributed implementation in, say, C++ could have been faster than the proposed implementation in Julia, from our perspective as machine-learning researchers (as opposed to HPC researchers), the latter also offers a practical and monetary value due to the ease of development and abstraction level. Our code is publicly available at https://github.com/dinarior/dpmm subclusters.jl