III: Small: Collaborative Research: Efficient, Nonparametric and Local-Minimum-Free Latent Variable Models: With Application to Large-Scale Computer Vision and Genomics

从计算机视觉到生物学的许多现代应用都需要基于分布的高维连续变量的建模和推断，该分布具有多峰、偏度和丰富的潜在结构。大多数现有的模型在很大程度上依赖于参数假设，其中模型的组件通常被假设为离散或多元高斯，或者变量之间的关系是线性的，这可能与实际的数据生成过程有很大不同。此外，现有的发现潜在依赖结构和学习潜在参数的算法大多局限于局部搜索启发式算法，如期望最大化。如果违反了基本假设，或者如果次优解与全局最优解有很大不同，则根据这些受限假设和次优解得出的结论可能具有误导性。该项目旨在开发一种新的框架，该框架可以(I)发现并利用数据中的潜在结构，同时(Ii)允许部分处理近乎任意的分布，以及(Iii)允许模型以局部最小值自由的方式扩展到现代海量数据集。该项目的关键创新之处在于提出了一种基于分布核嵌入的非参数潜变量建模框架。其基本思想是使用核将分布映射到无限维特征空间，从而可以通过特征空间运算，如内积、距离、投影、线性变换和谱分析来实现对分布的后续比较和处理。从概念上讲，该框架将潜在变量模型中的分量分别表示为无限维向量、矩阵、张量和高阶张量，例如单个变量上的边际分布、变量对上的联合分布、三元组上的联合分布和更多变量上的联合分布。这些成分之间的概率关系，即条件分布、求和规则、乘积规则等转化为这些特征空间成分之间的线性变换和关系，该框架支持具有不同统计特征的数据建模，而不需要对分布和关系的类型做出限制性假设。它支持应用大量的线性和多线性代数(张量)工具来处理存在潜在变量的具有挑战性的图形模型问题，包括结构发现、推理、参数学习和潜在特征提取。该框架不仅适用于一般的连续变量，还适用于取值于字符串、图、群、紧致流形和其他可定义核的域上的变量。除了促进机器学习的最新进展外，该项目产生的新的非参数方法在图像数据和理解以及基因表达数据分析中得到了应用。它还有助于佐治亚理工学院和CMU的研究生和本科生的研究型培训。

Many modern applications ranging from computer vision to biology require modeling and inferring high-dimensional continuous variables based on distributions with multimodality, skewness, and rich latent structures. Most existing models in this regime rely heavily on parametric assumptions where the components of the model are typically assumed to be discrete or multivariate Gaussian, or the relations between variables are linear, which may be very different from the actual data generating processes. Furthermore, existing algorithms for discovering the latent dependency structures and learning the latent parameters largely are restricted to local search heuristics such as expectation maximization. Conclusions inferred under these restricted assumptions and suboptimal solutions can be misleading, if the underlying assumptions are violated or if the suboptimal solutions differ greatly from the globally optimal ones. This project aims to develop a novel framework which can (i) discover and take advantage of latent structures in the data, while (ii) allowing parts to handle near-arbitrary distributions, and (iii) allowing the models to scale to modern massive datasets in a local-minimum-free fashion. The key innovation in the project is a novel nonparametric latent variable modeling framework based on kernel embedding of distributions. The basic idea is to map distributions into infinite dimensional feature spaces using kernels, such that subsequent comparisons and manipulations of distributions can be achieved via feature space operations, such as inner products, distances, projections, linear transformations and spectral analysis. Conceptually, the framework represents components from latent variable models, such as marginal distributions over a single variable, joint distributions over variable pairs, triplets and more variables, as infinite dimensional vectors, matrices, tensors and high-order tensors respectively. Probabilistic relations between these components, i.e., conditional distributions, Sum Rule, Product Rule etc. become linear transformations and relations between these feature space components.The framework supports modeling data with diverse statistical features without the need for making restrictive assumptions about the type of distributions and relations. It supports the application of a large pool of linear and multi-linear algebraic (tensor) tools for addressing challenging graphical model problems in the presence of latent variables, including structure discovery, inference, parameter learning and latent feature extraction. The framework applies not only to general continuous variables, but also to variables that take values on strings, graphs, groups, compact manifolds, and other domains on which kernels may be defined.Besides advancing the state of the art in machine learning,the new non-parametric methods resulting from the project find applications in image data and understanding and gene expression data analysis. It also contributes to research-based training of graduate and undergraduate students at Georgia Tech and CMU.