III: Small: Collaborative Research: Efficient, Nonparametric and Local-Minimum-Free Latent Variable Models: With Application to Large-Scale Computer Vision and Genomics

从计算机视觉到生物学的许多现代应用都需要基于具有多模态、偏度和丰富潜在结构的分布来建模和推断高维连续变量。该体系中的大多数现有模型严重依赖于参数假设，其中模型的组成部分通常被假设为离散或多元高斯，或者变量之间的关系是线性的，这可能与实际的数据生成过程有很大不同。此外，现有的发现潜在依赖结构和学习潜在参数的算法在很大程度上局限于期望最大化等局部搜索启发式算法。在这些受限假设和次优解下得出的结论可能会误导，如果基本假设被违反，或者次优解与全局最优解相差很大。该项目旨在开发一种新的框架，该框架可以(i)发现并利用数据中的潜在结构，同时（ii）允许部分处理近乎任意的分布，以及（iii）允许模型以局部最小自由的方式扩展到现代大规模数据集。该项目的关键创新是一种基于分布核嵌入的非参数潜变量建模框架。基本思想是使用核将分布映射到无限维的特征空间，这样分布的后续比较和操作可以通过特征空间操作来实现，如内积、距离、投影、线性变换和谱分析。从概念上讲，该框架将潜在变量模型中的分量，如单变量的边际分布、变量对、三元组和更多变量的联合分布，分别表示为无限维向量、矩阵、张量和高阶张量。这些成分之间的概率关系，即条件分布、求和规则、乘积规则等，成为这些特征空间成分之间的线性变换和关系。该框架支持对具有不同统计特征的数据进行建模，而无需对分布和关系的类型做出限制性假设。它支持大量线性和多线性代数（张量）工具的应用，用于解决存在潜在变量的具有挑战性的图形模型问题，包括结构发现、推理、参数学习和潜在特征提取。该框架不仅适用于一般的连续变量，还适用于在字符串、图、群、紧流形和其他可以定义内核的领域上取值的变量。除了推进机器学习的最新技术，该项目产生的新的非参数方法在图像数据和理解以及基因表达数据分析中也有应用。它还为佐治亚理工学院和CMU的研究生和本科生的研究培训做出了贡献。

Many modern applications ranging from computer vision to biology require modeling and inferring high-dimensional continuous variables based on distributions with multimodality, skewness, and rich latent structures. Most existing models in this regime rely heavily on parametric assumptions where the components of the model are typically assumed to be discrete or multivariate Gaussian, or the relations between variables are linear, which may be very different from the actual data generating processes. Furthermore, existing algorithms for discovering the latent dependency structures and learning the latent parameters largely are restricted to local search heuristics such as expectation maximization. Conclusions inferred under these restricted assumptions and suboptimal solutions can be misleading, if the underlying assumptions are violated or if the suboptimal solutions differ greatly from the globally optimal ones. This project aims to develop a novel framework which can (i) discover and take advantage of latent structures in the data, while (ii) allowing parts to handle near-arbitrary distributions, and (iii) allowing the models to scale to modern massive datasets in a local-minimum-free fashion. The key innovation in the project is a novel nonparametric latent variable modeling framework based on kernel embedding of distributions. The basic idea is to map distributions into infinite dimensional feature spaces using kernels, such that subsequent comparisons and manipulations of distributions can be achieved via feature space operations, such as inner products, distances, projections, linear transformations and spectral analysis. Conceptually, the framework represents components from latent variable models, such as marginal distributions over a single variable, joint distributions over variable pairs, triplets and more variables, as infinite dimensional vectors, matrices, tensors and high-order tensors respectively. Probabilistic relations between these components, i.e., conditional distributions, Sum Rule, Product Rule etc. become linear transformations and relations between these feature space components.The framework supports modeling data with diverse statistical features without the need for making restrictive assumptions about the type of distributions and relations. It supports the application of a large pool of linear and multi-linear algebraic (tensor) tools for addressing challenging graphical model problems in the presence of latent variables, including structure discovery, inference, parameter learning and latent feature extraction. The framework applies not only to general continuous variables, but also to variables that take values on strings, graphs, groups, compact manifolds, and other domains on which kernels may be defined.Besides advancing the state of the art in machine learning,the new non-parametric methods resulting from the project find applications in image data and understanding and gene expression data analysis. It also contributes to research-based training of graduate and undergraduate students at Georgia Tech and CMU.