Continuous latent variable models for dimensionality reduction and sequential data reconstruction
Continuous latent variable models for dimensionality reduction and sequential data reconstruction
复制标题
DOI：
--
复制
发表时间：
2001
期刊：
影响因子：
--
通讯作者：
M. A. Carreira-Perpiñán
中科院分区：
文献类型：
--
作者：
M. A. Carreira-Perpiñán
连续潜变量模型（英语：Continuous latent variable models，cLVMs）是一种概率模型，它使用少量的连续潜变量来表示高维欧氏空间中的分布。本文从理论和实践两个方面探讨了cLVMs的降维和序列数据重构能力。论文的第一部分回顾和扩展了cLVMs的理论：定义了潜在空间中的先验分布、到数据空间的映射和噪声模型;使用期望最大化（EM）算法的最大似然参数估计;特定的cLVMs（因子分析、主成分分析（PCA）、独立成分分析、独立因子分析和生成式地形图（GTM））; cLVMs的混合;可识别性、可解释性和可视化;以及导出用于降维和重建的映射及其属性，例如每个cLVM的连续性。我们将GTM推广到对角噪声，并给出了相应的EM算法。我们还描述了一个离散的二进制数据，伯努利混合，在实践中广泛使用的LVM。我们表明，他们的对数似然曲面没有奇点，不像其他混合模型，这使得EM估计实用;他们的理论不可识别性很少在实际估计中实现，这使得他们可以解释。第二部分涉及降维。我们定义的问题，并给出了广泛的，批判性的审查非概率方法：线性方法（PCA，投影寻踪），非线性autoassociators，核方法，局部降维，主曲线，矢量量化方法（弹性网络，自组织映射）和多维缩放方法。然后，我们凭经验评估，在重建误差，计算时间和可视化，几个潜在变量的方法，二进制epalatographic（EPG）数据的降维：PCA，因子分析，因子分析，GTM和伯努利混合物的混合物。我们比较这些方法与早期，非自适应EPG数据减少方法，并推导出2D地图的EPG序列用于语音研究和治疗。本文的最后一部分提出了一种新的序列数据缺失数据重建方法，其中包括多对一映射的反演。我们定义的问题，区分它从逆问题，并显示当两者重合。该方法是基于多点重构和约束优化。多点重建使用高斯混合联合密度模型的数据，方便地实现与非线性cLVM（GTM）。给定序列中每个点的当前值，缺失值的条件分布的模式表示局部候选重建。全局序列重构通过利用动态规划有效地优化约束（诸如连续性或平滑性）来获得。我们给出了一个概率的解释的方法。我们推导出两个算法，穷举模式发现高斯混合，基于梯度二次搜索和定点搜索，分别以及估计误差条为每个模式和分布稀疏性的措施。我们讨论的优势，以前的工作的基础上的条件平均或通用映射逼近（包括合奏和经常性网络），条件分布估计，矢量量化和统计分析的缺失数据的方法。我们研究了合成数据（玩具的例子和逆运动学问题）和真实的数据（EPG和声学数据之间的映射）的方法的性能。我们描述了几个著名的重建或反演问题的方法可能的应用：解码海马位置细胞的神经群体活动;风场检索散射计数据;逆运动学和动力学的冗余机械手;声学发音映射;语音识别的视听映射;和闭塞语音识别。
Continuous latent variable models (cLVMs) are probabilistic models that represent a distribution in a high-dimensional Euclidean space using a small number of continuous, latent variables. This thesis explores, theoretically and practically, the ability of cLVMs for dimensionality reduction and sequential data reconstruction. The first part of the thesis reviews and extends the theory of cLVMs: definition in terms of a prior distribution in latent space, a mapping to data space and a noise model; maximum likelihood parameter estimation with an expectation-maximisation (EM) algorithm; specific cLVMs (factor analysis, principal component analysis (PCA), independent component analysis, independent factor analysis and the generative topographic mapping (GTM)); mixtures of cLVMs; identifiability, interpretability and visualisation; and derivation of mappings for dimensionality reduction and reconstruction and their properties, such as continuity, for each cLVM. We extend GTM to diagonal noise and give a corresponding EM algorithm. We also describe a discrete LVM for binary data, Bernoulli mixtures, widely used in practice. We show that their log-likelihood surface has no singularities, unlike other mixture models, which makes EM estimation practical; and that their theoretical non-identifiability is rarely realised in actual estimates, which makes them interpretable. The second part deals with dimensionality reduction. We define the problem and give an extensive, critical review of nonprobabilistic methods for it: linear methods (PCA, projection pursuit), nonlinear autoassociators, kernel methods, local dimensionality reduction, principal curves, vector quantisation methods (elastic net, self-organising map) and multidimensional scaling methods. We then empirically evaluate, in terms of reconstruction error, computation time and visualisation, several latent-variable methods for dimensionality reduction of binary electropalatographic (EPG) data: PCA, factor analysis, mixtures of factor analysers, GTM and Bernoulli mixtures. We compare these methods with earlier, nonadaptive EPG data reduction methods and derive 2D maps of EPG sequences for use in speech research and therapy. The last part of this thesis proposes a new method for missing data reconstruction of sequential data that includes as particular case the inversion of many-to-one mappings. We define the problem, distinguish it from inverse problems, and show when both coincide. The method is based on multiple pointwise reconstruction and constraint optimisation. Multiple pointwise reconstruction uses a Gaussian mixture joint density model for the data, conveniently implemented with a nonlinear cLVM (GTM). The modes of the conditional distribution of missing values given present values at each point in the sequence represent local candidate reconstructions. A global sequence reconstruction is obtained by efficiently optimising a constraint, such as continuity or smoothness, with dynamic programming. We give a probabilistic interpretation of the method. We derive two algorithms for exhaustive mode finding in Gaussian mixtures, based on gradient-quadratic search and fixed-point search, respectively; as well as estimates of error bars for each mode and a measure of distribution sparseness. We discuss the advantages of the method over previous work based on the conditional mean or on universal mapping approximators (including ensembles and recurrent networks), conditional distribution estimation, vector quantisation and statistical analysis of missing data. We study the performance of the method with synthetic data (a toy example and an inverse kinematics problem) and real data (mapping between EPG and acoustic data). We describe the possible application of the method to several well-known reconstruction or inversion problems: decoding of neural population activity for hippocampal place cells; wind field retrieval from scatterometer data; inverse kinematics and dynamics of a redundant manipulator; acoustic-to-articulatory mapping; audiovisual mappings for speech recognition; and recognition of occluded speech.