Deep Restricted Kernel Machines Using Conjugate Feature Duality
Deep Restricted Kernel Machines Using Conjugate Feature Duality
复制标题
DOI：
10.1162/neco_a_00984
复制
发表时间：
2017-07
影响因子：
2.9
通讯作者：
J. Suykens
中科院分区：
文献类型：
--
作者：
J. Suykens
本文的目的是提出深度受限核机理论，为核机深度学习提供新的基础。从深度学习的角度来看，它部分地与受限Boltzmann机器有关，后者的特征是二部图中的可见单元和隐藏单元没有隐藏到隐藏的连接，以及深度学习扩展，如深度信念网络和深度Boltzmann机器。从核机器的角度来看，它包括用于分类和回归的最小二乘支持向量机、核主成分分析(PCA)、矩阵奇异值分解和Parzen型模型。一个关键因素是首先根据所谓的共轭特征对偶来表征这些核机器，产生具有可见和隐藏单元的表示。结果表明，在非概率设置下，这与连续变量的受限玻尔兹曼机中的能量形式是如何相关的。在这种所谓的受限核机(RKM)表示的新框架中，对偶变量对应于隐藏特征。深度RKM是通过耦合RKM得到的。该方法以深度RKM为例，由三个最小二乘支持向量机回归层和两个核PCA层组成。在其原始形式中，深度前馈神经网络也可以在该框架内训练。
Abstract The aim of this letter is to propose a theory of deep restricted kernel machines offering new foundations for deep learning with kernel machines. From the viewpoint of deep learning, it is partially related to restricted Boltzmann machines, which are characterized by visible and hidden units in a bipartite graph without hidden-to-hidden connections and deep learning extensions as deep belief networks and deep Boltzmann machines. From the viewpoint of kernel machines, it includes least squares support vector machines for classification and regression, kernel principal component analysis (PCA), matrix singular value decomposition, and Parzen-type models. A key element is to first characterize these kernel machines in terms of so-called conjugate feature duality, yielding a representation with visible and hidden units. It is shown how this is related to the energy form in restricted Boltzmann machines, with continuous variables in a nonprobabilistic setting. In this new framework of so-called restricted kernel machine (RKM) representations, the dual variables correspond to hidden features. Deep RKM are obtained by coupling the RKMs. The method is illustrated for deep RKM, consisting of three levels with a least squares support vector machine regression level and two kernel PCA levels. In its primal form also deep feedforward neural networks can be trained within this framework.