A sparse-response deep belief network based on rate distortion theory
A sparse-response deep belief network based on rate distortion theory
复制标题
基于率失真理论的稀疏响应深度置信网络
DOI：
10.1016/j.patcog.2014.03.025
复制
发表时间：
2014-09-01
影响因子：
8
通讯作者：
Zhang, Chun-Xia
中科院分区：
文献类型：
--
作者：
Ji, Nan-Nan;Zhang, Jiang-She;Zhang, Chun-Xia
深度信念网络（DBN）是目前用于建模大脑结构深度的主导技术，并且可以以贪婪的逐层无监督学习方式有效地训练。然而，没有狭窄的隐藏瓶颈的DBN通常会产生冗余的连续值代码和非结构化的权重模式。从率失真（RD）理论，使用尽可能少的比特编码的原始数据的启发，我们在本文中介绍了DBN的变体，称为稀疏响应DBN（SR-DBN）。该方法将数据分布与DBN构建块定义的平衡分布之间的Kullback-Leibler偏差视为失真函数，并使用代码L-1范数诱导的稀疏响应正则化来实现小码率。通过从不同尺度的图像数据集提取特征的实验表明，我们的方法SR-DBN学习代码的速度很小，提取特征的多个层次的抽象模仿计算的皮层层次，并获得更多的歧视性表示比PCA和DBN的几个基本算法。(C)2014爱思唯尔有限公司版权所有。
Deep belief networks (DBNs) are currently the dominant technique for modeling the architectural depth of brain, and can be trained efficiently in a greedy layer-wise unsupervised learning manner. However, DBNs without a narrow hidden bottleneck typically produce redundant, continuous-valued codes and unstructured weight patterns. Taking inspiration from rate distortion (RD) theory, which encodes original data using as few bits as possible, we introduce in this paper a variant of DBN, referred to as sparse-response DBN (SR-DBN). In this approach, Kullback-Leibler divergence between the distribution of data and the equilibrium distribution defined by the building block of DBN is considered as a distortion function, and the sparse response regularization induced by L-1-norm of codes is used to achieve a small code rate. Several experiments by extracting features from different scale image datasets show that our approach SR-DBN learns codes with small rate, extracts features at multiple levels of abstraction mimicking computations in the cortical hierarchy, and obtains more discriminative representation than PCA and several basic algorithms of DBNs. (C) 2014 Elsevier Ltd. All rights reserved.