A fast learning algorithm for deep belief nets
A fast learning algorithm for deep belief nets
复制标题
DOI：
10.1162/neco.2006.18.7.1527
复制
发表时间：
2006-07-01
影响因子：
2.9
通讯作者：
Teh, Yee-Whye
中科院分区：
文献类型：
--
作者：
Hinton, Geoffrey E.;Osindero, Simon;Teh, Yee-Whye
我们展示了如何使用“互补先验”来消除解释效应，这种解释效应使得在具有许多隐藏层的密集连接的信念网中难以进行推理。使用互补先验，我们得到了一个快速，贪婪的算法，可以一次一层地学习深度，有向的信念网络，前提是顶部的两层形成无向联想记忆。快速贪婪算法用于初始化较慢的学习过程，该过程使用唤醒-睡眠算法的对比版本来微调权重。经过微调，一个具有三个隐藏层的网络形成了一个很好的手写数字图像及其标签的联合分布的生成模型。这种生成模型比最好的判别式学习算法给出了更好的数字分类。数字所在的低维流形是由顶层联想记忆的自由能景观中的长峡谷模拟的，通过使用有向连接来显示联想记忆的想法，很容易探索这些峡谷。
We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.