Unsupervised learning for perception

通常用于训练人工神经网络的反向传播学习过程对于输入和输出之间具有多层非线性神经元的深层网络并不起作用。我的团队最近引入了一种新的方法，即使用一种高效的无监督算法来尝试对输入数据中的规则进行建模，每次预先训练深层网络的一层。经过预训练，深度网络的中间层已经包含了许多有用的特征检测器，而反向传播在微调预先训练的深度网络方面比从零开始学习深度网络要好得多。在最近的一篇科学论文中，我们证明了这种无监督预训练和监督微调的组合在经过充分研究的识别任务中以及在非线性降维方面的性能优于所有其他学习方法。使用这种新的方法，我们计划将深度神经网络应用于各种重要的识别任务，包括对象和人脸识别。我们还将使用深度网络对人体运动、语音和视频序列进行建模。除了使用深度网络从大型高维数据集中提取低维代码外，我们还将完善基于网络的推荐系统。我们还将开发深度网络，可以将文档或图像转换为二进制地址，从而使语义相似的文档具有相似的二进制地址。这可以极快地搜索相似的文档或图像。最后，我们将研究如何在大脑皮层实施这些新的学习算法。

The backpropagation learning procedure that is often used to train artificial neural networks does notwork well for deep networks that have many layers of non-linear neurons between the input and theoutput.  My group has recently introduced a new approach in which deep networks are "pretrained" onelayer at a time using an efficient unsupervised algorithm that tries to model regularities in theinput data. After pretraining, the intermediate layers of a deep network already contain many usefulfeature detectors and backpropagation is much better at fine-tuning a pretrained deep network than itis at learning a deep network from scratch.  In a recent Science paper we showed that this combinationof unsupervised pretraining followed by supervised fine-tuning outperforms all other learning methodsat a well-studied discrimination task and also for non-linear dimensionality reduction.Using this new approach, we plan to apply deep neural networks to a variety of important discrimintiontasks including object and face recognition. We will also use deep networks to model human motion,speech, and video sequences.  Using deep networks to extract low-dimensional codes from large, highdimensional datasets, we will improve web-based recommendation systems. We will also develop deepnetworks that can convert documents or images into binary addresses in such a way that semanticallysimilar documents have similar binary addresses. This allows extremely fast search for similardocuments or images. Finally, we will investigate ways in which these new learning algorithms might beimplemented in the cortex.