Unsupervised learning for perception

通常用于训练人工神经网络的反向传播学习过程对于在输入和输出之间具有多层非线性神经元的深度网络并不适用。 我的团队最近引入了一种新方法，使用一种有效的无监督算法，一次一层地对深度网络进行“预训练”，该算法试图对输入数据中的错误进行建模。在预训练之后，深度网络的中间层已经包含了许多有用的特征检测器，反向传播在微调预训练的深度网络方面比从头开始学习深度网络要好得多。 在最近的一篇科学论文中，我们表明，这种无监督预训练与监督微调的组合在一个经过充分研究的辨别任务和非线性降维方面优于所有其他学习方法。使用这种新方法，我们计划将深度神经网络应用于各种重要的辨别任务，包括物体和人脸识别。我们还将使用深度网络来模拟人类运动、语音和视频序列。 使用深度网络从大型高维数据集中提取低维代码，我们将改进基于Web的推荐系统。我们还将开发深度网络，可以将文档或图像转换为二进制地址，使得语义相似的文档具有相似的二进制地址。这允许对类似文档或图像进行极快的搜索。最后，我们将研究这些新的学习算法可能在皮层中实现的方法。

The backpropagation learning procedure that is often used to train artificial neural networks does notwork well for deep networks that have many layers of non-linear neurons between the input and theoutput.  My group has recently introduced a new approach in which deep networks are "pretrained" onelayer at a time using an efficient unsupervised algorithm that tries to model regularities in theinput data. After pretraining, the intermediate layers of a deep network already contain many usefulfeature detectors and backpropagation is much better at fine-tuning a pretrained deep network than itis at learning a deep network from scratch.  In a recent Science paper we showed that this combinationof unsupervised pretraining followed by supervised fine-tuning outperforms all other learning methodsat a well-studied discrimination task and also for non-linear dimensionality reduction.Using this new approach, we plan to apply deep neural networks to a variety of important discrimintiontasks including object and face recognition. We will also use deep networks to model human motion,speech, and video sequences.  Using deep networks to extract low-dimensional codes from large, highdimensional datasets, we will improve web-based recommendation systems. We will also develop deepnetworks that can convert documents or images into binary addresses in such a way that semanticallysimilar documents have similar binary addresses. This allows extremely fast search for similardocuments or images. Finally, we will investigate ways in which these new learning algorithms might beimplemented in the cortex.