CAREER: Towards theoretical foundations of neural network based representation learning

构建输入数据的良好表示一直是机器学习的核心要素。例如，在计算机视觉中，人们希望具有表示图像中主要对象的特征。在自然语言处理中，人们希望具有指示不同单词之间关系的特征。基于深度学习技术的机器学习的新范式转变已经证明了机器在没有任何先验知识的情况下从训练数据集中自动学习良好表示的能力。然而，尽管这些特征对机器学习数据集真的很有用，但按照人类的标准，它们真的“好”吗？该项目旨在促进对深度表示学习的基本理解，并为深度学习的实际进展提供信息，提高其在大数据体系中的可解释性、鲁棒性和效率。研究者还将通过这个项目的课程开发一个新的研究生水平课程和一个公共互动软件。该项目旨在为新一代基于神经网络的表示学习技术建立一个全面的理论。这包括表征表征的统计特性以及它们如何在实际的神经网络中编码的核心问题。这个项目有三个主要组成部分。第一个重点是描述什么时候最小化表征学习任务的训练目标会导致神经网络中唯一的表征：利用新的理论发展，研究者将建立新的训练目标来鼓励这种独特性。第二个重点是从理论上研究哪些表征可以通过深度学习模型有效地学习，以及如何在训练后将它们编码到神经网络的隐藏权重中。最后，研究者将研究学习表征的统计特性使它们适合下游任务，这对于提高这些基于神经网络的表征的可解释性至关重要。此外，它将使人类能够更好地与深度学习模型进行交互，以用于自动驾驶汽车等更广泛的应用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Building up good representations of input data has always been a central ingredient in machine learning. In computer vision, for example, one would like to have features representing the main objects in the image. In natural language processing, one would like to have features indicating the relationship between different words. A new paradigm shift in machine learning based on deep learning techniques has demonstrated the ability of machines to automatically learn good representations from the training data set without any prior knowledge. However, although these features are really useful for machines to learn the data set, are they actually "good" according to human standards? The project aims to contribute to the fundamental understanding of deep representation learning and inform the practical advancement of deep learning, improving its interpretability, robustness, and efficiency in large data regimes. The investigator will also develop a new graduate-level course and a public interactive software through the course of this project. The project aims to build a comprehensive theory for the new generation of neural-network-based representation-learning techniques. This includes the central questions of characterizing the statistical properties of the representations and how they are encoded in an actual neural network. This project has three major components. The first thrust is to characterize when would minimizing the training objective of the representation learning task leads to a unique representation in the neural network: leveraging the new theoretical development, the investigator will build up new training objectives that encourage such uniqueness. The second thrust is to theoretically study what representations can be efficiently learned by deep-learning models, and how are they encoded in the hidden weights of the neural networks after training. Finally, the investigator will study what statistical properties of the learned representations made them good for downstream tasks, which is critical to improving the interpretability of these neural network-based representations. Moreover, it will allow humans to better interact with deep-learning models for broader applications such as self-driving cars.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.