A Theory of Learned Representations in Artificial and Natural Neural Networks

深度学习在实践中是成功的，因为通过大量的数据和计算，学习到了有用的一般表示，使复杂任务的执行成为可能。这种表征学习是深度学习最重要但最不被了解的方面之一--目前没有表征质量的量化衡量标准，也没有办法证明方法达到了预期的质量。这个项目致力于获得这样的测量方法，并使用经验和理论方法来获得经过认证的表征学习算法，以及将这些算法与人类和动物大脑中的表征学习联系起来。这样的理解对于获得可用于各种应用和任务的健壮的通用算法至关重要。这个项目将在机器学习、信号处理、统计学和计算神经科学之间建立新的联系。它还将为表示学习带来更强有力的统计保证，使其建立在更坚实的数学基础上。随着深度学习被用于越来越重要的决策，像这里所追求的那样的严格保证变得越来越重要。该项目的成果还将用于K-12、大学和研究生水平的教育工作，包括针对计算机历史上代表性不足的群体的项目。该项目结合了机器学习、统计学、信号处理和计算神经科学的见解，以获得人工神经网络和自然神经网络的表示理论。具体地说，该项目的目的是制定依赖任务和独立于任务的代表性质量衡量标准。任务相关的测量通过其在下游任务中的表现来捕捉表征的质量，而任务无关的测量则根据表征的内在属性和输入分布来定义质量。该项目将获得这两种类型之间的关系，从而表征表示学习算法转移的条件。该项目还将在假设的情况下对表示质量产生严格的限制。通过对表示质量的研究，该项目将致力于解释现实世界中自然和人工神经网络的普遍特征。这些特征包括：视觉和听觉系统中神经反应的参数空间的局部性，对不同类型的信号做出反应的神经元的混合选择性(例如，小鼠的嗅觉神经元对空间和气味的变化都做出反应)，以及在不同类型的信号中对同一概念做出反应的跨模式神经元(例如，大脑中的视觉和听觉信号)。该项目还将把表示学习与信号处理和学习中的经典概念(如词典学习)以及深度学习中众所周知的开放问题联系起来，包括分层表示的盛行、过度参数模型的泛化和简单性偏见。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning is successful in practice because through large amounts of data and computation, useful general representations are learned that enable performance of complex tasks. Such representation learning is one of the most important and least understood aspects of deep learning — there are currently no quantitative measures for quality of representations, nor ways to certify that methods achieve the desired quality. This project is concerned with obtaining such measures and using both empirical and theoretical approaches to obtain certified representation-learning algorithms, as well as connecting these to representation learning in human and animal brains. Such an understanding is crucial for obtaining robust general algorithms that can be used for a wide variety of applications and tasks. This project will form new connections between machine learning, signal processing, statistics, and computational neuroscience. It will also result in stronger statistical guarantees for representation learning, placing it on a firmer mathematical foundation. As deep learning is used for increasingly consequential decisions, rigorous guarantees such as the ones pursued here become ever more important. Results of the project will also be used in education efforts at the K-12, college, and graduate level, including in programs aimed at groups historically under-represented in computing.This project combines insights from machine learning, statistics, signal processing, and computational neuroscience to obtain a theory of representations in both artificial and natural neural networks. Specifically, the project aims to develop both task-dependent and task-independent measures of representation quality. Task-dependent measures capture the quality of representation through its performance in down-stream tasks, while task-independent measures define quality in terms of intrinsic properties of the representation and input distribution. The project will obtain relations between the two types and hence characterize conditions under which representation-learning algorithms transfer. The project will also result in rigorous bounds on representation quality under assumptions. Through the study of representation quality, the project will aim to explain prevalent features in real-world natural and artificial neural networks. These features include: locality in parameter space of neural responses in the visual and auditory system, mixed-selectivity of neurons that respond to signals of different types (for example, olfactory neurons in mice that respond to both spatial and odorant changes), and cross-modal neurons that respond to the same concept in signals of different types (for example, visual and auditory signals in the brain). The project will also connect representation learning to classical notions in signal processing and learning such as dictionary learning, as well as to well-known open questions in deep learning, including the prevalence of hierarchical representations, generalization of over-parameterized models, and simplicity bias.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.