A Theory of Learned Representations in Artificial and Natural Neural Networks

人工和自然神经网络中的学习表示理论

• 批准号: 2134157
• 负责人: Cengiz Pehlevan
• 金额: $ 110万
• 依托单位: Harvard University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134157&HistoricalAwards=false
• 关键词: Theory; Learned; Representations; Artificial; Natural

Deep learning is successful in practice because through large amounts of data and computation, useful general representations are learned that enable performance of complex tasks. Such representation learning is one of the most important and least understood aspects of deep learning — there are currently no quantitative measures for quality of representations, nor ways to certify that methods achieve the desired quality. This project is concerned with obtaining such measures and using both empirical and theoretical approaches to obtain certified representation-learning algorithms, as well as connecting these to representation learning in human and animal brains. Such an understanding is crucial for obtaining robust general algorithms that can be used for a wide variety of applications and tasks. This project will form new connections between machine learning, signal processing, statistics, and computational neuroscience. It will also result in stronger statistical guarantees for representation learning, placing it on a firmer mathematical foundation. As deep learning is used for increasingly consequential decisions, rigorous guarantees such as the ones pursued here become ever more important. Results of the project will also be used in education efforts at the K-12, college, and graduate level, including in programs aimed at groups historically under-represented in computing.This project combines insights from machine learning, statistics, signal processing, and computational neuroscience to obtain a theory of representations in both artificial and natural neural networks. Specifically, the project aims to develop both task-dependent and task-independent measures of representation quality. Task-dependent measures capture the quality of representation through its performance in down-stream tasks, while task-independent measures define quality in terms of intrinsic properties of the representation and input distribution. The project will obtain relations between the two types and hence characterize conditions under which representation-learning algorithms transfer. The project will also result in rigorous bounds on representation quality under assumptions. Through the study of representation quality, the project will aim to explain prevalent features in real-world natural and artificial neural networks. These features include: locality in parameter space of neural responses in the visual and auditory system, mixed-selectivity of neurons that respond to signals of different types (for example, olfactory neurons in mice that respond to both spatial and odorant changes), and cross-modal neurons that respond to the same concept in signals of different types (for example, visual and auditory signals in the brain). The project will also connect representation learning to classical notions in signal processing and learning such as dictionary learning, as well as to well-known open questions in deep learning, including the prevalence of hierarchical representations, generalization of over-parameterized models, and simplicity bias.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习在实践中是成功的，因为通过大量的数据和计算，可以学习到有用的一般表示，从而能够执行复杂的任务。这种表示学习是深度学习中最重要也是最不为人所知的方面之一--目前还没有量化的方法来衡量表示的质量，也没有方法来证明方法达到了预期的质量。该项目关注的是获得这些措施，并使用经验和理论方法来获得认证的表征学习算法，以及将这些连接到人类和动物大脑中的表征学习。这种理解对于获得可用于各种应用和任务的鲁棒通用算法至关重要。该项目将在机器学习、信号处理、统计学和计算神经科学之间形成新的联系。它还将为表征学习提供更强的统计保证，使其建立在更坚实的数学基础上。随着深度学习被用于越来越重要的决策，这里所追求的严格保证变得越来越重要。该项目的成果还将用于K-12、大学和研究生层面的教育工作，包括针对历史上在计算方面代表性不足的群体的计划。该项目结合了机器学习、统计学、信号处理和计算神经科学的见解，以获得人工和自然神经网络的表示理论。具体而言，该项目旨在开发任务依赖和任务独立的表征质量的措施。任务相关的措施捕捉质量的表示通过其性能在下游任务，而任务无关的措施定义质量的表示和输入分布的内在属性。该项目将获得这两种类型之间的关系，从而表征条件下，表示学习算法转移。该项目还将导致在假设下的代表性质量的严格界限。通过对表示质量的研究，该项目旨在解释现实世界中自然和人工神经网络的普遍特征。这些功能包括：视觉和听觉系统中神经响应的参数空间中的局部性、响应于不同类型的信号的神经元的混合选择性（例如，响应于空间和气味变化的小鼠中的嗅觉神经元）以及响应于不同类型的信号中的相同概念的交叉模态神经元（例如，大脑中的视觉和听觉信号）。该项目还将把表示学习与信号处理和学习中的经典概念（如字典学习）以及深度学习中众所周知的开放性问题联系起来，包括分层表示的流行，过度参数化模型的泛化，该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准。