A mathematical theory of semantic development in deep neural networks
A mathematical theory of semantic development in deep neural networks
复制标题
DOI：
10.1073/pnas.1820226116
复制
发表时间：
2019-06-04
影响因子：
11.1
通讯作者：
Ganguli, Surya
中科院分区：
文献类型：
--
作者：
Saxe, Andrew M.;McClelland, James L.;Ganguli, Surya
大量的实证研究揭示了人类语义知识的获取、组织、部署和神经表征的显著差异，从而提出了一个基本的概念性问题：通过整合许多个体经验，管理神经网络获取、组织和部署抽象知识的能力的理论原则是什么？我们通过数学分析深度线性网络中学习的非线性动力学来解决这个问题。我们找到了这种学习动态的确切解决方案，这些解决方案为语义认知中许多不同现象的流行提供了概念解释，包括通过快速发展过渡的概念层次分化，这种过渡之间语义错觉的普遍存在，项目典型性和类别连贯性的出现作为控制语义处理速度的因素，在发展过程中不断变化的归纳投射模式，以及在不同物种的神经表征中保持语义相似性。因此，令人惊讶的是，我们的简单神经模型定性地概括了许多不同的语义发展基础，同时提供了对环境的统计结构如何与非线性深度学习动态相互作用以产生这些语义的分析见解。
An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: What are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep-learning dynamics to give rise to these regularities.