From lazy to rich to exclusive task representations in neural networks and neural codes.
From lazy to rich to exclusive task representations in neural networks and neural codes.
复制标题
从懒惰到丰富，再到神经网络和神经代码中的专有任务表示。
DOI：
10.1016/j.conb.2023.102780
复制
发表时间：
2023
影响因子：
5.7
通讯作者：
Shea-Brown,Eric
中科院分区：
文献类型：
--
作者：
Farrell,Matthew;Recanatesi,Stefano;Shea-Brown,Eric
神经回路——无论是在大脑中还是在“人工”神经网络模型中——都在学习解决各种各样的任务，目前有很大的机会使用神经网络作为大脑功能的模型。这一努力的关键是表征人工和生物大脑形成的表征的能力。在这里，我们通过最近发展的理论来研究这种潜力，该理论将神经网络的特征描述为“懒惰”或“丰富”，这取决于它们用于解决任务的方法：懒惰网络通过对连通性进行微小改变来解决任务，而丰富网络通过显著修改整个网络（包括“隐藏层”）的权重来解决任务。我们通过压缩和“神经崩溃”的镜头进一步阐明了丰富的网络，这些想法最近引起了神经科学和机器学习的极大兴趣。然后，我们展示了这些想法如何应用于对这两个领域都越来越重要的领域：通过自监督学习提取潜在结构。
Neural circuits—both in the brain and in “artificial” neural network models—learn to solve a remarkable variety of tasks, and there is a great current opportunity to use neural networks as models for brain function. Key to this endeavor is the ability to characterize therepresentationsformed by both artificial and biological brains. Here, we investigate this potential through the lens of recently developing theory that characterizes neural networks as “lazy” or “rich” depending on the approach they use to solve tasks: lazy networks solve tasks by making small changes in connectivity, while rich networks solve tasks by significantly modifying weights throughout the network (including “hidden layers”). We further elucidate rich networks through the lens of compression and “neural collapse”, ideas that have recently been of significant interest to neuroscience and machine learning. We then show how these ideas apply to a domain of increasing importance to both fields: extracting latent structures through self-supervised learning.