Building transformers from neurons and astrocytes.
Building transformers from neurons and astrocytes.
复制标题
DOI：
10.1073/pnas.2219150120
复制
发表时间：
2023-08-22
影响因子：
11.1
通讯作者：
Krotov, Dmitry
中科院分区：
文献类型：
--
作者：
Kozachkov, Leo;Kastanenka, Ksenia V.;Krotov, Dmitry
变压器已经成为许多机器学习应用程序的神经架构的默认选择。他们在语言、视觉和言语等多个领域的成功提出了一个问题：如何使用生物计算单元构建变形金刚？与此同时，在神经胶质细胞群体中，逐渐积累的证据表明，以前被认为是大脑中被动管家细胞的星形胶质细胞实际上在大脑的信息处理和计算中发挥着重要作用。在这项工作中，我们假设神经元-星形胶质细胞网络可以自然地实现AI中Transformer块执行的核心计算。星形胶质细胞在几乎任何大脑区域的无处不在可以解释变形金刚在各种信息领域和计算任务中的成功。神经胶质细胞占所有人脑细胞的50%至90%，并提供各种重要的发育，结构和代谢功能。最近的实验结果表明，星形胶质细胞，一种神经胶质细胞，也直接参与核心认知过程，如学习和记忆。虽然已经确定星形胶质细胞和神经元在许多时间尺度和空间尺度的反馈回路中相互连接，但在理解神经元-星形胶质细胞相互作用的计算作用方面存在差距。为了帮助弥合这一差距，我们利用人工智能和星形胶质细胞成像技术的最新进展。特别是，我们证明了神经元-星形胶质细胞网络可以自然地执行Transformer的核心计算，这是一种特别成功的AI架构类型。在这样做的过程中，我们提供了一个具体的，规范的，和实验测试的帐户神经元星形胶质细胞通信。由于变形金刚在语言、视觉和听觉等各种任务领域都非常成功，我们的分析可能有助于解释大脑神经元-星形胶质细胞网络的普遍性、灵活性和力量。
Transformers have become the default choice of neural architecture for many machine learning applications. Their success across multiple domains such as language, vision, and speech raises the question: How can one build Transformers using biological computational units? At the same time, in the glial community, there is gradually accumulating evidence that astrocytes, formerly believed to be passive house-keeping cells in the brain, in fact play an important role in the brain’s information processing and computation. In this work we hypothesize that neuron–astrocyte networks can naturally implement the core computation performed by the Transformer block in AI. The omnipresence of astrocytes in almost any brain area may explain the success of Transformers across a diverse set of information domains and computational tasks. Glial cells account for between 50% and 90% of all human brain cells, and serve a variety of important developmental, structural, and metabolic functions. Recent experimental efforts suggest that astrocytes, a type of glial cell, are also directly involved in core cognitive processes such as learning and memory. While it is well established that astrocytes and neurons are connected to one another in feedback loops across many timescales and spatial scales, there is a gap in understanding the computational role of neuron–astrocyte interactions. To help bridge this gap, we draw on recent advances in AI and astrocyte imaging technology. In particular, we show that neuron–astrocyte networks can naturally perform the core computation of a Transformer, a particularly successful type of AI architecture. In doing so, we provide a concrete, normative, and experimentally testable account of neuron–astrocyte communication. Because Transformers are so successful across a wide variety of task domains, such as language, vision, and audition, our analysis may help explain the ubiquity, flexibility, and power of the brain’s neuron–astrocyte networks.
登录
查看更多内容
影响因子：
4.3
作者：
通讯作者：
--
DOI：
10.1073/pnas.2207912119
发表时间：
2022-10-25
影响因子：
11.1
作者：
通讯作者：
--
影响因子：
3.2
作者：
Erickson MA;Maramara LA;Lisman J
通讯作者：
Lisman J
影响因子：
7.8
作者：
Chance, FS;Abbott, LF
通讯作者：
Abbott, LF
影响因子：
5.9
作者：
Caucheteux C;King JR
通讯作者：
King JR