EAGER: Exploring Cognitively Plausible Computational Models for Processing Human Language

EAGER：探索处理人类语言的认知合理计算模型

• 批准号: 1844740
• 负责人: Anna Rumshisky
• 金额: $ 10.99万
• 依托单位: University of Massachusetts Lowell
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2022-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1844740&HistoricalAwards=false
• 关键词: EAGER; Exploring; Cognitively; Plausible; Computational

Despite the recent successes of artificial intelligence techniques designed to process human language, most contemporary solutions are designed to handle very specific language processing tasks. As a result, human-level language understanding is still out of reach for most current computational approaches, especially when retaining new information and reasoning over the accumulated knowledge is involved. This exploratory project advances the goal of developing more cognitively realistic computational models that can mimic some of the known properties of human language processing, and as a result, be more robust and better suited as general systems for language understanding, with human-like learning which involves obtaining and updating knowledge over time.While most contemporary deep learning approaches in natural language processing focus on task-specific end-to-end models, this project prioritizes generalist architectures that would be consistent with the current data on semantic priming, grouping and chunking effects in the formation and use of conceptual systems, and the effects of long- and short-term memory on the storage and retrieval of knowledge. In this project, novel neural network architectures are planned that model a subset of these properties. The processes that enable learning and memory via strengthening of synaptic connections in the brain will be emulated by a set of representational units (r-units) with bidirectional connections, modeling the interaction between small regions of neocortex during information processing. Memory Store Activation State Model represents the connections between r-units in terms of convolutional filters applied to the memory store. The priming effects will be modeled by a pre-activation pattern produced via a sequence of deconvolutional operation. Rate-Based Connectivity Network model combines reinforcement learning on per-node basis with a form of Hebbian learning applied to a time-varying system where each r-unit calculates rate of change of its output, allowing node activations to linger through time; it is trained with a discrete global reward signal. The goal of this project is to establish the feasibility of the proposed architectures by developing the initial proof-of-concept prototypes, demonstrating that they are able to converge on simple learning tasks, and applying them to the task of language modeling to ensure that a practically useful representation can be learned.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管最近人工智能技术在处理人类语言方面取得了成功，但大多数当代解决方案都是为了处理非常具体的语言处理任务而设计的。因此，对于大多数当前的计算方法来说，人类水平的语言理解仍然是遥不可及的，特别是当涉及到保留新信息和对积累的知识进行推理时。这个探索性项目推进了开发更多认知现实计算模型的目标，这些模型可以模仿人类语言处理的一些已知属性，因此，更健壮，更适合作为语言理解的一般系统，具有类似人类的学习，包括随着时间的推移获取和更新知识。虽然自然语言处理中大多数当代深度学习方法都侧重于任务特定的端到端模型，但该项目优先考虑的是通才架构，它将与当前关于语义启动、概念系统形成和使用中的分组和分块效应以及长期和短期记忆对知识存储和检索的影响的数据一致。在这个项目中，新的神经网络架构计划对这些属性的子集进行建模。通过加强大脑中的突触连接来实现学习和记忆的过程将由一组具有双向连接的表征单元（r-units）来模拟，模拟信息处理过程中新皮层小区域之间的相互作用。内存存储激活状态模型（Memory Store Activation State Model）以应用于内存存储的卷积滤波器的形式表示r-单元之间的连接。启动效应将通过一系列反卷积操作产生的预激活模式来建模。基于速率的连通性网络模型将基于每个节点的强化学习与应用于时变系统的Hebbian学习形式相结合，其中每个r-unit计算其输出的变化率，允许节点激活随时间而持续；它是用离散的全局奖励信号来训练的。这个项目的目标是通过开发最初的概念验证原型来建立所提出的架构的可行性，证明它们能够收敛于简单的学习任务，并将它们应用于语言建模任务，以确保可以学习到实际有用的表示。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Revealing the Dark Secrets of BERT

• DOI: 10.18653/v1/d19-1445
• 发表时间: 2019-08
• 期刊: ArXiv
• 作者: Olga Kovaleva;Alexey Romanov;Anna Rogers;Anna Rumshisky

Life after BERT: What do Other Muppets Understand about Language?

• DOI: 10.18653/v1/2022.acl-long.227
• 发表时间: 2022-05
• 作者: Vladislav Lialin;Kevin Zhao;Namrata Shivagunde;Anna Rumshisky

When BERT Plays the Lottery, All Tickets Are Winning

• DOI: 10.18653/v1/2020.emnlp-main.259
• 发表时间: 2020-05
• 作者: Sai Prasanna;Anna Rogers;Anna Rumshisky

BERT Busters: Outlier Dimensions that Disrupt Transformers

• DOI: 10.18653/v1/2021.findings-acl.300
• 发表时间: 2021-05
• 作者: Olga Kovaleva;Saurabh Kulshreshtha;Anna Rogers;Anna Rumshisky