EAGER: Collaborative Research: World Modeling for Natural Language Understanding

EAGER：协作研究：自然语言理解的世界建模

• 批准号: 1941178
• 负责人: Kevin Gimpel
• 金额: $ 22.61万
• 依托单位: Toyota Technological Institute at Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2021-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1941178&HistoricalAwards=false
• 关键词: EAGER; Collaborative; Research; World; Modeling

A key goal of artificial intelligence (AI) is to build systems that can read and understand language as humans do. This capability underlies a broad range of technologies, including question answering, machine translation, and dialogue systems. While progress has been made, AI systems currently lack the robustness and flexibility of human language understanding---typical systems leverage shallow pattern-matching strategies to perform tasks, and as a result are only effective at the specific tasks they are built for, and fail easily even within those settings. This project addresses these issues by improving the ability of systems to construct rich representations of the "world" described in text: Who are the entities involved, and what are their attributes and relationships? What events are taking place, who is participating in those events, and why are they occurring? The design of the systems' notion of a world uses concepts like these that have been identified by cognitive scientists and psychologists as fundamental in human language understanding. The expected benefit of this work is the development of AI systems that can use language flexibly and robustly because, like humans, these systems will perform tasks based on the core information conveyed in language, rather than superficial pattern-matching. In addition to improving systems, this project will have the benefit of building bridges between the AI community and cognitive scientists, psychologists, and linguists---the project's modeling framework provides a pathway through which insights from cognitive science can be translated to model implementation, which can be utilized both for improvement of AI systems and for testing of cognitive hypotheses. This exploratory EAGER project improves the capacity of systems to automatically construct the world underlying the text being analyzed, and designs targeted probing tasks to enable fine-grained assessment of the extent to which systems have captured this information. The modeling framework uses memory-augmented neural networks, leveraging the external memory components to represent worlds. Rather than explicit annotation, the project implements cognitively-inspired design of both world components themselves and inductive bias for encouraging particular components to capture what is intended. Learning is carried out via self-supervised objectives and auxiliary supervision on large datasets of narratives. System evaluation consists of both standard reading comprehension question answering tasks and the development of novel probing tasks. The use of controlled probing tasks draws critically from methodological approaches used in cognitive neuroscience and psycholinguistics, applying these scientific methods for interpretation of artificial systems. These probing tasks allow for targeted analysis of individual world components and provide guidance for model improvement. The methodology of the project iterates between model design and targeted testing via probing tasks, using the results of the latter to guide the former.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能 (AI) 的一个关键目标是构建能够像人类一样阅读和理解语言的系统。这种能力是一系列技术的基础，包括问答、机器翻译和对话系统。尽管取得了进步，但人工智能系统目前缺乏人类语言理解的稳健性和灵活性——典型的系统利用浅层模式匹配策略来执行任务，因此仅对它们所构建的特定任务有效，即使在这些设置中也很容易失败。该项目通过提高系统构建文本中描述的“世界”的丰富表示的能力来解决这些问题：涉及的实体是谁，它们的属性和关系是什么？正在发生哪些事件，谁参与了这些事件，以及为什么发生这些事件？系统世界概念的设计使用了诸如此类的概念，这些概念已被认知科学家和心理学家认为是人类语言理解的基础。这项工作的预期好处是开发能够灵活、稳健地使用语言的人工智能系统，因为像人类一样，这些系统将根据语言传达的核心信息来执行任务，而不是表面的模式匹配。除了改进系统之外，该项目还将有利于在人工智能社区与认知科学家、心理学家和语言学家之间架起桥梁——该项目的建模框架提供了一条途径，通过该途径，认知科学的见解可以转化为模型实施，这既可以用于改进人工智能系统，也可以用于测试认知假设。这个探索性的 EAGER 项目提高了系统自动构建所分析文本背后的世界的能力，并设计了有针对性的探测任务，以便能够对系统捕获此信息的程度进行细粒度评估。该建模框架使用记忆增强神经网络，利用外部记忆组件来表示世界。该项目不是明确的注释，而是对世界组件本身实施认知启发的设计，并采用归纳偏差来鼓励特定组件捕获预期内容。学习是通过自我监督目标和对大型叙述数据集的辅助监督来进行的。系统评估包括标准阅读理解问答任务和新颖的探究任务的开发。受控探测任务的使用严格借鉴了认知神经科学和心理语言学中使用的方法论，应用这些科学方法来解释人工系统。这些探测任务可以对各个世界组件进行有针对性的分析，并为模型改进提供指导。该项目的方法通过探测任务在模型设计和有针对性的测试之间进行迭代，利用后者的结果来指导前者。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks

• DOI: 10.18653/v1/2020.emnlp-main.685
• 发表时间: 2020-10
• 期刊: ArXiv
• 作者: Shubham Toshniwal;Sam Wiseman;Allyson Ettinger;Karen Livescu;Kevin Gimpel

Chess as a Testbed for Language Model State Tracking

国际象棋作为语言模型状态跟踪的测试平台

• DOI: 10.1609/aaai.v36i10.21390
• 发表时间: 2022
• 期刊: Proceedings of the AAAI Conference on Artificial Intelligence
• 作者: Toshniwal, Shubham;Wiseman, Sam;Livescu, Karen;Gimpel, Kevin
• 通讯作者: Gimpel, Kevin

Reconsidering the Past: Optimizing Hidden States in Language Models

• DOI: 10.18653/v1/2021.findings-emnlp.346
• 发表时间: 2021-12
• 作者: Davis Yoshida;Kevin Gimpel