EAGER: Collaborative Research: World Modeling for Natural Language Understanding

EAGER：协作研究：自然语言理解的世界建模

• 批准号: 1941160
• 负责人: Allyson Ettinger
• 金额: $ 2.39万
• 依托单位: University of Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2021-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1941160&HistoricalAwards=false
• 关键词: EAGER; Collaborative; Research; World; Modeling

A key goal of artificial intelligence (AI) is to build systems that can read and understand language as humans do. This capability underlies a broad range of technologies, including question answering, machine translation, and dialogue systems. While progress has been made, AI systems currently lack the robustness and flexibility of human language understanding---typical systems leverage shallow pattern-matching strategies to perform tasks, and as a result are only effective at the specific tasks they are built for, and fail easily even within those settings. This project addresses these issues by improving the ability of systems to construct rich representations of the "world" described in text: Who are the entities involved, and what are their attributes and relationships? What events are taking place, who is participating in those events, and why are they occurring? The design of the systems' notion of a world uses concepts like these that have been identified by cognitive scientists and psychologists as fundamental in human language understanding. The expected benefit of this work is the development of AI systems that can use language flexibly and robustly because, like humans, these systems will perform tasks based on the core information conveyed in language, rather than superficial pattern-matching. In addition to improving systems, this project will have the benefit of building bridges between the AI community and cognitive scientists, psychologists, and linguists---the project's modeling framework provides a pathway through which insights from cognitive science can be translated to model implementation, which can be utilized both for improvement of AI systems and for testing of cognitive hypotheses. This exploratory EAGER project improves the capacity of systems to automatically construct the world underlying the text being analyzed, and designs targeted probing tasks to enable fine-grained assessment of the extent to which systems have captured this information. The modeling framework uses memory-augmented neural networks, leveraging the external memory components to represent worlds. Rather than explicit annotation, the project implements cognitively-inspired design of both world components themselves and inductive bias for encouraging particular components to capture what is intended. Learning is carried out via self-supervised objectives and auxiliary supervision on large datasets of narratives. System evaluation consists of both standard reading comprehension question answering tasks and the development of novel probing tasks. The use of controlled probing tasks draws critically from methodological approaches used in cognitive neuroscience and psycholinguistics, applying these scientific methods for interpretation of artificial systems. These probing tasks allow for targeted analysis of individual world components and provide guidance for model improvement. The methodology of the project iterates between model design and targeted testing via probing tasks, using the results of the latter to guide the former.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能（AI）的一个关键目标是构建能够像人类一样阅读和理解语言的系统。这种能力是一系列技术的基础，包括问答、机器翻译和对话系统。虽然已经取得了进展，但人工智能系统目前缺乏人类语言理解的鲁棒性和灵活性-典型的系统利用浅层模式匹配策略来执行任务，因此只能在特定的任务中有效，即使在这些设置中也很容易失败。该项目通过提高系统构建文本中描述的“世界”的丰富表示的能力来解决这些问题：涉及的实体是谁，它们的属性和关系是什么？正在发生什么事件，谁参与了这些事件，为什么会发生？系统的世界概念的设计使用了认知科学家和心理学家认为是人类语言理解的基础的概念。这项工作的预期好处是开发出能够灵活而强大地使用语言的人工智能系统，因为像人类一样，这些系统将根据语言中传达的核心信息执行任务，而不是表面的模式匹配。除了改进系统外，该项目还将在人工智能社区与认知科学家，心理学家和语言学家之间建立桥梁-该项目的建模框架提供了一条途径，通过该途径可以将认知科学的见解转化为模型实现，这既可以用于改进人工智能系统，也可以用于测试认知假设。这个探索性的EAGER项目提高了系统自动构建被分析文本背后的世界的能力，并设计了有针对性的探测任务，以实现对系统捕获这些信息的程度的细粒度评估。建模框架使用内存增强神经网络，利用外部内存组件来表示世界。该项目没有明确的注释，而是实现了世界组件本身的认知启发设计和诱导偏见，以鼓励特定组件捕获预期内容。学习是通过自我监督的目标和辅助监督大型数据集的叙述。系统评估包括标准的阅读理解问题回答任务和新的探索任务的开发。控制探测任务的使用严格借鉴了认知神经科学和心理语言学中使用的方法论，将这些科学方法应用于解释人工系统。这些探测任务允许对各个世界组件进行有针对性的分析，并为模型改进提供指导。该项目的方法通过探测任务在模型设计和目标测试之间迭代，使用后者的结果来指导前者。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估而被认为值得支持。

项目成果

Sorting through the noise: Testing robustness of information processing in pre-trained language models

对噪音进行排序：测试预训练语言模型中信息处理的鲁棒性

• DOI: 10.18653/v1/2021.emnlp-main.119
• 发表时间: 2021
• 期刊: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
• 作者: Pandia, L.;Ettinger, A.
• 通讯作者: Ettinger, A.

Assessing Phrasal Representation and Composition in Transformers

• DOI: 10.18653/v1/2020.emnlp-main.397
• 发表时间: 2020-10
• 作者: Lang-Chi Yu;Allyson Ettinger

Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words

• DOI: 10.18653/v1/2020.acl-main.434
• 发表时间: 2020-05
• 期刊: Urban Rail Transit
• 影响因子: 1.5
• 作者: Josef Klafka;Allyson Ettinger

Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks

• DOI: 10.18653/v1/2020.emnlp-main.685
• 发表时间: 2020-10
• 期刊: ArXiv
• 作者: Shubham Toshniwal;Sam Wiseman;Allyson Ettinger;Karen Livescu;Kevin Gimpel

Pragmatic competence of pre-trained language models through the lens of discourse connectives

通过话语连接词的视角预训练语言模型的语用能力

• DOI: 10.18653/v1/2021.conll-1.29
• 发表时间: 2021
• 期刊: Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL
• 作者: Pandia, L.;Cong, Y.;Ettinger, A.
• 通讯作者: Ettinger, A.