CAREER: Flexible and Robust Reasoning in Natural Language

职业：灵活而稳健的自然语言推理

• 批准号: 2145280
• 负责人: Gregory Durrett
• 金额: $ 50.48万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-07-01 至 2027-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2145280&HistoricalAwards=false
• 关键词: CAREER; Flexible; Robust; Reasoning; Natural

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Modern question answering systems, embedded in search engines and digital assistants, have improved dramatically with the development of large neural network models. When a user asks a simple question, these systems can typically return an answer directly rather than just linking to a webpage. However, these systems still fail on more complex questions, and when they fail, they may mislead their users. They lack an important capability that humans have: the ability to reason about and synthesize the information they see, retrieve and integrate additional information, and arrive at a justified conclusion. This CAREER project aims to address this shortcoming by developing systems that "think through" textual evidence, leading to more reliable answers that can be explained to a user. Such advances fit into a broader thread of building trustable AI systems that explicitly show their work and are auditable before and during their deployment.This project specifically addresses the problems of question answering and fact-checking by developing a learning-based system that reasons in natural language. The system takes text as input, then applies pre-trained neural network models to reformulate that text, derive conclusions from it, and eventually check a claim or verify an answer. This process produces a series of logically connected statements understandable by a human. This outcome is enabled by two modules. First, a deduction module repeatedly combines two statements and generates a third that follows from the inputs, encapsulating common logical rules. Second, a verifier determines whether the final deduced evidence validates the original claim. Both systems are built from pre-trained models like T5 that have demonstrated strong generalization capabilities. Collecting training data for these models constitutes a core challenge; the project's approach blends multiple strategies including synthetic data generation and human-in-the-loop annotation. These techniques are applied to the domains of question answering and fact checking, problems where providing additional explanation and justification instead of just giving a best-effort answer are essential to make usable systems. This system paves the way for NLP tools that know what they don't know, provide interpretability for end users, and enable system developers to better understand and improve their models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项是根据2021年《美国救援计划法》的全部或部分资助（公共法117-2）。嵌入在搜索引擎和数字助理中的现代问答系统随着大型神经网络模型的发展，已大大改善。当用户提出一个简单的问题时，这些系统通常可以直接返回答案，而不仅仅是链接到网页。但是，这些系统仍然在更复杂的问题上失败，当它们失败时，它们可能会误导用户。他们缺乏人类拥有的重要能力：推理和综合他们看到的信息，检索和整合其他信息的能力，并得出合理的结论。 这个职业项目旨在通过开发“通过”文本证据“思考”的系统来解决这一缺点，从而导致可以向用户解释的更可靠的答案。这样的进步符合建立可信赖的AI系统的更广泛的线索，这些系统明确显示了他们的工作，并且在部署之前和期间都可以进行审核。本项目专门解决了通过开发自然语言理由的基于学习的系统来解决问题的问题和事实检查的问题。该系统将文本视为输入，然后应用预训练的神经网络模型来重新制定该文本，从中得出结论，并最终检查索赔或验证答案。这个过程产生了人类可以理解的一系列逻辑上连接的陈述。这个结果由两个模块启用。首先，扣除模块重复结合了两个语句，并生成了三分之一的输入，从而封装了常见的逻辑规则。其次，验证者确定最终推论的证据是否验证了原始索赔。这两种系统都是由像T5这样的预训练模型构建的，这些模型表现出强大的概括能力。为这些模型收集培训数据构成了核心挑战。该项目的方法融合了多种策略，包括综合数据生成和循环注释。这些技术应用于答案和事实检查的域，提供其他解释和理由，而不仅仅是给出最佳答案，这对于使可用系统至关重要。该系统为知道他们不知道的知识，为最终用户提供可解释性的NLP工具铺平了道路，并使系统开发人员能够更好地理解和改进其模型。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛影响的审查标准来通过评估来进行评估的。

项目成果

The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning

• 发表时间: 2022-05
• 作者: Xi Ye;Greg Durrett

Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge

LM 可以从描述中学习新实体吗？

• 发表时间: 2023
• 期刊: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers
• 作者: Onoe, Yasumasa;Zhang, Michael J.Q.;Padmanabhan, Shankar;Durrett, Greg;Choi, Eunsol
• 通讯作者: Choi, Eunsol

Generating Literal and Implied Subquestions to Fact-check Complex Claims

生成字面和隐含的子问题来事实检查复杂的声明

• 发表时间: 2022
• 期刊: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP
• 作者: Chen, Jifan;Sriram, Aniruddh;Choi, Eunsol;Durrett, Greg
• 通讯作者: Durrett, Greg

Natural Language Deduction through Search over Statement Compositions

• DOI: 10.18653/v1/2022.findings-emnlp.358
• 发表时间: 2022-01
• 期刊: ArXiv
• 作者: Kaj Bostrom;Zayne Sprague;Swarat Chaudhuri;Greg Durrett

Complementary Explanations for Effective In-Context Learning

• DOI: 10.48550/arxiv.2211.13892
• 发表时间: 2022-11
• 期刊: ArXiv
• 作者: Xi Ye;Srini Iyer;Asli Celikyilmaz;Ves Stoyanov;Greg Durrett;Ramakanth Pasunuru