RI: Small: Extracting and Representing Commonsense Knowledge Using Language Models

RI：小：使用语言模型提取和表示常识知识

• 批准号: 2006851
• 负责人: Douglas Downey
• 金额: $ 47万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2006851&HistoricalAwards=false
• 关键词: RI; Small; Extracting; Representing; Commonsense

As computers advance from serving as our tools to becoming our helpers and collaborators, they must be capable of commonsense reasoning--for example, knowing that a person needs to use their hand to open a door (and that this might be a problem if they are carrying groceries). This kind of commonsense reasoning is a longstanding, elusive goal of artificial intelligence, but is becoming within reach today due to the availability of vast amounts of data and more powerful computational models for learning from those data. This project is aimed at a key step in enabling commonsense reasoning by machines: the automatic acquisition of common sense knowledge. The project’s approach builds upon recent breakthroughs in language models that learn by reading large amounts of text, and combines these in novel ways with explicit commonsense knowledge gathered from humans. The project probes new, scalable methods for humans to impart their commonsense knowledge to the system, by using existing dictionaries and encyclopedias, building curricula that help machines build to commonsense mastery step by step, and by directly enforcing key logical constraints (for example, that if one item is bigger than another, then the second item must be smaller than the first). Success in this project could help power new virtual assistants, medical diagnosis and treatment systems, improved search engines, and other important applications of AI. The work also aims to enable the development of better language models themselves---improving current commercial technologies such as speech recognition and machine translation, and ultimately helping to power the next generation of computer systems capable of communicating with people more naturally using language. Along the way, the project will help train the next generation of students about these approaches and technologies, via education and outreach activities.The technical strategy used in the project involves learning unsupervised neural language models (LMs) to capture textual distributions, and then extracting common sense knowledge from those models. This approach is challenging because common sense knowledge is multifarious and massive, and yet is not often explicitly stated in text. The project aims to overcome this challenge using several methods for scalably incorporating human input in concert with neural language models. First, the project studies how to use explicit lexical knowledge found in dictionaries to improve LMs, extending prior work in modeling the definitions of terms with neural LMs. Next, the project is investigating a “scaffold” of semantic tasks (a task curriculum of increasing complexity) incrementally constructing models for each task in turn in a way that aims to improve the learning of each subsequent task. Third, the project is developing methods for encoding commonsense logical constraints within neural language models. Lastly, because time and energy cost is a potential barrier to the application of the proposed techniques, the project is also studying how to make its approaches efficient. In particular, the project is investigating ways to scale-up LMs to larger corpora while reducing the significant computational and energy cost in LM training, by learning how to automatically identify text that will be more informative for training.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着计算机从作为我们的工具发展成为我们的助手和合作者，它们必须能够进行常识性推理-例如，知道一个人需要用手开门（如果他们携带杂货，这可能是一个问题）。 这种常识推理是人工智能长期以来的一个难以捉摸的目标，但由于大量数据和更强大的计算模型的可用性，今天正在变得触手可及。 该项目旨在通过机器实现常识推理的关键步骤：自动获取常识知识。 该项目的方法建立在语言模型的最新突破之上，这些模型通过阅读大量文本来学习，并以新颖的方式将这些与从人类收集的明确常识知识结合起来。 该项目探索了新的、可扩展的方法，让人类将常识知识传授给系统，方法是使用现有的字典和词典，建立帮助机器逐步掌握常识的课程，并直接执行关键的逻辑约束（例如，如果一个项目比另一个项目大，那么第二个项目必须比第一个项目小）。 该项目的成功将有助于推动新的虚拟助手、医疗诊断和治疗系统、改进的搜索引擎以及人工智能的其他重要应用。 这项工作还旨在开发更好的语言模型，改进当前的商业技术，如语音识别和机器翻译，并最终帮助推动下一代计算机系统，使其能够更自然地使用语言与人交流。 沿着的是，该项目将通过教育和推广活动，帮助培训下一代学生了解这些方法和技术。该项目中使用的技术策略包括学习无监督神经语言模型（LM）来捕获文本分布，然后从这些模型中提取常识知识。 这种方法是具有挑战性的，因为常识知识是五花八门和庞大的，但往往没有明确的文本。 该项目旨在使用几种方法来克服这一挑战，这些方法可扩展地将人类输入与神经语言模型相结合。 首先，该项目研究如何使用字典中发现的显式词汇知识来改进LM，扩展了先前使用神经LM建模术语定义的工作。 接下来，该项目正在研究语义任务的“脚手架”（一个日益复杂的任务课程），以一种旨在改善每个后续任务学习的方式，依次递增地为每个任务构建模型。 第三，该项目正在开发在神经语言模型中编码常识逻辑约束的方法。 最后，由于时间和能源成本是应用拟议技术的潜在障碍，该项目还在研究如何使其方法有效。 特别是，该项目正在研究如何将LM扩展到更大的语料库，同时通过学习如何自动识别对培训更具信息性的文本，减少LM培训中的重大计算和能源成本。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Who’s on First?: Probing the Learning and Representation Capabilities of Language Models on Deterministic Closed Domains

• DOI: 10.18653/v1/2021.conll-1.16
• 发表时间: 2021
• 作者: David Demeter;Doug Downey

CODE: Compiler-based Neuron-aware Ensemble training

• 发表时间: 2021
• 作者: E. Trainiti;Thanapon Noraset;David Demeter;Doug Downey;Simone Campanoni

Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models

通过语言模型的组合微调学习执行复杂任务

• DOI: 10.18653/v1/2022.findings-emnlp.121
• 发表时间: 2022
• 期刊: Findings of EMNLP 2022
• 作者: Bursztyn, Victor;Demeter, David;Downey, Doug;Birnbaum, Larry
• 通讯作者: Birnbaum, Larry

“It doesn’t look good for a date”: Transforming Critiques into Preferences for Conversational Recommendation Systems

“约会看起来不太好”：将批评转化为对话推荐系统的偏好

• DOI: 10.18653/v1/2021.emnlp-main.145
• 发表时间: 2021
• 期刊: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
• 作者: Bursztyn, Victor;Healey, Jennifer;Lipka, Nedim;Koh, Eunyee;Downey, Doug;Birnbaum, Larry
• 通讯作者: Birnbaum, Larry