RI: Small: Extracting and Representing Commonsense Knowledge Using Language Models

随着计算机从我们的工具发展成为我们的帮手和合作者，它们必须具备常识性推理能力——例如，知道一个人需要用手开门（如果他们拿着杂货，这可能是个问题）。这种常识性推理是人工智能长期以来难以实现的目标，但由于大量数据的可用性和从这些数据中学习的更强大的计算模型，今天正变得触手可及。该项目旨在实现机器常识推理的关键步骤：自动获取常识知识。该项目的方法建立在通过阅读大量文本来学习的语言模型的最新突破之上，并以新颖的方式将这些模型与从人类那里收集的明确的常识性知识结合起来。该项目探索新的、可扩展的方法，通过使用现有的词典和百科全书，构建帮助机器逐步掌握常识的课程，以及直接执行关键的逻辑约束（例如，如果一个项目比另一个项目大，那么第二个项目必须比第一个项目小），让人类向系统传授常识知识。这个项目的成功可以为新的虚拟助手、医疗诊断和治疗系统、改进的搜索引擎和其他重要的人工智能应用提供动力。这项工作还旨在开发更好的语言模型本身——改进当前的商业技术，如语音识别和机器翻译，并最终帮助推动下一代能够更自然地使用语言与人交流的计算机系统。在此过程中，该项目将通过教育和推广活动，帮助培训下一代学生了解这些方法和技术。项目中使用的技术策略包括学习无监督神经语言模型（LMs）来捕获文本分布，然后从这些模型中提取常识性知识。这种方法是具有挑战性的，因为常识知识是多种多样和庞大的，但通常不会在文本中明确说明。该项目旨在克服这一挑战，使用几种方法可扩展地将人类输入与神经语言模型相结合。首先，该项目研究了如何使用字典中发现的显式词汇知识来改进LMs，扩展了之前使用神经LMs建模术语定义的工作。接下来，该项目正在研究语义任务的“脚手架”（一个日益复杂的任务课程），以一种旨在改善每个后续任务学习的方式依次为每个任务增量构建模型。第三，该项目正在开发在神经语言模型中编码常识性逻辑约束的方法。最后，由于时间和能源成本是拟议技术应用的潜在障碍，该项目还在研究如何使其方法高效。特别是，该项目正在研究如何将LM扩展到更大的语料库，同时通过学习如何自动识别对训练更有信息的文本，减少LM训练中显著的计算和能源成本。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

As computers advance from serving as our tools to becoming our helpers and collaborators, they must be capable of commonsense reasoning--for example, knowing that a person needs to use their hand to open a door (and that this might be a problem if they are carrying groceries). This kind of commonsense reasoning is a longstanding, elusive goal of artificial intelligence, but is becoming within reach today due to the availability of vast amounts of data and more powerful computational models for learning from those data. This project is aimed at a key step in enabling commonsense reasoning by machines: the automatic acquisition of common sense knowledge. The project’s approach builds upon recent breakthroughs in language models that learn by reading large amounts of text, and combines these in novel ways with explicit commonsense knowledge gathered from humans. The project probes new, scalable methods for humans to impart their commonsense knowledge to the system, by using existing dictionaries and encyclopedias, building curricula that help machines build to commonsense mastery step by step, and by directly enforcing key logical constraints (for example, that if one item is bigger than another, then the second item must be smaller than the first). Success in this project could help power new virtual assistants, medical diagnosis and treatment systems, improved search engines, and other important applications of AI. The work also aims to enable the development of better language models themselves---improving current commercial technologies such as speech recognition and machine translation, and ultimately helping to power the next generation of computer systems capable of communicating with people more naturally using language. Along the way, the project will help train the next generation of students about these approaches and technologies, via education and outreach activities.The technical strategy used in the project involves learning unsupervised neural language models (LMs) to capture textual distributions, and then extracting common sense knowledge from those models. This approach is challenging because common sense knowledge is multifarious and massive, and yet is not often explicitly stated in text. The project aims to overcome this challenge using several methods for scalably incorporating human input in concert with neural language models. First, the project studies how to use explicit lexical knowledge found in dictionaries to improve LMs, extending prior work in modeling the definitions of terms with neural LMs. Next, the project is investigating a “scaffold” of semantic tasks (a task curriculum of increasing complexity) incrementally constructing models for each task in turn in a way that aims to improve the learning of each subsequent task. Third, the project is developing methods for encoding commonsense logical constraints within neural language models. Lastly, because time and energy cost is a potential barrier to the application of the proposed techniques, the project is also studying how to make its approaches efficient. In particular, the project is investigating ways to scale-up LMs to larger corpora while reducing the significant computational and energy cost in LM training, by learning how to automatically identify text that will be more informative for training.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.