CAREER: Teaching Machines through Human Explanation for Information Extraction

社会产生的大部分数据都是自由格式的文本数据。随着文本数据量的不断增长，单靠人类是无法理解每一条发布的文本信息的。因此，需要基于机器的方法从大量文本数据中提取显著实体及其关系。虽然开发此类方法的努力在学术界已被证明是成功的，但这种成功很少转化为使用提取系统来解决现实世界问题的实践者。翻译失败的一个重要原因是机器学习提取模型需要大量的训练样例。即使存在足够数量的例子，机器也会学习非常严格的方法，以至于即使是轻微的拼写错误也可能导致失败。因此，我们需要重新思考如何开发、改进和维护这种基于机器的提取方法。这个项目提出了一种新的方法，基于为机器做出的正确和不正确的决定提供解释的想法，目的是为了机器训练需要更少的例子，以及提供一个软化当前提取方法刚性的过程。为了实现这一目标，该项目将要求（从人类）自然语言解释机器应该如何推理他们的任务，以及解释纠正错误的推理，并在机器的基本原理可能出错时发出警报系统。与其将人类仅仅视为“标签的来源”，该项目旨在开发一种新的学习框架，直接模拟人类的自然语言解释，为机器提供标签基本原理或纠正观察到的错误基本原理。该项目将开发基于解释的学习方法，以捕获人类自然语言解释的组成性质，并研究基于提供的人类对不良行为的解释来更新模型参数的基于解释的模型优化方法。为了适应不断变化的数据分布，本项目将制定一个“人在环”（human-in-the-loop）的持续模型优化框架，自动识别有问题的模型行为模式，并征求人类反馈来纠正模型。有了这些进步，该项目希望从根本上改变模型的训练、改进和更新方式，并希望通过利用人类解释中包含的专家知识来实现这一目标。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The majority of data being generated by society is free-form textual data. As the volume of text data continues to grow, humans alone cannot hope to be able to understand every piece of textual information published. Hence, a need for machine-based methods to extract salient entities along with their relationships from massive textual data is needed. While efforts to develop such methods have proven successful in academia, that success rarely translates over to practitioners employing extraction systems to solve real-world problems. A significant cause of this failure in translation is the requirement of copious amounts of training examples for a machine to learn extraction models. Even when a sufficient number of examples exist, machines learn very rigid methods, such that even a slight misspelling can cause a failure. Therefore, a re-think of how we develop, refine and maintain such machine-based extraction methods is required. This project proposes a new methodology based around the idea of providing explanations for both correct and incorrect decisions made by a machine, with the intention of requiring far fewer examples for machine training, as well as providing a process of softening the rigidity of current extraction methods. To achieve this goal, this project will solicit (from humans) natural language explanations on how a machine should reason about their task, as well as explanations correcting erroneous reasoning and an alerting system for when a machine’s rationale is possibly going wrong. Rather than treating humans as merely a “source of labels”, this project aims at developing a new learning framework that directly models a human’s natural language explanations to either provide a machine with labeling rationale or correct an observed erroneous rationale. The project will develop explanation-based learning methods that can capture the compositional nature of human natural language explanations, and study explanation-guided model refinement methods to update model parameters based on the provided human explanations regarding undesirable behaviors. To adapt to changing data distribution, this project will formulate a human-in-the-loop continual model refinement framework where problematic model behavioral patterns are automatically identified, and human feedback is solicited to correct the model. With these advancements, the project looks to fundamentally change the way models are trained, refined and updated, and look to do it by exploiting the expert knowledge contained within human explanations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.