Collaborative Research: RI: Medium: Expert-in-the-Loop Neural Summarization for Consequential Domains

自动摘要方法的目标是创建文本的简短版本(例如，新闻或科学文章)，这些文本仍然准确地传达其要点。摘要方法提供了一种潜在的方法来对抗在许多领域普遍存在的“信息过载”问题。但很多关于自动摘要的研究主要集中在一种类型的数据上：新闻文章。这并不是因为总结新闻文章被视为特别重要。相反，这是因为有了方便可用的大数据集，可以用来“训练”机器学习模型来执行摘要。然而，由此产生的对方法的关注假设了一个人可以访问大量“训练数据”来训练摘要模型，这扭曲了研究的优先事项；在研究如何将自动摘要方法用于重要但专门的领域，如医学或法律方面，几乎没有做过什么工作。在这类领域中，人们不太可能获得大量的手动编写摘要的数据集。此外，这些领域的领域专家不太可能盲目相信系统生成的摘要(他们也不应该)。这促使需要在模型如何生成特定摘要方面具有透明度，并需要能够使专家更一般地与模型进行交互的方法。该项目旨在通过在一个人具有有限的显式监督的领域和任务的背景下研究和扩展现代的、预先训练的神经总结模型的能力来解决这些问题，并且在这些领域和任务中对真实准确的总结的需求越来越高。该项目将涉及对最先进的模型进行批判性评估，当对其进行微调以在有限监督下进行医学等领域的总结时；一个具体目标是根据模型输出的真实性来表征它们的行为。然后，我们的想法是扩展这些模式，通过积极的学习方法、替代的监督类型(例如，专家“重点”)和新颖的培训前目标，允许进行互动和有效的监督。最后，调查人员将设计能够提高透明度和可控性的体系结构；这将使用潜在变量汇总模型来实现，这反过来将允许检查哪些输入部分通知了特定的输出。这将为最终用户(领域专家)提供一种验证模型输出的自然方法，也将提供一种“调试”摘要系统的方法。希望这些技术创新将使领域专家受益于自动摘要技术。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Automatic summarization methods aim to create shortened versions of texts (for example, news or scientific articles) that still accurately communicate their main points. Summarization methods provide a potential means to counteract the problem of “information overload” which is prevalent across many areas. But much of the research on automatic summarization has focussed largely on just one type of data: news articles. This is not because summarizing news articles is seen as particularly important. Rather, it is a result of there being conveniently available large datasets that can be used to “train” machine learning models to perform summarization. However, the resultant focus on approaches that assume a setting in which one has access to large volumes of “training data” to use to train summarization models has warped research priorities; little work has been done on investigating how automatic summarization methods might be used in important but specialized domains such as medicine or law. In these kinds of areas one is unlikely to have access to a massive dataset of manually written summaries. Furthermore, domain experts in such areas are not likely to blindly trust a system-generated summary (nor should they). This motivates a need for transparency with respect to how the model generated a particular summary, and for approaches that permit the expert to interact with the model more generally. This project aims to address these issues by investigating and extending the capabilities of modern, pre-trained, neural summarization models in the context of domains and tasks in which one has limited explicit supervision, and where there is a heightened need for factually accurate summaries. The project will involve critically evaluating state-of-the-art models when fine-tuned for summarization in domains like medicine under limited supervision; a specific aim is to characterize their behavior with respect to the factuality of model outputs. The idea is then to extend these models to permit interactive and efficient supervision, via active learning methods, alternative types of supervision (e.g., expert “highlights”), and novel pre-training objectives. Finally, the investigators will design architectures that afford increased transparency and controllability; this will be accomplished using latent variable summarization models, which will in turn allow one to inspect which input segments informed particular outputs. This will provide a natural means for the end-user (domain expert) to verify model outputs, and it will also provide a means to “debug” summarization systems. The hope is that these technical innovations will allow domain experts to benefit from automated summarization technology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.