eNeMILP: Non-Monotonic Incremental Language Processing

eNeMILP：非单调增量语言处理

• 批准号: EP/R021643/2
• 负责人: Andreas Vlachos
• 金额: $ 5.68万
• 依托单位: University of Cambridge
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2018
• 资助国家: 英国
• 起止时间: 2018 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FR021643%2F2
• 关键词: eNeMILP; Non; Monotonic; Incremental; Language

Research in natural language processing (NLP) is driving advances in many applications such as search engines and personal digital assistants, e.g. Apple's Siri and Amazon's Alexa. In many NLP tasks the output to be predicted is a graph representing the sentence, e.g. a syntax tree in syntactic parsing or a meaning representation in semantic parsing. Furthermore, in other tasks such as natural language generation and machine translation the predicted output is text, i.e. a sequence of words. Both types of NLP tasks have been tackled successfully with incremental modelling approaches in which prediction is decomposed into a sequence of actions constructing the output.Despite its success, a fundamental limitation in incremental modelling is that the actions considered typically construct the output monotonically, e.g. in natural language generation each action adds a word to the output but never removes or changes a previously predicted one. Thus, relying exclusively on monotonic actions can decrease accuracy, since the effect of incorrect actions cannot be amended. Furthermore, these actions will be used to predict the following ones, likely to result in an error cascade.We propose an 18-month project to address this limitation and learn non-monotonic incremental language processing models, i.e. incremental models that consider actions that can "undo" the outcome of previously predicted ones. The challenge in incorporating non-monotonic actions is that, unlike their monotonic counterparts, they are not straightforward to infer from the labelled data typically available for training, thus rendering standard supervised learning approaches inapplicable. To overcome this issue we will develop novel algorithms under the imitation learning paradigm to learn non-monotonic incremental models without assuming action-level supervision, relying instead on instance-level loss functions and the model's own predictions in order to learn how to recover from incorrect actions to avoid error cascades. To succeed in this goal, this proposal has the following research objectives:1) To model non-monotonic incremental prediction of structured outputs in a generic way that can be applied to a variety of tasks with natural language text as output2) To learn non-monotonic incremental predictors using imitation learning and improve upon the accuracy of monotonic incremental models both in terms of automatic measures such as BLEU and human evaluation. 3) To extend the proposed approach to structured prediction tasks with graph as output.4) To release software implementations of the proposed methods to facilitate reproducibility and wider adoption by the research community.The research proposed focuses on a fundamental limitation in incremental language processing models, which have been successfully applied to a variety of natural language processing tasks, thus we anticipate the proposal to have a wide academic impact. Furthermore, the tasks we will evaluate it on, namely natural language generation and semantic parsing, are essential components to natural language interfaces and personal digital assistants. Improving these technologies will enhance accessibility to digital information and services. We will demonstrate the benefits of our approach through our collaboration with our project partners Amazon who are supporting the proposal both in terms of cloud computing credits but also by hosting the research associate in order to apply the outcomes of the project to industry-scale datasets.

自然语言处理（NLP）的研究正在推动许多应用的进步，比如搜索引擎和个人数字助理，比如苹果的Siri和亚马逊的Alexa。在许多NLP任务中，要预测的输出是表示句子的图，例如句法分析中的语法树或语义分析中的意义表示。此外，在自然语言生成和机器翻译等其他任务中，预测输出是文本，即单词序列。这两种类型的NLP任务都已经通过增量建模方法成功地解决了，其中预测被分解为构建输出的一系列动作。尽管它取得了成功，但增量建模的一个基本限制是，所考虑的动作通常是单调地构建输出，例如，在自然语言生成中，每个动作向输出添加一个单词，但从不删除或改变先前预测的单词。因此，完全依赖单调动作会降低准确性，因为不正确动作的影响无法修正。此外，这些操作将用于预测接下来的操作，可能导致错误级联。我们提出了一个为期18个月的项目来解决这一限制，并学习非单调增量语言处理模型，即考虑可以“撤销”先前预测结果的动作的增量模型。整合非单调行为的挑战在于，与单调行为不同，它们不能直接从通常可用于训练的标记数据中推断出来，因此使得标准的监督学习方法不适用。为了克服这个问题，我们将在模仿学习范式下开发新的算法来学习非单调增量模型，而不假设行动级监督，而是依赖于实例级损失函数和模型自己的预测，以学习如何从错误的动作中恢复以避免错误级联。为了实现这一目标，本提案有以下研究目标：1)以一种通用的方式对结构化输出的非单调增量预测进行建模，该方法可以应用于以自然语言文本为输出的各种任务；2)使用模仿学习学习非单调增量预测器，并在自动测量（如BLEU）和人类评估方面提高单调增量模型的准确性。3)将提出的方法扩展到以图为输出的结构化预测任务。4)发布所提出方法的软件实现，以促进研究社区的可重复性和更广泛的采用。该研究主要关注增量语言处理模型的一个基本限制，该模型已经成功地应用于各种自然语言处理任务，因此我们预计该建议将产生广泛的学术影响。此外，我们将对其进行评估的任务，即自然语言生成和语义解析，是自然语言接口和个人数字助理的重要组成部分。改进这些技术将提高数字信息和服务的可及性。我们将通过与项目合作伙伴亚马逊的合作来展示我们方法的好处，亚马逊不仅在云计算积分方面支持该提案，而且还通过托管研究助理来将项目成果应用于工业规模的数据集。

项目成果

Sheffield at E2E: structured prediction approaches to end-to-end language generation

谢菲尔德在 E2E：端到端语言生成的结构化预测方法

• 发表时间: 2018
• 作者: Chen M

Merge and Label: A novel neural network architecture for nested NER

合并和标签：一种新颖的嵌套 NER 神经网络架构

• DOI: 10.17863/cam.46494
• 发表时间: 2019
• 期刊: Apollo - University of Cambridge Repository
• 作者: Joseph Fisher

HighRES: Highlight-based Reference-less Evaluation of Summarization

HighRES：基于突出显示的无参考摘要评估

• DOI: 10.17863/cam.40621
• 发表时间: 2019
• 期刊: Apollo - University of Cambridge Repository
• 作者: Hardy

Leveraging Type Descriptions for Zero-shot Named Entity Recognition and Classification

利用类型描述进行零样本命名实体识别和分类

• DOI: 10.18653/v1/2021.acl-long.120
• 发表时间: 2021
• 作者: Aly R

Incremental Beam Manipulation for Natural Language Generation

• DOI: 10.18653/v1/2021.eacl-main.219
• 发表时间: 2021-02
• 期刊: ArXiv
• 作者: J. Hargreaves;Andreas Vlachos;Guy Edward Toh Emerson