Joint Text Understanding and Coreference Resolution: Augmenting Recurrent Neural Networks with Entity-Centric Discourse Memory

该研究项目属于EPSRC的自然语言处理、人工智能技术和人机交互研究领域，主要属于ICT主题。尽管最近在机器学习和自然语言处理方面取得了进展，但准确的文本理解仍然是一个具有挑战性的问题。先前的工作通常使用递归神经网络，将问题、段落和每个候选答案嵌入到实值向量中，并可选择使用额外的注意机制。这种方法有两个缺点。首先，神经网络隐藏状态的有限容量意味着模型必须在网络容量范围内压缩所有相关信息。例如，模型需要建立提及“Hillary Clinton”和“Clinton”指同一个人的联系（即共指解决任务），以及不同实体之间的关系。其次，所得到的向量嵌入通常不容易被人类理解。这是一个重要的研究领域，因为人类和计算机之间通过文本或语音进行任何有意义的交互都需要正确理解和推理自然语言的意义和上下文的能力。设计能够识别实体及其之间关系的智能系统作为文本理解的一部分，将产生更准确和可解释的文本理解系统。本研究项目的目的是设计统计模型和算法，使用存储远程话语特征的实体中心记忆来联合执行共指解析和文本理解。最近的研究表明，实体和参考的推理有利于语言建模和文本理解，尽管所提出的方法是自动地和文本理解任务一起学习这些共同参考信息。该方法的第二个目标是通过将每个信息与相关实体相关联，并明确地对共同引用链接和实体关系进行预测，从而提高模型的可解释性。我们提出的方法同样利用了循环神经网络和注意机制的力量，但也用“话语记忆”来增强模型，该模型可以跟踪文本中的每个实体及其各种提及。这种方法的好处是以一种计算效率高的方式将关于一个实体的所有相关信息集成到同一个文件中，即使每个信息是单独提到的，并且可能在文章中很远的地方出现。虽然之前的一些工作类似地研究了具有话语层特征的以实体为中心的文本理解方法，但我们的方法的新颖性还在于与文本理解任务一起执行共指解析，这仍然是一项具有挑战性的任务。

This research project falls within the natural language processing, artificial intelligence technologies, and human-computer interaction research areas of EPSRC primarily within the ICT theme.Despite recent advances in machine learning and natural language processing, accurate text comprehension has remained a challenging problem. Prior works often used recurrent neural networks that embed the question, passage, and each answer candidate into real-valued vectors, optionally with an additional attention mechanism.Such approaches have two drawbacks. First, the finite capacity of the neural network hidden states means that the model has to compress all the relevant information within the network capacity. For instance, the model needs to make the connection that mentions of "Hillary Clinton" and "Clinton" refer to the same person (i.e. coreference resolution task), along with the relation between different entities. Second, the resulting vector embedding is often not easily interpretable to humans.This research area is an important one, since any meaningful interaction between humans and computers through text or speech requires the ability to properly understand and reason about meaning and context of natural language. Designing intelligent systems that can recognize entities and the relations between them as part of text comprehension would give rise to more accurate and interpretable text comprehension systems.The aim of this research project is to design statistical models and algorithms that can jointly perform coreference resolution and text comprehension, using an entity-centric memory that stores long-range discourse features. Recent work have demonstrated that reasoning about entities and references is beneficial for language modeling and text comprehension, although the proposed approach learns such coreference information automatically and jointly with the text comprehension task. A second goal of this approach is to improve model interpretability by associating each information with the relevant entity, and explicitly making predictions about coreference links and entity relations.Our proposed approach similarly leverages the strength of recurrent neural networks and attention mechanisms, but also augments the model with a "discourse memory" that keeps track of each entity and its various mentions within the text. This approach has the benefit of integrating all pertinent information regarding an entity into the same file in a computationally efficient manner, even though each information is mentioned separately and may occur distantly in the passage. While some prior work have similarly investigated an entity-centric approach to text comprehension with discourse-level features, the novelty of our approach additionally lies in performing the coreference resolution jointly with the text comprehension task, which remains a challenging task.