Exploring New Neural Computing Models for Natural Language Understanding

在这项研究计划中，我们的目标是探索新的神经计算模型，以解决当前自然语言理解深度学习方法的一些不足。首先，我们将研究新的方法来执行有效的无监督或半监督学习神经网络的语言理解。目前深度学习方法的成功主要依赖于流行的基于误差反向传播的监督学习算法，该算法需要大量标记数据作为任何有效学习的先决条件。与语音和视觉不同，为任何语言理解任务收集人类标记的数据更具挑战性。另一方面，我们可以很容易地从网络上的许多来源获得大量未标记的文本。一个成功的用于自然语言理解的神经计算模型需要使用更有效的无监督学习方法来从这些未标记的数据中引导自己，然后仅使用少量的标记数据进行微调以实现特定的理解任务。这种半监督学习是在不久的将来构建自然语言理解系统的一种有前途和可行的方法。其次，目前的神经网络在长期记忆机制方面非常薄弱，而长期记忆机制对于理解自然语言至关重要。同一个句子在不同的语境中可能有不同的含义，更糟糕的是，对纯文本的准确理解通常来自背景知识，而背景知识不是给定文本的一部分。在许多情况下，给定的文本只是作为一个触发器来检索一些背景知识或长期记忆，以输出为理解。我们将探索一种新的方法，结合联合收割机的神经网络连接模型与传统的符号方法来解决这个问题，因为符号方法在推理知识的巨大优势。神经网络将被用作一个灵活而强大的模型，将表层文本与背景知识源联系起来，以进行有效的推理。最后，我们提出了一个开放域的问答任务作为我们的研究如何进行有效的无监督学习和联合收割机连接模型与传统的符号方法的主要测试床。

In this research program, we aim to explore new neural computing models to address some deficiencies of the current deep learning approaches for natural language understanding. First of all, we will study new methods to perform effective unsupervised or semi-supervised learning of neural networks for language understanding. The current successes of deep learning approaches largely rely on the popular error back-propagation based supervised learning algorithm, which requires a large amount of labeled data as prerequisites for any effective learning. Unlike speech and vision, it is much more challenging to collect human-labeled data for any language understanding tasks. On the other hand, we may easily have access to tons of unlabeled text from many sources on the Web. A successful neural computing model for natural language understanding needs to use a more effective unsupervised learning method to bootstrap itself from these unlabeled data, and then fine-tune towards a specific understanding task using only a small amount of labeled data. This sort of semi-supervised learning is a promising and viable approach to build natural language understanding systems in the near future. Secondly, the current neural networks are quite weak on long-term memory mechanisms, which are crucial in understating natural language. A same sentence may have very different meanings when appearing in various contexts, and even worse, the exact understanding of plain text normally stems from the background knowledge, which is not part of given text. In many cases, the given text simply serves as a trigger to retrieve some background knowledge or long-term memory to output as understanding. We will explore a novel approach to combine the neural networks based connectionist models with traditional symbolic approaches to address this issue since the symbolic approaches have shown tremendous advantages in reasoning over knowledge. The neural networks will be used as a flexible and powerful model to link surface text to background knowledge sources for effective reasoning. At last, we propose to investigate an open-domain question answer task as the main test bed for our research on both how to perform effective unsupervised learning and to combine connectionist models with traditional symbolic approaches.