Structured Attention in Neural Networks for Natural Language Processing

神经网络的建模和计算优势使表征学习成为可能，它彻底改变了自然语言处理（NLP）领域。适用于NLP的许多不同子领域的通用模型是编码器-解码器神经网络。对ACL 2017会议（NLP最负盛名的国际会议）发表的所有论文的非正式调查显示，四分之一的论文使用了编码器-解码器网络。对于NLP研究人员来说，这是一个前所未有的一致性，尽管这个模型被用于许多不同的NLP任务。因此，该提案的重点很可能会使大量NLP研究人员和行业从业者受益。具有注意力的编码器-解码器网络已被应用于几个任务：神经机器翻译（NMT），语言模型，摘要，图像字幕，神经图灵机（模拟简单编程语言的语义），视觉和非视觉问答系统，对话系统和许多其他应用。对编码器-解码器网络的进一步研究可能会对NLP研究产生广泛的重大影响。将注意力模型添加到编码器-解码器网络中是一项重要的改进，从而从根本上提高了许多不同任务的准确性。目前注意力模型使用相对简单的加权平均模型。注意力模型有相当大的改进空间，以便用自然语言对各种复杂的依赖关系进行建模，并将其实用性扩展到新的NLP任务和设置。在本提案中，我们探索了对编码器-解码器网络模型的各种改进，这些模型对自然语言处理和机器学习具有广泛的适用性。

Representation learning, enabled by the modeling and computational benefits of neural networks, has revolutionized the field of natural language processing (NLP). A general model that is applicable to many different sub-areas of NLP is the encoder-decoder neural network. An informal survey of all the papers published in the ACL 2017 conference (the most prestigious international conference in NLP) showed that one in four papers used encoder-decoder networks. This is an unprecedented amount of conformity for NLP researchers even though this model was used for many different NLP tasks. As a result, the focus of this proposal is likely to benefit a large number of NLP researchers and industry practitioners. Encoder-decoder networks with attention have been applied to several tasks: neural machine translation (NMT), language models, summarization, captioning of images, neural Turing machines (modeling the semantics of simple programming languages), visual and non-visual question answering systems, dialog systems, and many other applications. Further research into encoder-decoder networks is likely to have a significant impact broadly in NLP research. The addition of attention models to encoder-decoder networks has been an essential improvement, resulting in radically improved accuracies in many different tasks. Currently attention models use relatively simple weighted average models. There is considerable room for improvements to attention models in order to model various complex dependencies in natural language and in order to expand their utility to new NLP tasks and settings. In this proposal we explore various improvements to the encoder-decoder network models that have broad applicability to natural language processing and machine learning.