A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks
复制标题
DOI：
10.1609/aaai.v33i01.33016949
复制
发表时间：
2018-11
期刊：
影响因子：
3.7
通讯作者：
Victor Sanh;Thomas Wolf;Sebastian Ruder
中科院分区：
文献类型：
--
作者：
Victor Sanh;Thomas Wolf;Sebastian Ruder
已经投入了大量的精力来评估是否可以利用多任务学习来学习可用于各种自然语言处理（NLP）下游应用程序的丰富表示。然而，仍然缺乏对多任务学习具有显着效果的设置的理解。在这项工作中，我们介绍了一个分层模型，在一组精心挑选的语义任务的多任务学习设置训练。该模型以分层的方式进行训练，通过监督模型底层的一组低级任务和模型顶层的更复杂任务来引入归纳偏差。该模型在许多任务上实现了最先进的结果，即命名实体识别，实体提及检测和关系提取，而无需手工设计的功能或外部NLP工具，如语法分析器。分层训练监督在模型的较低层诱导一组共享的语义表示。我们表明，当我们从模型的底层移动到顶层时，层的隐藏状态往往表示更复杂的语义信息。
Much effort has been devoted to evaluate whether multi-task learning can be leveraged to learn rich representations that can be used in various Natural Language Processing (NLP) down-stream applications. However, there is still a lack of understanding of the settings in which multi-task learning has a significant effect. In this work, we introduce a hierarchical model trained in a multi-task learning setup on a set of carefully selected semantic tasks. The model is trained in a hierarchical fashion to introduce an inductive bias by supervising a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model. This model achieves state-of-the-art results on a number of tasks, namely Named Entity Recognition, Entity Mention Detection and Relation Extraction without hand-engineered features or external NLP tools like syntactic parsers. The hierarchical training supervision induces a set of shared semantic representations at lower layers of the model. We show that as we move from the bottom to the top layers of the model, the hidden states of the layers tend to represent more complex semantic information.