On the Limits of Learning to Actively Learn Semantic Representations
On the Limits of Learning to Actively Learn Semantic Representations
复制标题
DOI：
10.18653/v1/k19-1042
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Omri Koshorek;Gabriel Stanovsky;Yichu Zhou;Vivek Srikumar;Jonathan Berant
中科院分区：
文献类型：
--
作者：
Omri Koshorek;Gabriel Stanovsky;Yichu Zhou;Vivek Srikumar;Jonathan Berant
自然语言理解的目标之一是开发将句子映射为意义表示的模型。然而，训练这样的模型需要对复杂结构进行昂贵的注释，这阻碍了它们的采用。学习主动学习（LTAL）是一种最近的范例，用于通过学习选择哪些样本应该被标记的策略来减少标记数据的数量。在这项工作中，我们研究LTAL学习语义表示，如QA-SRL。我们表明，即使是一个Oracle的政策，允许选择的例子，最大限度地提高性能的测试集（并构成了上限的潜力LTAL），并没有实质性地提高性能相比，随机的政策。我们调查的因素，可以解释这一发现，并表明LTAL的成功应用程序的一个显着特点是优化和Oracle的政策选择过程之间的相互作用。在LTAL的成功应用中，Oracle策略选择的示例基本上不依赖于优化过程，而在我们的设置中，优化的随机性强烈影响Oracle选择的示例。我们的结论是，目前的适用性LTAL提高数据效率学习语义表示是有限的。
One of the goals of natural language understanding is to develop models that map sentences into meaning representations. However, training such models requires expensive annotation of complex structures, which hinders their adoption. Learning to actively-learn(LTAL) is a recent paradigm for reducing the amount of labeled data by learning a policy that selects which samples should be labeled. In this work, we examine LTAL for learning semantic representations, such as QA-SRL. We show that even an oracle policy that is allowed to pick examples that maximize performance on the test set (and constitutes an upper bound on the potential of LTAL), does not substantially improve performance compared to a random policy. We investigate factors that could explain this finding and show that a distinguishing characteristic of successful applications of LTAL is the interaction between optimization and the oracle policy selection process. In successful applications of LTAL, the examples selected by the oracle policy do not substantially depend on the optimization procedure, while in our setup the stochastic nature of optimization strongly affects the examples selected by the oracle. We conclude that the current applicability of LTAL for improving data efficiency in learning semantic meaning representations is limited.