Information-Theoretic Probing for Linguistic Structure
Information-Theoretic Probing for Linguistic Structure
复制标题
语言结构的信息论探索
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Ryan Cotterell
中科院分区：
文献类型：
--
作者：
Tiago Pimentel;Josef Valvoda;R. Maudslay;Ran Zmigrod;Adina Williams;Ryan Cotterell
神经网络在NLP任务的潜水员中的成功使研究人员质疑这些网络实际上“知道”自然语言是一种自然的评估方式。监督模型可以从网络的学会中预测该语言任务，如果探测器表现良好，研究人员可能会包括对与该探测的知识有关任务。一个通常的信念是，使用更简单的模型是更好的逻辑，即简单的模型可以识别语言结构，但我们不学习任务本身。收到的智慧：一个人应该始终选择最高的探针，即使它更复杂，它也会导致更严格的估计，从而揭示更多语言信息代表性的固有。再加上英语 - 提取11种语言。
The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually “know” about natural language. Probes are a natural way of assessing this. When probing, a researcher chooses a linguistic task and trains a supervised model to predict annotations in that linguistic task from the network’s learned representations. If the probe does well, the researcher may conclude that the representations encode knowledge related to the task. A commonly held belief is that using simpler models as probes is better; the logic is that simpler models will identify linguistic structure, but not learn the task itself. We propose an information-theoretic operationalization of probing as estimating mutual information that contradicts this received wisdom: one should always select the highest performing probe one can, even if it is more complex, since it will result in a tighter estimate, and thus reveal more of the linguistic information inherent in the representation. The experimental portion of our paper focuses on empirically estimating the mutual information between a linguistic property and BERT, comparing these estimates to several baselines. We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research—plus English—totalling eleven languages. Our implementation is available in https://github.com/rycolab/info-theoretic-probing.