Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment
Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment
复制标题
您知道佛罗伦萨挤满了游客吗？
DOI：
10.18653/v1/p19-1412
复制
发表时间：
2019
影响因子：
10.9
通讯作者：
M. Marneffe
中科院分区：
文献类型：
--
作者：
Nan;M. Marneffe
当演讲者玛丽问“你知道佛罗伦萨挤满了游客吗？"时，我们让她相信佛罗伦萨挤满了游客，但如果她问“你认为佛罗伦萨挤满了游客吗？"，就不会了。说话人承诺（也称为事件真实性）的推断对于信息抽取和问答系统至关重要。在这里，我们探讨的假设，语言赤字驱动现有的扬声器承诺模型的错误模式，通过分析语言相关的模型错误的一个具有挑战性的自然数据集。我们评估两个国家的最先进的扬声器承诺模型的承诺银行，英语数据集自然发生的话语。CommitmentBank在四种蕴涵取消环境（否定、情态、疑问、条件）下标注了说话人对嵌入小句的动词（“know”、“think”）的补语（在我们的例子中是“佛罗伦萨挤满了游客”）内容的承诺。按语言特征对项目进行细分揭示了不对称的错误模式：虽然模型在某些类别上取得了良好的性能（例如，否定），它们未能推广到不同的语言结构（例如，条件句），突出改进方向。
When a speaker, Mary, asks “Do you know that Florence is packed with visitors?”, we take her to believe that Florence is packed with visitors, but not if she asks “Do you think that Florence is packed with visitors?”. Inferring speaker commitment (aka event factuality) is crucial for information extraction and question answering. Here, we explore the hypothesis that linguistic deficits drive the error patterns of existing speaker commitment models by analyzing the linguistic correlates of model error on a challenging naturalistic dataset. We evaluate two state-of-the-art speaker commitment models on the CommitmentBank, an English dataset of naturally occurring discourses. The CommitmentBank is annotated with speaker commitment towards the content of the complement (“Florence is packed with visitors” in our example) of clause-embedding verbs (“know”, “think”) under four entailment-canceling environments (negation, modal, question, conditional). A breakdown of items by linguistic features reveals asymmetrical error patterns: while the models achieve good performance on some classes (e.g., negation), they fail to generalize to the diverse linguistic constructions (e.g., conditionals) in natural language, highlighting directions for improvement.