Are Red Roses Red? Evaluating Consistency of Question-Answering Models
Are Red Roses Red? Evaluating Consistency of Question-Answering Models
复制标题
DOI：
10.18653/v1/p19-1621
复制
发表时间：
2019-07
期刊：
影响因子：
--
通讯作者：
Marco Tulio Ribeiro;Carlos Guestrin;Sameer Singh
中科院分区：
文献类型：
--
作者：
Marco Tulio Ribeiro;Carlos Guestrin;Sameer Singh
尽管当前的问题 - 避开系统的评估是孤立的，但我们需要考虑预测之间的关系以衡量真正的理解。 “玫瑰是什么颜色？”。人类评估表明，这些产生的含义是良好的，并且一致性评估为现有模型的差距提供了关键的见解，同时用含义的agement数据进行了重新培训，可以提高对合成和人类生成的含义的一致性。
Although current evaluation of question-answering systems treats predictions in isolation, we need to consider the relationship between predictions to measure true understanding. A model should be penalized for answering “no” to “Is the rose red?” if it answers “red” to “What color is the rose?”. We propose a method to automatically extract such implications for instances from two QA datasets, VQA and SQuAD, which we then use to evaluate the consistency of models. Human evaluation shows these generated implications are well formed and valid. Consistency evaluation provides crucial insights into gaps in existing models, while retraining with implication-augmented data improves consistency on both synthetic and human-generated implications.