Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering
Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering
复制标题
DOI：
10.1162/tacl_a_00279
复制
发表时间：
2018-09
影响因子：
10.9
通讯作者：
Eric Wallace;Pedro Rodriguez;Shi Feng;Ikuya Yamada;Jordan L. Boyd-Graber
中科院分区：
文献类型：
--
作者：
Eric Wallace;Pedro Rodriguez;Shi Feng;Ikuya Yamada;Jordan L. Boyd-Graber
对抗性评估压力测试模型对自然语言的理解。由于过去的方法暴露了肤浅的模式，因此产生的对抗性例子在复杂性和多样性方面是有限的。我们提出了人在环中对抗生成，其中人类作者被引导打破模型。我们通过交互式用户界面帮助作者解释模型预测。我们将这个生成框架应用于一个名为QuizBowl的问题回答任务，在这个任务中，琐事爱好者会提出对抗性的问题。由此产生的问题通过实况人机匹配得到验证：尽管这些问题对人类来说很普通，但它们系统性地阻碍了神经和信息检索模型。对抗性问题涵盖了从多跳推理到实体类型干扰的各种现象，暴露了健壮问题回答中的开放挑战。
Adversarial evaluation stress-tests a model’s understanding of natural language. Because past approaches expose superficial patterns, the resulting adversarial examples are limited in complexity and diversity. We propose human- in-the-loop adversarial generation, where human authors are guided to break models. We aid the authors with interpretations of model predictions through an interactive user interface. We apply this generation framework to a question answering task called Quizbowl, where trivia enthusiasts craft adversarial questions. The resulting questions are validated via live human–computer matches: Although the questions appear ordinary to humans, they systematically stump neural and information retrieval models. The adversarial questions cover diverse phenomena from multi-hop reasoning to entity type distractors, exposing open challenges in robust question answering.