Fool Me Twice: Entailment from Wikipedia Gamification
Fool Me Twice: Entailment from Wikipedia Gamification
复制标题
DOI：
10.18653/v1/2021.naacl-main.32
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Julian Martin Eisenschlos;Bhuwan Dhingra;Jannis Bulian;Benjamin Borschinger;Jordan L. Boyd-Graber
中科院分区：
文献类型：
--
作者：
Julian Martin Eisenschlos;Bhuwan Dhingra;Jannis Bulian;Benjamin Borschinger;Jordan L. Boyd-Graber
我们发布了FoolMeTwice（简称FM2），这是一个通过有趣的多人游戏收集的具有挑战性的蕴涵对的大型数据集。游戏化鼓励对抗性的例子，与其他流行的蕴涵数据集相比，大大减少了可以使用“快捷方式”解决的例子的数量。玩家有两个任务。第一个任务要求玩家根据维基百科页面上的证据写一个合理的声明。第二个展示了其他参与者写的两个看似合理的声明，其中一个是错误的，目标是在时间耗尽之前识别它。玩家“付费”来查看从证据池中检索到的线索：玩家需要的证据越多，索赔就越难。有动机的玩家之间的游戏导致不同的策略来制作索赔，如时间推断和转移到无关的证据，并导致更高质量的数据的蕴涵和证据检索任务。我们开源了数据集和游戏代码。
We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, drastically lowering the number of examples that can be solved using “shortcuts” compared to other popular entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The second one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players “pay” to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play between motivated players leads to diverse strategies for crafting claims, such as temporal inference and diverting to unrelated evidence, and results in higher quality data for the entailment and evidence retrieval tasks. We open source the dataset and the game code.