Evidence-based Fact-Checking of Health-related Claims
Evidence-based Fact-Checking of Health-related Claims
复制标题
对健康相关声明进行基于证据的事实核查
DOI：
10.18653/v1/2021.findings-emnlp.297
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Dina Demner
中科院分区：
文献类型：
--
作者：
Mourad Sarrouti;Asma Ben Abacha;Yassine Mrabet;Dina Demner
验证文本文件中声明的真实性或事实核查的任务近年来受到了极大的关注。许多现有的基于证据的事实检查数据集包含合成声明，并且在这些数据上训练的模型可能无法验证真实世界的声明。特别是，很少有研究涉及对需要医学专业知识或科学文献证据的健康相关声明进行循证事实核查。在本文中，我们介绍了H EALTH V ER，这是一个新的数据集，用于对健康相关声明进行基于证据的事实检查，可以通过评估科学文章的真实性来研究现实世界声明的有效性。我们使用三步数据创建方法，首先从搜索引擎返回的有关COVID-19问题的片段中检索真实世界的声明。然后，我们使用基于T5相关性的模型自动检索并重新排名相关的科学论文。最后，每一项证据陈述与相关索赔之间的关系被手工标注为S UPPORT、R EFUTE和N EUTRAL。为了验证创建的14，330个证据-声明对的数据集，我们基于预训练的语言模型开发了基线模型。我们的实验表明，与在合成和开放域索赔上训练的模型相比，在真实世界的医疗索赔上训练深度学习模型大大提高了性能。我们的结果和手动分析表明，HEALTH V ER为未来对健康相关索赔进行循证事实核查的工作提供了一个现实而具有挑战性的数据集。数据集、源代码和
The task of verifying the truthfulness of claims in textual documents, or fact-checking, has received significant attention in recent years. Many existing evidence-based fact-checking datasets contain synthetic claims and the models trained on these data might not be able to verify real-world claims. Partic-ularly few studies addressed evidence-based fact-checking of health-related claims that re-quire medical expertise or evidence from the scientific literature. In this paper, we introduce H EALTH V ER , a new dataset for evidence-based fact-checking of health-related claims that allows to study the validity of real-world claims by evaluating their truthfulness against scientific articles. Using a three-step data creation method, we first retrieved real-world claims from snippets returned by a search engine for questions about COVID-19. Then we automatically retrieved and re-ranked relevant scientific papers using a T5 relevance-based model. Finally, the relations between each evidence statement and the associated claim were manually annotated as S UPPORT , R EFUTE and N EUTRAL . To validate the created dataset of 14,330 evidence-claim pairs, we developed baseline models based on pretrained language models. Our experiments showed that training deep learning models on real-world medical claims greatly improves performance compared to models trained on synthetic and open-domain claims. Our results and manual analysis suggest that H EALTH V ER provides a realistic and challenging dataset for future efforts on evidence-based fact-checking of health-related claims. The dataset, source code, and
DOI：
10.18653/v1/k19-1046
发表时间：
2019-08
期刊：
ArXiv
影响因子：
--
作者：
Andreas Hanselowski;Christian Stab;Claudia Schulz;Zile Li;Iryna Gurevych
通讯作者：
Andreas Hanselowski;Christian Stab;Claudia Schulz;Zile Li;Iryna Gurevych
影响因子：
--
作者：
Neema Kotonya;Francesca Toni
通讯作者：
Neema Kotonya;Francesca Toni