An Interpretable Joint Graphical Model for Fact-Checking From Crowds
An Interpretable Joint Graphical Model for Fact-Checking From Crowds
复制标题
用于群体事实核查的可解释的联合图形模型
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Byron C. Wallace
中科院分区：
文献类型：
--
作者：
An T. Nguyen;Aditya Kharosekar;Matthew Lease;Byron C. Wallace
评估在互联网上提出的声明的真实性是一个重要的、具有挑战性的、及时的问题。虽然自动事实核查模型有可能帮助人们更好地评估他们读到的内容，但我们认为，此类模型必须是可解释、准确和快速的，才能在实践中有用；尽管预测准确性显然很重要，但模型透明度对于用户信任系统并将自己的知识与模型预测相结合至关重要。为了实现这一点，我们提出了一种新的概率图模型(PGM)，它结合了机器学习和人群标注。我们的模型中的节点对应于声明的真实性、关于声明的文章立场、新闻来源的声誉和注释器的可靠性。我们介绍了一种参数估计的快速变分方法。对两个真实世界数据集和三个场景的评估表明：(1)PGM中来源、索赔和人群注释器的联合建模提高了预测索赔准确性的预测性能和可解释性；(2)我们的变分推理方法实现了可伸缩的快速参数估计，与Gibbs抽样相比，性能仅略有下降。关于模型透明度，我们设计并部署了一个原型事实检查器Web工具，包括用于解释模型预测的可视界面。一项小型用户研究的结果表明，模型解释提高了用户对模型预测的满意度和信任度。我们分享我们的网络演示、模型源代码和我们收集的13K个众筹标签。
Assessing the veracity of claims made on the Internet is an important, challenging, and timely problem. While automated fact-checking models have potential to help people better assess what they read, we argue such models must be explainable, accurate, and fast to be useful in practice; while prediction accuracy is clearly important, model transparency is critical in order for users to trust the system and integrate their own knowledge with model predictions. To achieve this, we propose a novel probabilistic graphical model (PGM) which combines machine learning with crowd annotations. Nodes in our model correspond to claim veracity, article stance regarding claims, reputation of news sources, and annotator reliabilities. We introduce a fast variational method for parameter estimation. Evaluation across two real-world datasets and three scenarios shows that: (1) joint modeling of sources, claims and crowd annotators in a PGM improves the predictive performance and interpretability for predicting claim veracity; and (2) our variational inference method achieves scalably fast parameter estimation, with only modest degradation in performance compared to Gibbs sampling. Regarding model transparency, we designed and deployed a prototype fact-checker Web tool, including a visual interface for explaining model predictions. Results of a small user study indicate that model explanations improve user satisfaction and trust in model predictions. We share our web demo, model source code, and the 13K crowd labels we collected.