Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference
Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference
复制标题
DOI：
10.1609/icwsm.v16i1.19362
复制
发表时间：
2020-09
期刊：
影响因子：
--
通讯作者：
Galen Cassebeer Weld;Peter West;M. Glenski;D. Arbour;Ryan A. Rossi;Tim Althoff
中科院分区：
文献类型：
--
作者：
Galen Cassebeer Weld;Peter West;M. Glenski;D. Arbour;Ryan A. Rossi;Tim Althoff
使用文本社交媒体数据的因果推理研究可以为人类行为提供可操作的见解。用文本进行准确的因果推断需要控制混杂，否则可能会产生偏见。最近，已经提出了许多不同的方法来调整混杂因素，我们表明，这些现有的方法不同意彼此的两个数据集的灵感来自以前的社交媒体研究。评估因果方法是具有挑战性的，因为地面真相反事实几乎是不可用的。目前，使用文本的因果方法没有经验评估框架，因此，从业者必须在没有指导的情况下选择他们的方法。我们贡献了第一个这样的框架，其中包括五个任务来自真实的世界的研究。我们的框架使任何因果推理方法使用文本的评价。通过648个实验和两个数据集，我们评估了每一种常用的因果推理方法，并确定了它们的优缺点，以告知寻求使用这些方法的社交媒体研究人员，并指导未来的改进。我们公开所有任务、数据和模型，以告知应用程序并鼓励更多的研究。
Causal inference studies using textual social media data can provide actionable insights on human behavior. Making accurate causal inferences with text requires controlling for confounding which could otherwise impart bias. Recently, many different methods for adjusting for confounders have been proposed, and we show that these existing methods disagree with one another on two datasets inspired by previous social media studies. Evaluating causal methods is challenging, as ground truth counterfactuals are almost never available. Presently, no empirical evaluation framework for causal methods using text exists, and as such, practitioners must select their methods without guidance. We contribute the first such framework, which consists of five tasks drawn from real world studies. Our framework enables the evaluation of any casual inference method using text. Across 648 experiments and two datasets, we evaluate every commonly used causal inference method and identify their strengths and weaknesses to inform social media researchers seeking to use such methods, and guide future improvements. We make all tasks, data, and models public to inform applications and encourage additional research.