Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons
Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons
复制标题
DOI：
10.1002/pam.20375
复制
发表时间：
2008-09-01
影响因子：
3.8
通讯作者：
Wong, Vivian C.
中科院分区：
文献类型：
--
作者：
Cook, Thomas D.;Shadish, William R.;Wong, Vivian C.
本文分析了最近12项研究内比较，将随机实验的因果估计与来自同一治疗组的观察性研究的因果估计进行了对比。目的是测试当一个反事实组形成时，无论是否随机分配，以及何时，是否会产生不同的因果估计。在没有随机分配的组中对选择进行统计调整。我们确定了三个比较实验和回归不连续（RD）研究的研究。他们在RD截止点附近产生了相当可比的因果估计。我们确定了另外三个研究，其中准实验涉及在预测试上进行仔细的完整组匹配。尽管在这种情况下存在隐藏偏见的逻辑可能性，但所有三种情况都重现了他们的实验估计，特别是如果匹配是地理上的局部匹配。然后，我们确定了两项研究，其中治疗组和非随机对照组在前测时明显不同，但治疗的选择过程是完全或非常合理的。这里也重现了实验结果。剩下的两项研究在某些情况下得出了相应的实验和非实验结果，而在其他情况下则没有，而另外两项研究得出了不同的实验和非实验估计，尽管在每种情况下，观察性研究的设计和分析都很差。这些证据比过去的研究内部比较更有希望，其中大多数涉及工作培训。讨论了造成这种差异的原因。(C) 2008年，公共政策分析与管理协会。
This paper analyzes 12 recent within-study comparisons contrasting causal estimates from a randomized experiment with those from an observational study sharing the same treatment group. The aim is to test whether different causal estimates result when a counterfactual group is formed, either with or without random assignment, and when. statistical adjustments for selection are made in the group from which random assignment is absent. We identify three studies comparing experiments and regression-discontinuity (RD) studies. They produce quite comparable causal estimates at points around the RD cutoff We identify three other studies where the quasi-experiment involves careful intact group matching on the pretest. Despite the logical possibility of hidden bias in this instance, all three cases also reproduce their experimental estimates, especially if the match is geographically local. We then identify two studies where the treatment and nonrandomized comparison groups manifestly differ at pretest but where the selection process into treatment is completely or very plausibly known. Here too, experimental results are recreated. Two of the remaining studies result in correspondent experimental and nonexperimental results under some circumstances but not others, while two others produce different experimental and nonexperimental estimates, though in each case the observational study was poorly designed and analyzed. Such evidence is more promising than what was achieved in past within-study comparisons, most involving job training. Reasons for this difference are discussed. (C) 2008 by the Association for Public Policy Analysis and Management.