Using the Results from Rigorous Multisite Evaluations to Inform Local Policy Decisions
Using the Results from Rigorous Multisite Evaluations to Inform Local Policy Decisions
复制标题
DOI：
10.1002/pam.22154
复制
发表时间：
2019-09-01
影响因子：
3.8
通讯作者：
Stuart, Elizabeth A.
中科院分区：
文献类型：
--
作者：
Orr, Larry L.;Olsen, Robert B.;Stuart, Elizabeth A.
地方一级的循证政策要求预测干预措施的影响，以告知是否应采取干预措施。越来越多的地方决策者可以从国家研究信息交换所获得已发表的研究报告，这些研究报告评估了政策干预措施的有效性，这些研究信息交换所审查并传播了方案评估的证据。通过这些评估，地方决策者掌握了大量的证据，说明哪些措施有效，但不一定在哪里有效。多地点评估可能会对研究样本中干预措施的平均影响产生无偏估计，但对样本以外地区的影响仍然产生不准确的预测，原因有二：（1）干预措施的影响可能因地区而异，（2）评估估计会受到抽样误差的影响。不幸的是，有多少的政策干预措施的影响，从一个地方到另一个不同的证据相对较少，几乎没有证据表明这种变化的影响的准确性，采取干预措施的地方影响可以预测使用的结果，从其他地方的评价。在本文中，我们提出了一套方法，用于量化的准确性，可以使用多地点随机试验的结果，并评估的可能性，预测误差将导致错误的地方政策决策的本地预测。我们使用三种教育干预措施的评估来证明这些方法，提供了第一个经验证据，证明使用多地点评估来预测个别地区的影响，即，“循证政策”改善地方政策的能力。
Evidence-based policy at the local level requires predicting the impact of an intervention to inform whether it should be adopted. Increasingly, local policymakers have access to published research evaluating the effectiveness of policy interventions from national research clearinghouses that review and disseminate evidence from program evaluations. Through these evaluations, local policymakers have a wealth of evidence describing what works, but not necessarily where. Multisite evaluations may produce unbiased estimates of the average impact of an intervention in the study sample and still produce inaccurate predictions of the impact for localities outside the sample for two reasons: (1) the impact of the intervention may vary across localities, and (2) the evaluation estimate is subject to sampling error. Unfortunately, there is relatively little evidence on how much the impacts of policy interventions vary from one locality to another and almost no evidence on the implications of this variation for the accuracy with which the local impact of adopting an intervention can be predicted using findings from an evaluation in other localities. In this paper, we present a set of methods for quantifying the accuracy of the local predictions that can be obtained using the results of multisite randomized trials and for assessing the likelihood that prediction errors will lead to errors in local policy decisions. We demonstrate these methods using three evaluations of educational interventions, providing the first empirical evidence of the ability to use multisite evaluations to predict impacts in individual localities-i.e., the ability of "evidence-based policy" to improve local policy.