Methods for Investigating Causal Mechanisms in Multi-Site Experimental and Quasi-Experimental Studies

该研究项目将开发新的策略，用于调查干预计划的多地点实验和准实验评估中的因果机制。过去的研究在广泛的领域往往有相当大的跨站点异质性的平均程序的效果。然而，方案评估人员还未能充分利用多地点数据。在这个项目中开发的分析策略将适用于测试有关干预的中间过程的科学理论，并评估该计划机制是否以及如何在不同环境中运作。新的策略将允许应用研究人员灵活地提出一组新的实证问题，这些问题对于测试干预理论在不同环境中的普遍性和解开复杂的调解机制至关重要，从而有可能重新审视干预理论，并建议对干预实践进行具体的现场修改。该项目的动机是应用实例，包括国家就业团队研究，该研究评估旨在促进弱势青年经济独立的方案;开端影响研究，该研究评估针对贫困儿童的联邦幼儿教育方案;以及芝加哥公立学校的数学课程改革，旨在改善成绩差的九年级学生的数学学习。该研究小组将提供开源R包和Stata ado文件沿着用户手册和教学数据示例供传播。本研究建立在社会科学中介分析的传统和因果中介分析的统计理论和方法的最新发展之上。现有的方法，在一般情况下，需要分析师指定中介模型和结果模型，涉及比较强的模型为基础的假设。评估站点间的异质性的因果机制是具有挑战性的，特别是当许多基于模型的假设是不切实际的。研究小组已作出初步尝试，将倾向分数为基础的加权方法，简单的调解机制的多站点分析。这种加权方法不需要结果模型规格。然而，最新的分析程序仅限于每个站点相对较大样本量的实验。为了克服现有的挑战，该项目将推进因果参数估计，倾向评分估计和敏感性分析的方法发展。研究人员将开发一种新的伪结果随机效应策略，用于通过矩法分析因果参数估计，以避免基于模型的假设。通过汇集所有地点的数据，分析将不受每个地点小样本量的限制，并将灵活地调查复杂的调解机制。将获得考虑倾向评分估计不确定性的渐近方差。为了增强对倾向评分模型错误指定的稳健性，将纳入协变量平衡倾向评分估计方法。研究人员还将开发一种新的基于权重的方法，用于遗漏治疗后和治疗前混杂因素的敏感性分析。这些新的策略将从简单的机制扩展到复杂的机制，从实验到准实验的多站点数据。

This research project will develop new strategies for investigating causal mechanisms in multi-site experimental and quasi-experimental evaluations of intervention programs. Past research in a wide range of fields often has reported considerable cross-site heterogeneity in the average program effect. However, program evaluators have not been able to take full advantage of multi-site data. The analytic strategies to be developed in this project will be suitable for testing scientific theories about the intermediate process of an intervention and for assessing whether and how the program mechanism operates differently across settings. The new strategies will allow applied researchers to flexibly ask a new set of empirical questions crucial for testing the generalizability of an intervention theory across settings and for unpacking complex mediation mechanisms, making it possible to revisit the intervention theory and to suggest specific site-level modifications of the intervention practice. This project is motivated by application examples, including the National Job Corps Study, which evaluates a program designed to promote economic independence of disadvantaged youths; the Head Start Impact Study, which evaluates the federal early childhood education program for children living in poverty; and math curricular reforms in Chicago Public Schools aimed at improving the math learning of low-achieving ninth graders. The research team will provide open-source R packages and Stata ado files along with user manuals and pedagogical data examples for dissemination.This research is built on the tradition of mediation analysis in the social sciences and on the recent developments in statistical theories and methods for causal mediation analysis. The existing methods, in general, require the analyst to specify both mediator models and outcome models involving comparatively strong model-based assumptions. Assessing between-site heterogeneity in causal mechanisms is challenging especially when many of the model-based assumptions are impractical. The research team has made initial attempts to extend a propensity score-based weighting method to multisite analysis of simple mediation mechanisms. This weighting method does not require outcome model specifications. The latest analytic procedure, however, is limited to experiments with a relatively large sample size per site. To overcome the existing challenges, this project will advance methodological developments in causal parameter estimation, propensity score estimation, and sensitivity analysis. The investigators will develop a novel pseudo-outcome random-effects strategy for causal parameter estimation analyzed through method-of-moments to avoid model-based assumptions. By pooling data from all the sites, the analysis will be unconstrained by a small sample size per site and will be flexible for investigating complex mediation mechanisms. Asymptotic variances will be obtained that take into account uncertainty in propensity score estimation. For enhancing robustness to propensity score model misspecification, a covariate-balancing propensity score estimation approach will be incorporated. The investigators also will develop a new weighting-based approach to sensitivity analysis for omitted post-treatment as well as pre-treatment confounders. These new strategies will be extended from simple to complex mechanisms and from experimental to quasi-experimental multi-site data.