Methods for Investigating Causal Mechanisms in Multi-Site Experimental and Quasi-Experimental Studies

本研究项目将为研究干预方案的多地点实验和准实验评估中的因果机制开发新的策略。过去在广泛领域的研究经常报告了平均程序效应的相当大的跨站点异质性。然而，项目评估者还不能充分利用多地点的数据。在这个项目中开发的分析策略将适用于测试关于干预中间过程的科学理论，以及评估项目机制在不同环境下是否以及如何不同地运作。新的策略将允许应用研究人员灵活地提出一套新的经验问题，这些问题对于测试跨环境干预理论的普遍性和解开复杂的中介机制至关重要，从而使重新审视干预理论并提出干预实践的具体地点级别修改建议成为可能。这个项目是由应用实例推动的，包括国家就业团研究，该研究评估了一个旨在促进弱势青年经济独立的项目；“开端影响研究”（Head Start Impact Study），评估针对贫困儿童的联邦早期儿童教育项目；以及芝加哥公立学校的数学课程改革，旨在提高成绩较差的九年级学生的数学学习。研究小组将提供开源R软件包和Stata ado文件，以及用户手册和教学数据示例，以供传播。本研究是建立在社会科学中中介分析的传统和因果中介分析的统计理论和方法的最新发展。一般来说，现有的方法要求分析人员指定中介模型和结果模型，这些模型涉及相对较强的基于模型的假设。评估因果机制的站点间异质性具有挑战性，特别是当许多基于模型的假设不切实际时。研究小组已经初步尝试将基于倾向得分的加权方法扩展到简单中介机制的多地点分析。这种加权方法不需要结果模型规范。然而，最新的分析程序仅限于每个站点相对较大样本量的实验。为了克服现有的挑战，本项目将推进因果参数估计、倾向得分估计和敏感性分析的方法学发展。研究人员将开发一种新的伪结果随机效应策略，用于通过矩量法分析因果参数估计，以避免基于模型的假设。通过汇集来自所有站点的数据，分析将不受每个站点小样本量的限制，并且可以灵活地调查复杂的中介机制。将得到考虑不确定性的渐近方差。为了提高对倾向得分模型错配的稳健性，将采用协变量平衡倾向得分估计方法。研究人员还将开发一种新的基于权重的方法来对省略的后处理和预处理混杂因素进行敏感性分析。这些新策略将从简单的机制扩展到复杂的机制，从实验到准实验的多站点数据。

This research project will develop new strategies for investigating causal mechanisms in multi-site experimental and quasi-experimental evaluations of intervention programs. Past research in a wide range of fields often has reported considerable cross-site heterogeneity in the average program effect. However, program evaluators have not been able to take full advantage of multi-site data. The analytic strategies to be developed in this project will be suitable for testing scientific theories about the intermediate process of an intervention and for assessing whether and how the program mechanism operates differently across settings. The new strategies will allow applied researchers to flexibly ask a new set of empirical questions crucial for testing the generalizability of an intervention theory across settings and for unpacking complex mediation mechanisms, making it possible to revisit the intervention theory and to suggest specific site-level modifications of the intervention practice. This project is motivated by application examples, including the National Job Corps Study, which evaluates a program designed to promote economic independence of disadvantaged youths; the Head Start Impact Study, which evaluates the federal early childhood education program for children living in poverty; and math curricular reforms in Chicago Public Schools aimed at improving the math learning of low-achieving ninth graders. The research team will provide open-source R packages and Stata ado files along with user manuals and pedagogical data examples for dissemination.This research is built on the tradition of mediation analysis in the social sciences and on the recent developments in statistical theories and methods for causal mediation analysis. The existing methods, in general, require the analyst to specify both mediator models and outcome models involving comparatively strong model-based assumptions. Assessing between-site heterogeneity in causal mechanisms is challenging especially when many of the model-based assumptions are impractical. The research team has made initial attempts to extend a propensity score-based weighting method to multisite analysis of simple mediation mechanisms. This weighting method does not require outcome model specifications. The latest analytic procedure, however, is limited to experiments with a relatively large sample size per site. To overcome the existing challenges, this project will advance methodological developments in causal parameter estimation, propensity score estimation, and sensitivity analysis. The investigators will develop a novel pseudo-outcome random-effects strategy for causal parameter estimation analyzed through method-of-moments to avoid model-based assumptions. By pooling data from all the sites, the analysis will be unconstrained by a small sample size per site and will be flexible for investigating complex mediation mechanisms. Asymptotic variances will be obtained that take into account uncertainty in propensity score estimation. For enhancing robustness to propensity score model misspecification, a covariate-balancing propensity score estimation approach will be incorporated. The investigators also will develop a new weighting-based approach to sensitivity analysis for omitted post-treatment as well as pre-treatment confounders. These new strategies will be extended from simple to complex mechanisms and from experimental to quasi-experimental multi-site data.