The Internal Validity of External Validity: Using Experiments to Validate Three Approaches to Extrapolating Causal Inferences Beyond the Cutoff in Regression Discontinuity

促进评估方法的研究和创新（PRIME）计划旨在支持评估研究，特别强调：(1)探索确定STEM教育项目和计划的影响和有用性的创新方法；(2)建立和扩展评估STEM教育和劳动力发展倡议的理论基础，包括翻译和适应其他领域的方法；(3)发展评价领域的能力和基础设施。该计划将支持三种类型的提案：探索性项目，包括概念验证和可行性研究；更广泛的全面项目；还有研讨会和会议。拟议的研究仔细考虑了上述第1项。此外，传播计划还涉及项目3。STEM教育中的随机对照试验被认为是因果推理的黄金标准。通常这些模型在实践中过于严格（出于道德和其他原因）。而且，随着研究的深入，很多这样的研究都失败了。例如，高中领导层发生变化，学校退出了一项研究。当这种情况发生时，随机化过程受到损害，由此产生的推论也受到损害。本研究将评估使用回归不连续设计（RDD）来评估影响STEM结果的教育干预的新方法的有效性和稳健性。本研究将系统地比较和对比推广RDD的新方法，评估它们产生偏离临界值的无偏因果估计的程度。该研究将使用来自过去教育干预的12个异构数据集进行，这些数据集(a)最初通过RE进行评估，(b)将数学成绩作为结果衡量标准。使用研究内比较方法（Cook et al. 2008），本研究将评估RE和给定RDD方法在同一人群中产生相似因果估计的程度。用STEM结果分析多个数据集，将提供关于新RDD方法在不同类型干预措施和学生群体中的稳健性的信息。

The Promoting Research and Innovation in Methodologies for Evaluation (PRIME) program seeks to support research on evaluation with special emphasis on: (1) exploring innovative approaches for determining the impacts and usefulness of STEM education projects and programs; (2) building on and expanding the theoretical foundations for evaluating STEM education and workforce development initiatives, including translating and adapting approaches from other fields; and (3) growing the capacity and infrastructure of the evaluation field. Three types of proposals will be supported by the program: Exploratory Projects that include proof-of-concept and feasibility studies; more extensive Full-Scale Projects; and workshops and conferences. The proposed research attends carefully to item 1 above. In addition, the dissemination plan also attends to item 3. Randomized control trials in STEM education are considered the gold standard for causal inference. Often these models are overly restrictive in practice (for ethical and other reasons). Moreover, many such studies collapse as the research progresses. For example, senior school leadership changes and a school is withdrawn from a study. When this occurs the randomizing process is compromised and so are the resulting inferences. The proposed research will assess the validity and robustness of new approaches for using the regression discontinuity design (RDD) to evaluate educational interventions affecting STEM outcomes. This study will systematically compare and contrast the new approaches for generalizing RDD, assessing how well they produce unbiased causal estimates away from the cutoff score. The research will be conducted using 12 heterogeneous datasets from past educational interventions that (a) were originally evaluated through an RE and (b) included math achievement as an outcome measure. Using the within-study comparison method (Cook et al. 2008), this study will assess the extent to which the RE and a given RDD approach produce similar causal estimates away from the cutoff for the same population. Analyzing multiple datasets with STEM outcomes will provide information about the robustness of the new RDD methods across different types of interventions and student populations.