The Internal Validity of External Validity: Using Experiments to Validate Three Approaches to Extrapolating Causal Inferences Beyond the Cutoff in Regression Discontinuity

促进评估方法研究和创新(PRIME)计划旨在支持评估研究，特别强调：(1)探索确定STEM教育项目和方案的影响和有用性的创新方法；(2)建立和扩展评估STEM教育和劳动力发展倡议的理论基础，包括翻译和调整其他领域的方法；以及(3)提高评估领域的能力和基础设施。该计划将支持三种类型的提案：包括概念验证和可行性研究的探索性项目；更广泛的全面项目；以及研讨会和会议。拟议的研究仔细考虑了上文第1项。此外，传播计划还注意到项目3。STEM教育中的随机对照试验被认为是因果推断的黄金标准。通常，这些模式在实践中过于严格(出于道德和其他原因)。此外，随着研究的进展，许多这样的研究崩溃了。例如，高中领导层更迭，一所学校退出学习。当这种情况发生时，随机化过程就会受到影响，由此得出的推论也会受到影响。这项拟议的研究将评估使用回归间断设计(RDD)评估影响STEM结果的教育干预措施的新方法的有效性和稳健性。这项研究将系统地比较和对比推广RDD的新方法，评估它们产生偏离截止分数的无偏因果估计的情况。这项研究将使用来自过去教育干预的12个不同的数据集进行，这些数据集(A)最初通过RE进行评估，(B)将数学成绩作为结果衡量标准。使用研究内比较方法(Cook等人2008年)，这项研究将评估RE和一种给定的RDD方法在多大程度上为同一人群产生远离临界点的类似因果估计。分析具有STEM结果的多个数据集将提供有关新的RDD方法在不同类型的干预和学生群体中的稳健性的信息。

The Promoting Research and Innovation in Methodologies for Evaluation (PRIME) program seeks to support research on evaluation with special emphasis on: (1) exploring innovative approaches for determining the impacts and usefulness of STEM education projects and programs; (2) building on and expanding the theoretical foundations for evaluating STEM education and workforce development initiatives, including translating and adapting approaches from other fields; and (3) growing the capacity and infrastructure of the evaluation field. Three types of proposals will be supported by the program: Exploratory Projects that include proof-of-concept and feasibility studies; more extensive Full-Scale Projects; and workshops and conferences. The proposed research attends carefully to item 1 above. In addition, the dissemination plan also attends to item 3. Randomized control trials in STEM education are considered the gold standard for causal inference. Often these models are overly restrictive in practice (for ethical and other reasons). Moreover, many such studies collapse as the research progresses. For example, senior school leadership changes and a school is withdrawn from a study. When this occurs the randomizing process is compromised and so are the resulting inferences. The proposed research will assess the validity and robustness of new approaches for using the regression discontinuity design (RDD) to evaluate educational interventions affecting STEM outcomes. This study will systematically compare and contrast the new approaches for generalizing RDD, assessing how well they produce unbiased causal estimates away from the cutoff score. The research will be conducted using 12 heterogeneous datasets from past educational interventions that (a) were originally evaluated through an RE and (b) included math achievement as an outcome measure. Using the within-study comparison method (Cook et al. 2008), this study will assess the extent to which the RE and a given RDD approach produce similar causal estimates away from the cutoff for the same population. Analyzing multiple datasets with STEM outcomes will provide information about the robustness of the new RDD methods across different types of interventions and student populations.