Explaining Inconsistent Effects of Teaching Quality on Educational Outcomes

教育有效性研究的重点是找出促进学生学习成果的教学方面。最终目标是告知政策制定者和实践者有关教学实践和教育倡议，以提高学生的学习。在研究领域中存在各种元分析。他们表明，在调查相同实质性研究问题的研究中，效应大小存在很大差异，这意味着复制失败。不一致的，甚至部分矛盾的结果阻碍了直接的建议，并引起了对研究结果可信度的怀疑。产生偏差的原因可能是研究在几个方面有所不同：(a)环境，(b)设计，(c)结构的操作化，以及(d)统计分析中使用的方法。荟萃分析和回顾显示，其中一些方面影响了教学质量对学生成绩的估计效果。然而，由于荟萃分析和综述收集的信息来自已经存在的研究，他们无法保持某些方面，如统计模型恒定，因此无法理清造成异质效应的不同方面。通过对现有数据进行系统的重新分析，我们的两个主要目标是：(1)确定并精确定义可能解释教学质量影响差异的可能因素，以及(2)提供这些因素对结果异质性的相对重要性的估计。为了实现第一个研究目标，我们将根据(a)设计和设置以及(b)构造的操作化方面的差异对数据集进行分类。我们将进一步定义可能影响估计效果的方法学因素。为了实现第二个研究目标，我们将协调数据集，使用不同的方法方法进行二次分析，从单个分析中提取估计的影响，并将其转换到相同的尺度上（以效应大小的形式），并估计哪些方面解释了影响中方差的比例。最后，这些发现有助于规划未来的概括性分析和概念复制，从而为从业者提供有意义的建议。

The focus of educational effectiveness research is to identify aspects of teaching that foster student learning outcomes. The ultimate goal is to inform policy makers and practitioners in the field about teaching practices and educational initiatives that improve student learning. Various meta-analyses within the research field exist. They show large deviations in effect sizes across studies that investigated the same substantial research question, which implies replication failure. Inconsistent and, in part, even contradictory results hinder straightforward recommendations and raise doubts about the credibility of the research results. Reasons for deviations could be that the studies vary with respect to several aspects: (a) the setting, (b) the design, (c) the operationalization of the constructs, and (d) the methodology used in the statistical analyses. Meta-analyses and reviews showed that some of those aspects influenced the estimated effect of teaching quality on student outcome. Since the meta-analyses and reviews gathered information from already existing studies, however, they were unable to keep certain aspects such as the statistical model constant, and were thus unable to disentangle the different aspects that are responsible for heterogeneous effects. Conducting a systematic re-analysis of already existing data, our two main aims are (1) to identify and precisely define possible factors that might explain variance in effects of teaching quality, and (2) to provide estimates of the relative importance of those factors for heterogeneity of results. In pursuit of the first research objective, we will categorize data sets with respect to differences in (a) the design and setting and (b) the operationalization of constructs. We will further define methodological factors that might influence the estimated effect. To achieve the second research objective, we will harmonize the data sets, run the secondary analyses using different methodological approaches, extract the estimated effects from the individual analyses and transform them onto the same scale (in the form of an effect size), and estimate which aspects explain which proportion of variance in the effects. In the end, these findings aid in the planning of future generalizability analyses and conceptual replications that result in meaningful recommendations for practitioners.