Optimizing the selection and training of raters: How do rater characteristics and training features affect the quality of judgments of teaching quality?

优化评估者的选择和培训：评估者特征和培训特征如何影响教学质量的判断质量？

• 批准号: 490658944
• 负责人: Professor Dr. Aiso Heinze
• 金额: --
• 依托单位: IPN - Leibniz-Institut für die Pädagogik der Naturwissenschaften und Mathematik
• 依托单位国家: 德国
• 项目类别: Research Grants
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/490658944?language=en
• 关键词: Optimizing; selection; training; raters; How

Classroom observations are frequently used in teacher evaluation, training, professional development, and research. Ensuring that the observations are reliable and valid is therefore of great importance. Studies have shown however, that they are often neither, even when the raters are trained. There is limited systematic and causal evidence about ways of enhancing the reliability and validity of such ratings. One promising avenue of research is looking at the characteristics of the raters and how well they are trained.The aim of this project is to systematically investigate fundamental rater characteristics and aspects of rater training and their effect on the quality (i.e., reliability and validity) of judgments of teaching quality. We aim to generate evidence for how the selection and training of raters can be optimized, taking into account feasibility issues. The project focuses on the Three Basic Dimensions of teaching quality (Praetorius, Klieme, Herbert, & Pinger, 2018): classroom management, student support, and cognitive activation. We will investigate the relevance of the following aspects of observations on teaching quality: (RQ 1) the raters’ content-related characteristics (content knowledge and pedagogical content knowledge with respect to the content of the observed lessons, their study major) and their generic characteristics (personality, pedagogical knowledge, beliefs, and teaching experience) and (RQ 2) specific rater constellations based on typically used numbers of raters (n = 2-4) in a heterogeneous pool of raters, (RQ 3) experimental manipulations of raters’ content knowledge and pedagogical content knowledge of the topic of the observed lessons, (RQ 4) experimental manipulations of the practice phase (guided discussion vs. self-directed reflection) of rater training, and (RQ 5) experimental manipulations of training duration (12h versus 6h). We will maximize the variance of the characteristics of the project’s rater pool by selecting participants with a range of educational attainments, who are on different tracks, and specializing in a variety of subjects. We will also take advantage of the uniquely heterogeneous nature of education settings in Germany and Switzerland. The lessons that will be rated comprise two grade levels and two core subjects, science (focus on floating and sinking, grades 3-4) and mathematics (focus on the Pythagorean theorem, grades 8-9). RQ 1 and RQ 2 will be investigated using the control groups in four experiments. RQ 3-5 are investigated using experimental manipulations of each of the areas of interest. The central dependent variables are inter-rater reliability, rater agreement with reference ratings, and predictive validity with respect to students’ conceptual understanding as well as content-specific interest. For RQ 3 and RQ 4, think-alouds and cognitive interviews with subsamples will be conducted to gain further insight into the differences between the experimental conditions.

课堂观察经常用于教师评估、培训、专业发展和研究。因此，确保观察结果的可靠性和有效性非常重要。然而研究表明，即使评估者接受过培训，他们往往两者都不是。关于提高此类评级的可靠性和有效性的方法，系统性和因果性证据有限。一种有前途的研究途径是研究评估者的特征以及他们的培训程度。该项目的目的是系统地研究评估者的基本特征和评估者培训的方面及其对教学质量判断质量（即信度和效度）的影响。我们的目标是在考虑可行性问题的情况下，为如何优化评估者的选择和培训提供证据。该项目重点关注教学质量的三个基本维度（Praetorius、Klieme、Herbert 和 Pinger，2018）：课堂管理、学生支持和认知激活。我们将调查教学质量观察的以下方面的相关性：（RQ 1）评估者的内容相关特征（与所观察的课程内容、其学习专业相关的内容知识和教学内容知识）及其一般特征（个性、教学知识、信仰和教学经验）和（RQ 2）基于异构中通常使用的评估者数量（n = 2-4）的特定评估者星座 评估者群体，(RQ 3) 对评估者的内容知识和观察课程主题的教学内容知识的实验操作，(RQ 4) 对评估者培训的实践阶段（引导讨论与自我导向反思）的实验操作，以及 (RQ 5) 对培训持续时间的实验操作（12 小时与 6 小时）。我们将通过选择具有不同教育程度、处于不同轨道、专注于不同学科的参与者来最大化项目评估者库特征的差异。我们还将利用德国和瑞士教育环境独特的异构性。评分的课程包括两个年级和两个核心科目：科学（重点关注漂浮和下沉，3-4 年级）和数学（重点关注毕达哥拉斯定理，8-9 年级）。 RQ 1 和 RQ 2 将在四个实验中使用对照组进行研究。 RQ 3-5 通过对每个感兴趣领域的实验操作进行研究。中心因变量是评估者间的可靠性、评估者与参考评级的一致性以及对学生概念理解和特定内容兴趣的预测有效性。对于 RQ 3 和 RQ 4，将对子样本进行有声思考和认知访谈，以进一步了解实验条件之间的差异。