Optimizing the selection and training of raters: How do rater characteristics and training features affect the quality of judgments of teaching quality?

课堂观察经常用于教师评估、培训、专业发展和研究。因此，确保观察结果是可靠和有效的是非常重要的。然而，研究表明，即使在评分员接受过培训的情况下，他们往往都不是。关于如何提高这种评级的可靠性和有效性，系统和因果证据有限。一个很有前途的研究途径是考察评分员的特点和他们的培训情况。本项目的目的是系统地调查评分员的基本特征和培训的各个方面，以及它们对教学质量判断质量(即信度和效度)的影响。我们的目标是在考虑到可行性问题的情况下，为如何优化评分员的选择和培训提供证据。该项目关注教学质量的三个基本维度(Praetorius，Klieme，Herbert，&Pinger，2018)：课堂管理，学生支持，和认知激活。我们将调查以下观察对教学质量的相关性：(RQ 1)评分员的内容相关特征(与所观察课程的内容及其学习专业相关的内容知识和教学内容知识)及其一般特征(个性、教学知识、信念和教学经验)和(RQ 2)基于不同评分员池中典型使用的评分员数量(n=2-4)的特定评分员星座，(RQ 3)对评分员的内容知识和关于所观察课程主题的教学内容知识的实验操作，(RQ 4)评分员培训练习阶段(引导性讨论与自我反思)的实验操作，以及(RQ 5)培训持续时间(12小时与6小时)的实验操作。我们将通过选择具有不同教育成就的参与者，他们在不同的轨道上，并专门从事各种科目，最大限度地提高项目评分员队伍的特点。我们还将利用德国和瑞士独特的不同教育环境。将被评分的课程包括两个年级水平和两个核心科目，科学(关注漂浮和下沉，3-4年级)和数学(关注毕达哥拉斯定理，8-9年级)。RQ1和RQ2将在四个实验中使用对照组进行研究。RQ3-5是使用每个感兴趣的领域的实验操作来调查的。主要的因变量是评分员之间的可靠性，评分员与参考评分的一致性，以及关于学生概念理解和具体内容兴趣的预测有效性。对于RQ 3和RQ 4，我们将对子样本进行有声思维和认知访谈，以进一步了解实验条件之间的差异。

Classroom observations are frequently used in teacher evaluation, training, professional development, and research. Ensuring that the observations are reliable and valid is therefore of great importance. Studies have shown however, that they are often neither, even when the raters are trained. There is limited systematic and causal evidence about ways of enhancing the reliability and validity of such ratings. One promising avenue of research is looking at the characteristics of the raters and how well they are trained.The aim of this project is to systematically investigate fundamental rater characteristics and aspects of rater training and their effect on the quality (i.e., reliability and validity) of judgments of teaching quality. We aim to generate evidence for how the selection and training of raters can be optimized, taking into account feasibility issues. The project focuses on the Three Basic Dimensions of teaching quality (Praetorius, Klieme, Herbert, & Pinger, 2018): classroom management, student support, and cognitive activation. We will investigate the relevance of the following aspects of observations on teaching quality: (RQ 1) the raters’ content-related characteristics (content knowledge and pedagogical content knowledge with respect to the content of the observed lessons, their study major) and their generic characteristics (personality, pedagogical knowledge, beliefs, and teaching experience) and (RQ 2) specific rater constellations based on typically used numbers of raters (n = 2-4) in a heterogeneous pool of raters, (RQ 3) experimental manipulations of raters’ content knowledge and pedagogical content knowledge of the topic of the observed lessons, (RQ 4) experimental manipulations of the practice phase (guided discussion vs. self-directed reflection) of rater training, and (RQ 5) experimental manipulations of training duration (12h versus 6h). We will maximize the variance of the characteristics of the project’s rater pool by selecting participants with a range of educational attainments, who are on different tracks, and specializing in a variety of subjects. We will also take advantage of the uniquely heterogeneous nature of education settings in Germany and Switzerland. The lessons that will be rated comprise two grade levels and two core subjects, science (focus on floating and sinking, grades 3-4) and mathematics (focus on the Pythagorean theorem, grades 8-9). RQ 1 and RQ 2 will be investigated using the control groups in four experiments. RQ 3-5 are investigated using experimental manipulations of each of the areas of interest. The central dependent variables are inter-rater reliability, rater agreement with reference ratings, and predictive validity with respect to students’ conceptual understanding as well as content-specific interest. For RQ 3 and RQ 4, think-alouds and cognitive interviews with subsamples will be conducted to gain further insight into the differences between the experimental conditions.