Optimizing the selection and training of raters: How do rater characteristics and training features affect the quality of judgments of teaching quality?

课堂观察经常用于教师评价、培训、专业发展和研究。因此，确保观察结果的可靠性和有效性至关重要。然而，研究表明，他们往往两者都不是，即使评级员受过培训。关于如何提高这种评级的可靠性和有效性，系统和因果关系方面的证据有限。一个很有前途的研究途径是研究评分员的特征以及他们的培训情况。本项目的目的是系统地调查评分员的基本特征和评分员培训的各个方面及其对质量的影响（即，教学质量评价的信度和效度。我们的目标是为如何优化评分员的选择和培训提供证据，同时考虑到可行性问题。该项目的重点是教学质量的三个基本维度（Praetorius，Klieme，赫伯特和Pinger，2018）：课堂管理，学生支持和认知激活。我们将调查以下方面的观察结果与教学质量的相关性：（RQ 1）评分者的内容相关特征（关于观察课程内容的内容知识和教学内容知识，他们的学习专业）及其一般特征（个性、教育知识、信念、和教学经验）和（RQ 2）基于异质评级者池中通常使用的评级者数量（n = 2-4）的特定评级者星座，（RQ 3）评分员对观察课主题的内容知识和教学内容知识的实验操作，（RQ 4）练习阶段的实验操作（引导讨论与自我导向的反思），以及（RQ 5）训练持续时间的实验操作（12小时与6小时）。我们将通过选择具有一系列教育成就的参与者，他们在不同的轨道上，并专注于各种科目，来最大限度地提高项目评分员库特征的差异。我们还将利用德国和瑞士教育环境的独特多样性。将被评定的课程包括两个年级和两个核心科目，科学（侧重于浮动和下沉，3-4年级）和数学（侧重于勾股定理，8-9年级）。将在四项实验中使用对照组研究RQ 1和RQ 2。RQ 3-5使用每个感兴趣区域的实验操作进行研究。中心因变量是评分员间信度、评分员与参考评分的一致性、以及学生概念理解和内容特定兴趣的预测效度。对于RQ 3和RQ 4，将对子样本进行大声思考和认知访谈，以进一步了解实验条件之间的差异。

Classroom observations are frequently used in teacher evaluation, training, professional development, and research. Ensuring that the observations are reliable and valid is therefore of great importance. Studies have shown however, that they are often neither, even when the raters are trained. There is limited systematic and causal evidence about ways of enhancing the reliability and validity of such ratings. One promising avenue of research is looking at the characteristics of the raters and how well they are trained.The aim of this project is to systematically investigate fundamental rater characteristics and aspects of rater training and their effect on the quality (i.e., reliability and validity) of judgments of teaching quality. We aim to generate evidence for how the selection and training of raters can be optimized, taking into account feasibility issues. The project focuses on the Three Basic Dimensions of teaching quality (Praetorius, Klieme, Herbert, & Pinger, 2018): classroom management, student support, and cognitive activation. We will investigate the relevance of the following aspects of observations on teaching quality: (RQ 1) the raters’ content-related characteristics (content knowledge and pedagogical content knowledge with respect to the content of the observed lessons, their study major) and their generic characteristics (personality, pedagogical knowledge, beliefs, and teaching experience) and (RQ 2) specific rater constellations based on typically used numbers of raters (n = 2-4) in a heterogeneous pool of raters, (RQ 3) experimental manipulations of raters’ content knowledge and pedagogical content knowledge of the topic of the observed lessons, (RQ 4) experimental manipulations of the practice phase (guided discussion vs. self-directed reflection) of rater training, and (RQ 5) experimental manipulations of training duration (12h versus 6h). We will maximize the variance of the characteristics of the project’s rater pool by selecting participants with a range of educational attainments, who are on different tracks, and specializing in a variety of subjects. We will also take advantage of the uniquely heterogeneous nature of education settings in Germany and Switzerland. The lessons that will be rated comprise two grade levels and two core subjects, science (focus on floating and sinking, grades 3-4) and mathematics (focus on the Pythagorean theorem, grades 8-9). RQ 1 and RQ 2 will be investigated using the control groups in four experiments. RQ 3-5 are investigated using experimental manipulations of each of the areas of interest. The central dependent variables are inter-rater reliability, rater agreement with reference ratings, and predictive validity with respect to students’ conceptual understanding as well as content-specific interest. For RQ 3 and RQ 4, think-alouds and cognitive interviews with subsamples will be conducted to gain further insight into the differences between the experimental conditions.