The Expanded Hierarchical Rater Model: A Framework for the Analysis of Ratings

对个人在复杂任务中的熟练程度的评估通常通过观察和评级来完成。例如，教师或考试机构会对学生的论文及其对数学和科学中复杂问题的解决方案进行评分。学区聘请训练有素的观察员来评估教师在课堂上的表现。专家对放射科医生对X射线图像进行分类的能力进行了评级。然而，评级可能会随着时间的推移而变化，因为评分者对工作的看法发生了变化，和/或个人的熟练程度发生了变化。被评分的材料也可能反映出不止一个方面的熟练程度。最后，当数据收集设计包括分组(学校、医院等)时，这些评级的汇总可能会产生误导。这在评级数据中引入了无关的统计相关性。该项目将把分级评价者模型(HRM)--一种解释同一工作的多个评分之间相关性的多级项目反应理论模型--扩展为一个框架，以适应(A)评分随时间的变化；(B)多层面评估；(C)分组和数据收集设计引入的其他层级结构。这一新的框架将允许人力资源管理局在各种实际评级情况下，提供对个人在评级任务上的总体熟练程度的估计，以及对精确度、准确度和其他评分员特征的估计。将使用分析工作、模拟研究和实际数据应用来探索和论证扩展的人力资源管理框架的可行性和适用性。特别是，对有效教学措施项目(MET；比尔和梅琳达·盖茨基金会，2012年)的数据进行有计划的分析，这是一项关于美国课堂教学的大型研究，将证明拟议的方法进步对人力资源管理的可行性。这项研究的最终结果将是一个新的人力资源管理框架，具有统一的符号和公式，以便研究人员可以根据需要指定和估计广义模型的特殊情况。该项目还将提供包括算法和源代码在内的计算工具，以便研究人员可以轻松地应用该框架。新的人力资源管理框架将从两个方面促进科学和实用知识的发展。它将使研究人员和从业者能够获得高质量的熟练程度估计，并根据评级中的复杂结构进行调整。它还将提供有关评分者和评级过程的丰富信息。在人力资源、医疗诊断和心理学等许多领域，对工作、表现和行为的评级越来越多地成为高风险决策的一部分。该项目最大的影响可能是在教育政策和研究方面，在这些领域，教师和学生的评级越来越普遍。新的人力资源管理框架将允许这些领域的研究人员和从业人员对被评级的个人进行更准确的评估，并诊断衡量和评级设计中可能存在的问题，有助于改进基于评级数据的高风险决策。

Assessment of individuals' proficiency at complex tasks is often accomplished by observation and rating. Teachers or testing agencies, for example, rate students' essays and their solutions to complex problems in mathematics and science. School districts employ trained observers to rate teachers' performance in the classroom. Experts rate radiologists' ability to classify x-ray images. Ratings, however, may change over time due to changes in the way the rater perceives the work and/or changes in individuals' proficiency. The material being rated also may reflect more than one dimension of proficiency. Finally, summaries of these ratings may be misleading when the data collection design includes groupings (schools, hospitals, etc.) that introduce extraneous statistical dependence into the rating data. This project will expand the Hierarchical Rater Model (HRM), a multilevel item response theory model that accounts for dependencies between multiple ratings of the same work, into a framework that will accommodate (a) variation in ratings over time; (b) multidimensional assessments; and (c) clusters and other hierarchical structure introduced by the data collection design. This new framework will allow the HRM to provide estimates of the overall proficiencies of individuals on the rated tasks, as well as estimates of precision, accuracy, and other rater characteristics, under a broad variety of practical rating situations. Analytical work, simulation studies, and real data applications will be used to explore and demonstrate the feasibility and applicability of the expanded HRM framework. In particular, planned analysis of data from the Measures of Effective Teaching project (MET; Bill and Melinda Gates Foundation, 2012), a large study of class-room teaching in the United States, will demonstrate feasibility of the proposed methodological advancements to the HRM. The research will culminate with a new HRM framework with unified notation and formulations so that researchers may specify and estimate special cases of the generalized model as needed. The project also will provide computational tools including algorithms and source code, so that researchers can apply the framework with ease.The new HRM framework will advance scientific and practical knowledge in two ways. It will enable researchers and practitioners to obtain high-quality estimates of proficiency that account and adjust for complex structure in the ratings. It also will provide rich information about raters and the rating process. Ratings of work, performance, and behavior are an increasing part of high-stakes decisions in many fields including human resources, medical diagnosis, and psychology. The largest impact of this project may be in education policy and research, where ratings of teachers and students are increasingly common. The new HRM framework will allow researchers and practitioners in these fields to produce more accurate assessments of individuals being rated, and to diagnose possible issues in the measurement and rating design, contributing to improved high-stakes decision making based on rating data.