The Expanded Hierarchical Rater Model: A Framework for the Analysis of Ratings

评估个人在复杂任务中的熟练程度通常通过观察和评级来完成。 例如，教师或考试机构会对学生的论文以及他们对数学和科学中复杂问题的解决方案进行评分。 学区聘请训练有素的观察员来评价教师在课堂上的表现。 专家们对放射科医生对X光图像进行分类的能力进行了评估。 然而，评级可能会随着时间的推移而变化，这是由于评级者对工作的看法和/或个人熟练程度的变化。 被评定的材料也可能反映不止一个方面的熟练程度。 最后，当数据收集设计包括分组（学校、医院等）时，这些评级的摘要可能会产生误导。这将无关的统计依赖性引入评级数据。 该项目将把分层评分模型（HRM）（一个多层次项目反应理论模型，用于说明同一工作的多个评分之间的依赖关系）扩展为一个框架，以适应（a）评分随时间的变化;（B）多维评估;（c）数据收集设计引入的聚类和其他分层结构。 这一新的框架将使人力资源管理局能够在各种实际评级情况下，对个人在评级任务上的总体专业水平以及对精确度、准确度和其他评级者特征的估计作出估计。 分析工作、模拟研究和真实的数据应用将用于探索和证明扩大的人力资源管理框架的可行性和适用性。 特别是，计划对有效教学措施项目（MET;比尔和梅林达·盖茨基金会，2012年）的数据进行分析，这是对美国教室教学的一项大型研究，将证明人力资源管理局拟议的方法改进的可行性。 这项研究将最终与一个新的人力资源管理框架统一的符号和配方，使研究人员可以指定和估计特殊情况下的广义模型的需要。 该项目还将提供包括算法和源代码在内的计算工具，以便研究人员可以轻松地应用该框架。新的人力资源管理框架将从两个方面促进科学和实践知识的发展。 它将使研究人员和从业人员能够获得高质量的熟练程度估计，并根据评级中的复杂结构进行调整。 它还将提供有关评分员和评分过程的丰富信息。 在许多领域，工作、绩效和行为的评级越来越成为高风险决策的一部分，包括人力资源、医疗诊断和心理学。 该项目的最大影响可能是在教育政策和研究方面，对教师和学生的评级越来越普遍。 新的人力资源管理框架将使这些领域的研究人员和从业人员能够对被评级的个人进行更准确的评估，并诊断测量和评级设计中可能存在的问题，有助于改进基于评级数据的高风险决策。

Assessment of individuals' proficiency at complex tasks is often accomplished by observation and rating. Teachers or testing agencies, for example, rate students' essays and their solutions to complex problems in mathematics and science. School districts employ trained observers to rate teachers' performance in the classroom. Experts rate radiologists' ability to classify x-ray images. Ratings, however, may change over time due to changes in the way the rater perceives the work and/or changes in individuals' proficiency. The material being rated also may reflect more than one dimension of proficiency. Finally, summaries of these ratings may be misleading when the data collection design includes groupings (schools, hospitals, etc.) that introduce extraneous statistical dependence into the rating data. This project will expand the Hierarchical Rater Model (HRM), a multilevel item response theory model that accounts for dependencies between multiple ratings of the same work, into a framework that will accommodate (a) variation in ratings over time; (b) multidimensional assessments; and (c) clusters and other hierarchical structure introduced by the data collection design. This new framework will allow the HRM to provide estimates of the overall proficiencies of individuals on the rated tasks, as well as estimates of precision, accuracy, and other rater characteristics, under a broad variety of practical rating situations. Analytical work, simulation studies, and real data applications will be used to explore and demonstrate the feasibility and applicability of the expanded HRM framework. In particular, planned analysis of data from the Measures of Effective Teaching project (MET; Bill and Melinda Gates Foundation, 2012), a large study of class-room teaching in the United States, will demonstrate feasibility of the proposed methodological advancements to the HRM. The research will culminate with a new HRM framework with unified notation and formulations so that researchers may specify and estimate special cases of the generalized model as needed. The project also will provide computational tools including algorithms and source code, so that researchers can apply the framework with ease.The new HRM framework will advance scientific and practical knowledge in two ways. It will enable researchers and practitioners to obtain high-quality estimates of proficiency that account and adjust for complex structure in the ratings. It also will provide rich information about raters and the rating process. Ratings of work, performance, and behavior are an increasing part of high-stakes decisions in many fields including human resources, medical diagnosis, and psychology. The largest impact of this project may be in education policy and research, where ratings of teachers and students are increasingly common. The new HRM framework will allow researchers and practitioners in these fields to produce more accurate assessments of individuals being rated, and to diagnose possible issues in the measurement and rating design, contributing to improved high-stakes decision making based on rating data.