The Expanded Hierarchical Rater Model: A Framework for the Analysis of Ratings

扩展的分层评级模型：评级分析框架

• 批准号: 1324587
• 负责人: Brian Junker
• 金额: $ 35万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2013
• 资助国家: 美国
• 起止时间: 2013-09-01 至 2018-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1324587&HistoricalAwards=false
• 关键词: Expanded; Hierarchical; Rater; Model; Framework

Assessment of individuals' proficiency at complex tasks is often accomplished by observation and rating. Teachers or testing agencies, for example, rate students' essays and their solutions to complex problems in mathematics and science. School districts employ trained observers to rate teachers' performance in the classroom. Experts rate radiologists' ability to classify x-ray images. Ratings, however, may change over time due to changes in the way the rater perceives the work and/or changes in individuals' proficiency. The material being rated also may reflect more than one dimension of proficiency. Finally, summaries of these ratings may be misleading when the data collection design includes groupings (schools, hospitals, etc.) that introduce extraneous statistical dependence into the rating data. This project will expand the Hierarchical Rater Model (HRM), a multilevel item response theory model that accounts for dependencies between multiple ratings of the same work, into a framework that will accommodate (a) variation in ratings over time; (b) multidimensional assessments; and (c) clusters and other hierarchical structure introduced by the data collection design. This new framework will allow the HRM to provide estimates of the overall proficiencies of individuals on the rated tasks, as well as estimates of precision, accuracy, and other rater characteristics, under a broad variety of practical rating situations. Analytical work, simulation studies, and real data applications will be used to explore and demonstrate the feasibility and applicability of the expanded HRM framework. In particular, planned analysis of data from the Measures of Effective Teaching project (MET; Bill and Melinda Gates Foundation, 2012), a large study of class-room teaching in the United States, will demonstrate feasibility of the proposed methodological advancements to the HRM. The research will culminate with a new HRM framework with unified notation and formulations so that researchers may specify and estimate special cases of the generalized model as needed. The project also will provide computational tools including algorithms and source code, so that researchers can apply the framework with ease.The new HRM framework will advance scientific and practical knowledge in two ways. It will enable researchers and practitioners to obtain high-quality estimates of proficiency that account and adjust for complex structure in the ratings. It also will provide rich information about raters and the rating process. Ratings of work, performance, and behavior are an increasing part of high-stakes decisions in many fields including human resources, medical diagnosis, and psychology. The largest impact of this project may be in education policy and research, where ratings of teachers and students are increasingly common. The new HRM framework will allow researchers and practitioners in these fields to produce more accurate assessments of individuals being rated, and to diagnose possible issues in the measurement and rating design, contributing to improved high-stakes decision making based on rating data.

对个人完成复杂任务的熟练程度的评估通常是通过观察和评级来完成的。 例如，教师或测试机构会对学生的论文及其对数学和科学中复杂问题的解决方案进行评分。 学区聘请经过培训的观察员来评估教师在课堂上的表现。 专家对放射科医生对 X 射线图像进行分类的能力进行评分。 然而，由于评级者对工作的看法的变化和/或个人熟练程度的变化，评级可能会随着时间的推移而变化。 被评级的材料也可能反映不止一个维度的熟练程度。 最后，当数据收集设计包括将无关统计依赖性引入评级数据的分组（学校、医院等）时，这些评级的摘要可能会产生误导。 该项目将把分层评分者模型（HRM）（一种多级项目响应理论模型，考虑同一作品的多个评分之间的依赖关系）扩展到一个框架，该框架将适应（a）随着时间的推移评分的变化； (b) 多层面评估； (c) 数据收集设计引入的集群和其他层次结构。 这个新框架将允许人力资源管理部门在各种实际评级情况下提供个人对评级任务的总体熟练程度的估计，以及精度、准确性和其他评级者特征的估计。 将通过分析工作、模拟研究和实际数据应用来探索和论证扩展的人力资源管理框架的可行性和适用性。 特别是，计划对有效教学措施项目（MET；比尔和梅琳达·盖茨基金会，2012 年）（一项针对美国课堂教学的大型研究）的数据进行分析，将证明拟议的人力资源管理方法改进的可行性。 该研究最终将形成一个具有统一符号和公式的新人力资源管理框架，以便研究人员可以根据需要指定和估计广义模型的特殊情况。 该项目还将提供包括算法和源代码在内的计算工具，以便研究人员能够轻松应用该框架。新的人力资源管理框架将从两个方面推进科学和实践知识的发展。 它将使研究人员和从业者能够获得高质量的熟练程度估计，并考虑和调整评级中的复杂结构。 它还将提供有关评级者和评级过程的丰富信息。 在人力资源、医疗诊断和心理学等许多领域，对工作、绩效和行为的评级越来越成为高风险决策的一部分。 该项目最大的影响可能是在教育政策和研究方面，对教师和学生的评分越来越普遍。 新的人力资源管理框架将使这些领域的研究人员和从业者能够对被评级的个人进行更准确的评估，并诊断测量和评级设计中可能存在的问题，从而有助于根据评级数据改进高风险决策。

项目成果

Accounting for Rater Effects With the Hierarchical Rater Model Framework When Scoring Simple Structured Constructed Response Tests

在对简单结构化构建响应测试进行评分时，使用分层评分者模型框架考虑评分者效应

• DOI: 10.1111/jedm.12225
• 发表时间: 2019
• 期刊: Journal of Educational Measurement
• 影响因子: 1.3
• 作者: Nieto, Ricardo;Casabianca, Jodi M.
• 通讯作者: Casabianca, Jodi M.