Using Scientific Materials to Compare Disaggregated Versus Holistic Ratings

美国国立卫生研究院(NIH)、国家科学基金会(NSF)和其他机构每年花费数十亿美元评估科学提案。只有最有功绩的人才应该得到纳税人的钱。但这些机构应该如何决定哪些提案是最有价值的？评估提案、候选人、战略和几乎任何其他实体的一般方法有两种。第一种方法被称为“整体”。整体评级的特点是只有一个反映个人意见的总体数字。第二种方式被称为“分解”。分类评级的特点是对几个标准中的每一个都给予一个评级。然后，将这些标准的评级相加或平均，以获得一个反映个人意见的单一数字。一些研究表明，分类评级优于整体评级。然而，在1990年代后期，国家卫生研究院和国家科学基金会都拒绝在自己的提案评估程序中采用分类评级，因为以前没有对整体评级和分类评级进行比较的研究使用了科学提案等材料。拟议中的研究将填补这一空白。科学大会的与会者将被要求以整体或分类的方式对口头陈述进行评分。两个五人的整体小组和两个五人的分类小组将对每一次发言进行评分。两个整体小组对发言的评分之间的相关性将与两个分类小组对发言的评分之间的相关性进行比较。如果先前的研究适用于科学陈述，则分类小组评分之间的相关性应高于整体小组评分之间的相关性。这将提供证据，证明对科学材料的分类评级比整体评级产生更高可靠性的评估。这一结果应反过来指导科学机构改进其评价过程的一种方式。

The National Institutes of Health (NIH), the National Science Foundation (NSF), and other agencies spend billions of dollars each year evaluating scientific proposals. Only the most meritorious should receive taxpayer dollars. But how should these agencies decide which proposals are the most meritorious?There are two general ways to evaluate proposals, candidates, strategies, and nearly any other entity. The first way is called "holistic." Holistic ratings are characterized by a single overall number which reflects one's opinion. The second way is called "disaggregated." Disaggregated ratings are characterized by one rating being given to each of several criteria. The criteria's ratings are then summed or averaged to obtain a single number, which reflects one's opinion. Some research suggests that disaggregated ratings are superior to holistic ones. However in the late 1990s, both NIH and NSF refused to adopt disaggregated ratings in their own proposal evaluation procedures, because no prior research comparing holistic and disaggregated ratings used materials such as scientific proposals. The proposed research will fill that gap. Attendees at a scientific convention will be asked to rate oral presentations in either a holistic or disaggregated fashion. Two holistic panels of five persons each and two disaggregated panels of five persons each will rate every presentation. The correlation between the two holistic panels' ratings of the presentations will be compared to the correlation between the two disaggregated panels' ratings of the presentations. If prior research applies to scientific presentations, then the correlation between the disaggregated panels' ratings should be higher than the correlation between the holistic panels' ratings. This will provide evidence that disaggregated ratings of scientific materials yields evaluations of higher reliability than do holistic ratings. This result should in turn provide guidance to the scientific agencies as to one way in which their evaluation processes might be improved.