权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Using Scientific Materials to Compare Disaggregated Versus Holistic Ratings

使用科学材料比较分类评级与整体评级

基本信息

批准号：
0109250
负责人：
Hal Arkes
金额：
$ 8.44万
依托单位：
Ohio State University Research Foundation -DO NOT USE
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2001
资助国家：
美国
起止时间：
2001-08-01 至 2003-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=0109250&HistoricalAwards=false
关键词：
Using Scientific Materials Compare Disaggregated

项目摘要

The National Institutes of Health (NIH), the National Science Foundation (NSF), and other agencies spend billions of dollars each year evaluating scientific proposals. Only the most meritorious should receive taxpayer dollars. But how should these agencies decide which proposals are the most meritorious?There are two general ways to evaluate proposals, candidates, strategies, and nearly any other entity. The first way is called "holistic." Holistic ratings are characterized by a single overall number which reflects one's opinion. The second way is called "disaggregated." Disaggregated ratings are characterized by one rating being given to each of several criteria. The criteria's ratings are then summed or averaged to obtain a single number, which reflects one's opinion. Some research suggests that disaggregated ratings are superior to holistic ones. However in the late 1990s, both NIH and NSF refused to adopt disaggregated ratings in their own proposal evaluation procedures, because no prior research comparing holistic and disaggregated ratings used materials such as scientific proposals. The proposed research will fill that gap. Attendees at a scientific convention will be asked to rate oral presentations in either a holistic or disaggregated fashion. Two holistic panels of five persons each and two disaggregated panels of five persons each will rate every presentation. The correlation between the two holistic panels' ratings of the presentations will be compared to the correlation between the two disaggregated panels' ratings of the presentations. If prior research applies to scientific presentations, then the correlation between the disaggregated panels' ratings should be higher than the correlation between the holistic panels' ratings. This will provide evidence that disaggregated ratings of scientific materials yields evaluations of higher reliability than do holistic ratings. This result should in turn provide guidance to the scientific agencies as to one way in which their evaluation processes might be improved.

美国国立卫生研究院（NIH）、国家科学基金会（NSF）和其他机构每年花费数十亿美元评估科学提案。只有最有价值的人才应该得到纳税人的钱。但是，这些机构应该如何决定哪些建议是最有价值的？有两种通用的方法来评估提案、候选人、战略和几乎任何其他实体。第一种方法被称为“整体”。“整体评级的特点是一个单一的整体数字，反映了一个人的意见。第二种方式被称为“分解”。“分类评级的特点是对几项标准中的每一项都给予一个评级。然后将这些标准的评分相加或平均，得出一个反映个人意见的数字。一些研究表明，分类评级上级整体评级。然而，在20世纪90年代末，NIH和NSF都拒绝在他们自己的提案评估程序中采用分类评级，因为之前没有比较整体和分类评级的研究使用科学提案等材料。这项研究将填补这一空白。科学会议的与会者将被要求以整体或分类的方式对口头报告进行评分。两个各由五人组成的综合小组和两个各由五人组成的分类小组将对每一份报告进行评分。将比较两个综合小组对专题介绍评分之间的相关性和两个分类小组对专题介绍评分之间的相关性。如果先前的研究适用于科学陈述，那么分类小组评分之间的相关性应该高于整体小组评分之间的相关性。这将提供证据表明，科学材料的分类评级比整体评级产生更高的可靠性评价。这一结果应反过来为科学机构提供指导，使其了解如何改进其评价进程。