A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants.
A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants.
复制标题
DOI：
10.1371/journal.pone.0014331
复制
发表时间：
2010-12-14
期刊：
影响因子：
3.7
通讯作者：
Daniel HD
中科院分区：
文献类型：
--
作者：
Bornmann L;Mutz R;Daniel HD
本文首次对期刊同行评审的评价者间可靠性(IRR)进行了Meta分析。IRR被定义为同一科学文献的两个或两个以上独立审查一致的程度。Meta分析共考虑了48项研究的70个信度系数(Cohen‘s Kappa、类内相关[ICC]和皮尔逊积差相关[r])。这些研究总共基于19443份手稿；平均而言，每项研究的样本量为311份(最小：28份，最大：1983年)。Meta分析的结果证实了迄今为止已发表的叙述性文献综述的发现：内部回报率(Mean ICC/R2 = .34，Mean Cohen‘s Kappa = .17)水平较低。为了解释IRR系数在不同研究之间的差异，使用了7个协变量进行了Meta回归分析。在元回归分析中出现的两个具有统计学意义的协变量获得了类内相关性的近似同质性，这表明，首先，一项研究所依据的手稿越多，报告的IRR系数就越小。其次，如果在一项研究中报告了评审员评级系统的信息，那么这与较小的IRR系数有关，而不是在没有传达信息的情况下。报告高IRR的研究将被认为比那些低IRR的研究更不可信。根据我们的元分析，同行评估的IRR相当有限，需要改进(例如，阅读器系统)。
This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree. Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2 = .34, mean Cohen's Kappa = .17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed. Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).
登录
查看更多内容
影响因子：
1.4
作者：
Bateman, IJ;Jones, AP
通讯作者：
Jones, AP
影响因子：
16.6
作者：
Bornmann, Lutz;Daniel, Hans-Dieter
通讯作者：
Daniel, Hans-Dieter
影响因子：
120.7
作者：
Cho, MK;Justice, AC;Rennie, D
通讯作者：
Rennie, D
影响因子：
10.5
作者：
BEYER, JM;CHANOVE, RG;FOX, WB
通讯作者：
FOX, WB
影响因子：
8.2
作者：
Blackburn, JL;Hakel, MD
通讯作者：
Hakel, MD