The rating reliability calculator.
The rating reliability calculator.
复制标题
DOI：
10.1186/1471-2288-4-11
复制
发表时间：
2004-04-29
影响因子：
4
通讯作者：
Solomon, David J
中科院分区：
文献类型：
--
作者：
Solomon, David J
背景：评价表是收集评估数据的重要手段。由于重要决策通常基于这些评估，因此确定评级数据的可靠性可能至关重要。最常用的评估可靠性的方法需要一套完整的评级，即每个被评级的对象都必须由每个法官进行评级。五十多年前，伊贝尔描述了一种基于不完整数据估计评级可靠性的算法。尽管他的文章多年来被广泛引用，但基于该算法的软件并不容易获得。本文描述了一个简单易用的基于Web的实用程序，用于基于不完全数据使用Ebel算法来评估评级的可靠性。方法：该程序在我们的服务器上可供公众使用，源代码可在GNU通用公共许可证下免费获得。该实用程序是用PHP编写的，这是一种常见的开源嵌入式脚本语言。结果：当程序运行时，它会显示可信度、被评对象数、每个被评对象的调和平均评委数、每个被评对象平均评分的平均值和标准差。该程序还显示每个被评级科目的平均值、标准差和评级数量。此外，该程序将通过Spearman-Brown预测公式估计每个主题的平均评级的可靠性。结论：这个基于网络的简单程序提供了一种方便的方法来估计评级数据的可靠性，而不需要进行专门研究来提供完整的评级数据。我欢迎其他研究人员对该计划进行修改和改进。
BACKGROUND: Rating scales form an important means of gathering evaluation data. Since important decisions are often based on these evaluations, determining the reliability of rating data can be critical. Most commonly used methods of estimating reliability require a complete set of ratings i.e. every subject being rated must be rated by each judge. Over fifty years ago Ebel described an algorithm for estimating the reliability of ratings based on incomplete data. While his article has been widely cited over the years, software based on the algorithm is not readily available. This paper describes an easy-to-use Web-based utility for estimating the reliability of ratings based on incomplete data using Ebel's algorithm.METHODS: The program is available public use on our server and the source code is freely available under GNU General Public License. The utility is written in PHP, a common open source imbedded scripting language. The rating data can be entered in a convenient format on the user's personal computer that the program will upload to the server for calculating the reliability and other statistics describing the ratings.RESULTS: When the program is run it displays the reliability, number of subject rated, harmonic mean number of judges rating each subject, the mean and standard deviation of the averaged ratings per subject. The program also displays the mean, standard deviation and number of ratings for each subject rated. Additionally the program will estimate the reliability of an average of a number of ratings for each subject via the Spearman-Brown prophecy formula.CONCLUSION: This simple web-based program provides a convenient means of estimating the reliability of rating data without the need to conduct special studies in order to provide complete rating data. I would welcome other researchers revising and enhancing the program.