Development of statistical scoring system for essay-type tests

点击翻译按钮获取中文摘要

Procedures1) In the first experiment the connected scoring design was used. A design matrix consisted of six raters and 290 essays. Six raters rated 260 essays by using heuristic method or analytical method. We obtained an incomplete data matrix with a total of 1560 scores. On the other hand we developed an automated scoring system by using these answers.2) In the second experiment 10 raters rated 280 essays. We obtained a complete data matrix. These data were used in simulation studies and these answers were used to improve the accuracy of the automated scoring system.Results1) Optimal scoring designOn the basis of simulation studies we proposed a connected scoring design in which two or three raters rated a common block of essays. But each rater dose not rate all essays. Using this design we can obtain the scores more efficiently2) Reliability assessment We proposed assessment procedures for reliability of the connected scoring design in respect of Generalizability theory and ANOVA approach for an incomplete data matrix. Inter correlation coefficient was only 0.293 in heuristic scoring method. On the other hand, it was 0.498 in analytical scoring method. Furthermore, if the scores were transformed by the proposed method, the coefficient was 0.586.3) Development of an automated scoring systemWe developed an automated scoring system on the basis of SVM (Support Vector Machine) and improved the system. If we give about 200 sample answers the system, it is able to rate essays accurately as well as human raters. This result means that the system can assist human raters in scoring procedures.