Development of statistical scoring system for essay-type tests

1)第一个实验采用连通计分设计。设计矩阵由6位评分者和290篇论文组成。六名评判员对260篇文章分别采用启发式法和分析法进行打分。我们得到了一个不完整的数据矩阵，总共有1560个分数。另一方面，我们利用这些答案开发了一个自动评分系统。2)在第二个实验中，10名评分员对280篇文章进行评分。我们得到了一个完整的数据矩阵。这些数据用于模拟研究，这些答案用于提高自动评分系统的准确性。结果1)最优评分设计在模拟研究的基础上，我们提出了一种关联评分设计，其中两到三个评分者对共同的文章块进行评分。但并不是每个评分员都会对所有的文章进行评分。采用该设计，可以更有效地获得分数。2)可靠性评估我们提出了连接评分设计的可靠性评估程序，基于概化理论和不完全数据矩阵的方差分析方法。启发式评分法的相关系数仅为0.293。另一方面，分析评分法为0.498。基于SVM（支持向量机）的自动评分系统的开发我们开发了一个基于SVM（支持向量机）的自动评分系统，并对系统进行了改进。如果我们给系统提供大约200个样本答案，它就能像人类评分员一样准确地对文章进行评分。这一结果意味着该系统可以在评分过程中协助人类评分员。

Procedures1) In the first experiment the connected scoring design was used. A design matrix consisted of six raters and 290 essays. Six raters rated 260 essays by using heuristic method or analytical method. We obtained an incomplete data matrix with a total of 1560 scores. On the other hand we developed an automated scoring system by using these answers.2) In the second experiment 10 raters rated 280 essays. We obtained a complete data matrix. These data were used in simulation studies and these answers were used to improve the accuracy of the automated scoring system.Results1) Optimal scoring designOn the basis of simulation studies we proposed a connected scoring design in which two or three raters rated a common block of essays. But each rater dose not rate all essays. Using this design we can obtain the scores more efficiently2) Reliability assessment We proposed assessment procedures for reliability of the connected scoring design in respect of Generalizability theory and ANOVA approach for an incomplete data matrix. Inter correlation coefficient was only 0.293 in heuristic scoring method. On the other hand, it was 0.498 in analytical scoring method. Furthermore, if the scores were transformed by the proposed method, the coefficient was 0.586.3) Development of an automated scoring systemWe developed an automated scoring system on the basis of SVM (Support Vector Machine) and improved the system. If we give about 200 sample answers the system, it is able to rate essays accurately as well as human raters. This result means that the system can assist human raters in scoring procedures.