Handwriting Recognition and Automatic Scoring for Descriptive Answers in Japanese Language Tests
Handwriting Recognition and Automatic Scoring for Descriptive Answers in Japanese Language Tests
复制标题
DOI：
10.1007/978-3-031-21648-0_19
复制
发表时间：
2022-01
期刊：
影响因子：
--
通讯作者：
Hung Tuan Nguyen;C. Nguyen;Haruki Oka;T. Ishioka;M. Nakagawa
中科院分区：
文献类型：
--
作者：
Hung Tuan Nguyen;C. Nguyen;Haruki Oka;T. Ishioka;M. Nakagawa
本文介绍了在2017年和2018年对约12万名考生进行的新日本大学入学考试试用测试中自动评分手写描述性答案的实验。有大约40万个答案，超过2000万个字符。虽然所有的答案都是由人类考官评分的，但手写字符没有标记。我们尝试将在标记的手写数据集上训练的基于深度神经网络的手写识别器适应这个未标记的答案集。我们提出的方法结合了不同的训练策略，集成了多个识别器，并使用了从大型通用语料库构建的语言模型，以避免过度拟合到特定数据中。在我们的实验中，该方法使用约2，000个经过验证的标记答案记录了超过97%的字符准确率，这些答案占数据集的不到0.5%。然后，识别的答案被送入一个预先训练的自动评分系统的基础上的BERT模型，而不纠正错误识别的字符和提供标题注释。自动评分系统的二次加权Kappa值（QWK）为0.84 ~ 0.98。由于QWK超过0.8，它表示自动评分系统和人类考官之间的评分的可接受的相似性。这些结果为描述性答案的端到端自动评分的进一步研究提供了基础。
This paper presents an experiment of automatically scoring handwritten descriptive answers in the trial tests for the new Japanese university entrance examination, which were made for about 120,000 examinees in 2017 and 2018. There are about 400,000 answers with more than 20 million characters. Although all answers have been scored by human examiners, handwritten characters are not labeled. We present our attempt to adapt deep neural network-based handwriting recognizers trained on a labeled handwriting dataset into this unlabeled answer set. Our proposed method combines different training strategies, ensembles multiple recognizers, and uses a language model built from a large general corpus to avoid overfitting into specific data. In our experiment, the proposed method records character accuracy of over 97% using about 2,000 verified labeled answers that account for less than 0.5% of the dataset. Then, the recognized answers are fed into a pre-trained automatic scoring system based on the BERT model without correcting misrecognized characters and providing rubric annotations. The automatic scoring system achieves from 0.84 to 0.98 of Quadratic Weighted Kappa (QWK). As QWK is over 0.8, it represents an acceptable similarity of scoring between the automatic scoring system and the human examiners. These results are promising for further research on end-to-end automatic scoring of descriptive answers.