Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations
Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations
复制标题
DOI：
10.1007/s10956-011-9300-9
复制
发表时间：
2012-02-01
影响因子：
4.4
通讯作者：
Mayfield, Elijah
中科院分区：
文献类型：
--
作者：
Nehm, Ross H.;Ha, Minsu;Mayfield, Elijah
这项研究探索了使用机器学习来自动评估学生对进化变化的书面解释的准确性。总结综合开发环境（SIDE）程序的性能进行了比较，人类专家评分使用语料库的2,260进化的解释，由565名本科生写的两种不同的进化工具（EGALT-F和EGALT-P），其中包含提示，不同的表面特征（如物种和性状）。我们在一系列不同的培训和测试条件下测试了人类-SIDE评分对应性，使用Kappa评分员间一致性值大于0.80作为性能基准。此外，我们还研究了反应长度对评分成功率的影响，即SIDE评分模型在短反应和长反应上是否具有可比性。我们发现，SIDE性能是最有效的评分模型时，建立和测试在单个项目的水平和性能下降时，套件的项目或整个仪器被用来建立和测试评分模型。总体而言，SIDE被认为是一个强大的和具有成本效益的工具，用于评估学生的知识和表现在一个复杂的科学领域。
This study explored the use of machine learning to automatically evaluate the accuracy of students' written explanations of evolutionary change. Performance of the Summarization Integrated Development Environment (SIDE) program was compared to human expert scoring using a corpus of 2,260 evolutionary explanations written by 565 undergraduate students in response to two different evolution instruments (the EGALT-F and EGALT-P) that contained prompts that differed in various surface features (such as species and traits). We tested human-SIDE scoring correspondence under a series of different training and testing conditions, using Kappa inter-rater agreement values of greater than 0.80 as a performance benchmark. In addition, we examined the effects of response length on scoring success; that is, whether SIDE scoring models functioned with comparable success on short and long responses. We found that SIDE performance was most effective when scoring models were built and tested at the individual item level and that performance degraded when suites of items or entire instruments were used to build and test scoring models. Overall, SIDE was found to be a powerful and cost-effective tool for assessing student knowledge and performance in a complex science domain.