Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches
Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches
复制标题
DOI：
10.1007/s40593-019-00174-2
复制
发表时间：
2019-05-01
影响因子：
4.9
通讯作者：
Ullmann, Thomas Daniel
中科院分区：
文献类型：
--
作者：
Ullmann, Thomas Daniel
反思性写作是培养反思性思维的重要教育实践。目前，研究人员必须手动分析这些著作，这限制了实践和研究，因为分析非常耗时和资源。这项研究评估了机器学习是否可以用于自动化这种手动分析。该研究调查了评估反思性写作模型中经常使用的八个类别，评估基于 76 篇学生论文（5080 个句子），这些论文主要来自三年级和二年级的健康、商科和工程专业学生。为了测试对写作中反思的自动分析，我们根据 80% 的句子的随机样本构建了机器学习模型。然后对剩余 20% 的句子测试这些模型。总体而言，标准化评估表明，八个类别中的五个可以以相当大或几乎完美的可靠性自动检测，而其他三个类别可以以中等可靠性检测（Cohen 范围在 0.53 到 0.85 之间）。自动分析的准确度平均比手动分析的准确度低 10%。这些发现使得反射分析能够立即进行且可扩展。
Reflective writing is an important educational practice to train reflective thinking. Currently, researchers must manually analyze these writings, limiting practice and research because the analysis is time and resource consuming. This study evaluates whether machine learning can be used to automate this manual analysis. The study investigates eight categories that are often used in models to assess reflective writing, and the evaluation is based on 76 student essays (5080 sentences) that are largely from third- and second-year health, business, and engineering students. To test the automated analysis of reflection in writings, machine learning models were built based on a random sample of 80% of the sentences. These models were then tested on the remaining 20% of the sentences. Overall, the standardized evaluation shows that five out of eight categories can be detected automatically with substantial or almost perfect reliability, while the other three categories can be detected with moderate reliability (Cohen's ranges between .53 and .85). The accuracies of the automated analysis were on average 10% lower than the accuracies of the manual analysis. These findings enable reflection analytics that is immediate and scalable.