How Does Augmented Observation Facilitate Multimodal Representational Thinking? Applying Deep Learning to Decode Complex Student Construct
How Does Augmented Observation Facilitate Multimodal Representational Thinking? Applying Deep Learning to Decode Complex Student Construct
复制标题
DOI：
10.1007/s10956-020-09856-2
复制
发表时间：
2020-09
影响因子：
4.4
通讯作者：
S. Sung;Chenglu Li;Guanhua Chen;Xudong Huang;Charles Xie;Joyce Massicotte;Ji Shen
中科院分区：
文献类型：
--
作者：
S. Sung;Chenglu Li;Guanhua Chen;Xudong Huang;Charles Xie;Joyce Massicotte;Ji Shen
在本文中，我们演示了如何使用机器学习来快速评估学生的多模态表征思维。多模态表征思维是一种复杂的结构，它编码了学生如何在头脑中形成概念，感知，图形或数学符号。采用增强现实（AR）技术来多样化学生的表现。AR技术利用了一个低成本，高分辨率的热成像摄像头连接到智能手机，让学生探索看不见的热力学世界。九年级学生（N =314）从事预测-观察-解释（POE）查询周期脚手架利用上述设备提供的增强观察。目的是研究机器学习如何加快热能多模态表征思维的自动评估。采用两种自动文本分类方法来解码不同的心理表征学生用来解释他们的触觉感知，热成像，和在实验室收集的图形数据。由于目前科学教育中的自动评估很少考虑多标签分类，因此我们求助于最先进的深度学习技术-变压器双向编码器表示（BERT）。BERT模型将开放式响应分类到适当的类别中，比传统的机器学习方法具有更高的精度。深度学习在分配多个标签方面的令人满意的准确性在处理定性数据方面是革命性的。复杂的学生结构，如多模态表征思维，很少是相互排斥的。该研究利用一种方便的技术来分析不满足互斥假设的定性数据。最后讨论了研究的意义和未来的研究方向。
In this paper, we demonstrate how machine learning could be used to quickly assess a student’s multimodal representational thinking. Multimodal representational thinking is the complex construct that encodes how students form conceptual, perceptual, graphical, or mathematical symbols in their mind. The augmented reality (AR) technology is adopted to diversify student’s representations. The AR technology utilized a low-cost, high-resolution thermal camera attached to a smartphone which allows students to explore the unseen world of thermodynamics. Ninth-grade students (N =314) engaged in a prediction–observation–explanation (POE) inquiry cycle scaffolded to leverage the augmented observation provided by the aforementioned device. The objective is to investigate how machine learning could expedite the automated assessment of multimodal representational thinking of heat energy. Two automated text classification methods were adopted to decode different mental representations students used to explain their haptic perception, thermal imaging, and graph data collected in the lab. Since current automated assessment in science education rarely considers multilabel classification, we resorted to the help of the state-of-the-art deep learning technique—bidirectional encoder representations from transformers (BERT). The BERT model classified open-ended responses into appropriate categories with higher precision than the traditional machine learning method. The satisfactory accuracy of deep learning in assigning multiple labels is revolutionary in processing qualitative data. The complex student construct, such as multimodal representational thinking, is rarely mutually exclusive. The study avails a convenient technique to analyze qualitative data that does not satisfy the mutual-exclusiveness assumption. Implications and future studies are discussed.