DeepStealth: Game-Based Learning Stealth Assessment With Deep Neural Networks
DeepStealth: Game-Based Learning Stealth Assessment With Deep Neural Networks
复制标题
DOI：
10.1109/tlt.2019.2922356
复制
发表时间：
2020-04
影响因子：
3.7
通讯作者：
Wookhee Min;M. Frankosky;Bradford W. Mott;Jonathan P. Rowe;A. Smith;E. Wiebe;K. Boyer;James C. Lester
中科院分区：
文献类型：
--
作者：
Wookhee Min;M. Frankosky;Bradford W. Mott;Jonathan P. Rowe;A. Smith;E. Wiebe;K. Boyer;James C. Lester
基于游戏的学习环境的一个显着特点是它们能够进行隐形评估。Stealth Assessment分析来自基于游戏的学习环境的细粒度学生交互数据流，通过以证据为中心的设计动态地推断学生的能力。在以证据为中心的设计中，证据模型传统上是使用由领域专家编写的统计规则设计的，这些规则使用贝叶斯网络进行编码。本文介绍了DeepStealth，一个基于深度学习的隐形评估框架，它大大减少了以前创建隐形评估所需的功能工程劳动力。DeepStealth利用端到端可训练的基于深度神经网络的证据模型。使用这个框架，证据模型设计使用一组从原始的，低层次的交互数据中捕获的预测功能，推断能力的证据。我们研究了两种基于深度学习的证据模型，长短期记忆网络（LSTM）和n-gram编码的前馈神经网络（FFNN）。我们比较了这些模型在推断学生知识方面与线性链条件随机场（CRF）和朴素贝叶斯模型的预测性能。我们对游戏跟踪日志和外部预学习措施进行了特征集级别的分析，并检查了模型的早期预测能力。该框架进行了评估，从182名中学生互动的游戏为基础的学习环境中年级计算思维收集的数据。结果表明，基于LSTM的隐身评估器在预测精度和早期预测能力方面优于竞争基线方法。我们发现，LSTM，FFNN和CRF都受益于来自游戏跟踪日志和外部预学习措施的组合特征集。
A distinctive feature of game-based learning environments is their capacity for enabling stealth assessment. Stealth assessment analyzes a stream of fine-grained student interaction data from a game-based learning environment to dynamically draw inferences about students’ competencies through evidence-centered design. In evidence-centered design, evidence models have been traditionally designed using statistical rules authored by domain experts that are encoded using Bayesian networks. This article presents DeepStealth, a deep learning-based stealth assessment framework, that yields significant reductions in the feature engineering labor that has previously been required to create stealth assessments. DeepStealth utilizes end-to-end trainable deep neural network-based evidence models. Using this framework, evidence models are devised using a set of predictive features captured from raw, low-level interaction data to infer evidence for competencies. We investigate two deep learning-based evidence models, long short-term memory networks (LSTMs) and n-gram encoded feedforward neural networks (FFNNs). We compare these models’ predictive performance for inferring students’ knowledge to linear-chain conditional random fields (CRFs) and naïve Bayes models. We perform feature set-level analyses of game trace logs and external pre-learning measures, and we examine the models’ early prediction capacity. The framework is evaluated using data collected from 182 middle school students interacting with a game-based learning environment for middle grade computational thinking. Results indicate that LSTM-based stealth assessors outperform competitive baseline approaches with respect to predictive accuracy and early prediction capacity. We find that LSTMs, FFNNs, and CRFs all benefit from combined feature sets derived from both game trace logs and external pre-learning measures.