Can online educational games be scored?

在线教育游戏提供了对复杂技能进行评分的潜力，这些技能很少被测试，因此在主流教育环境中很少被教授。对一种测试这些技能的方法的需求正在增长，许多人认为这种需求早就应该出现了。然而，高风险测试依赖于定量性能数据分析技术来提高和完善其测试的准确性和公平性，这些技术与遥测数据集中产生的数据类型不兼容。基于游戏的评估领域目前主要由来自游戏设计领域的研究人员主导。许多建议使用遥测数据作为评估性能数据的游戏设计师所依赖的方法并未得到许多正式或法律的认可。将游戏分数与另一个外部标准化测试的分数相关联，或者使用专家判断，都过于简化了校准过程。贝叶斯数据分析方法与项目反应理论兼容，项目反应理论是评估者青睐的校准方法。此外，Bayes还处理要处理的动态的、有条件依赖的超文本数据。然而，很少有研究做了估计偏差程度的数据质量选择升级到贝叶斯；处理缺失数据的方法；最后是处理这些数据的数学模型。本项目旨在探索减少这种偏见的方法，并提出与计算和评估原则兼容的在线游戏数据分析方法。研究问题：1如何在游戏中评估游戏任务的难度和玩家的能力？如何公平地处理游戏中丢失的数据？如何将游戏特定变量（时间、迭代和选择）概念化，这些概念化如何影响学习者排名？研究方法：这是对提取的数据集进行离线定量数据分析。在确定合适的数据集后，将创建各种数据分析模型。在高风险评估中报告给学习者的分数通常是原始数据分数的数学函数，这些函数对集中趋势考虑很敏感。由于时间限制没有上限，第一阶段将是确定极端分数对其他数据的杠杆权重。这将通过前后分数的相关性，在极值封顶后并删除它们来完成。游戏设计师通常假设时间的物理度量与能力具有线性关系。这项研究将采用随机方法，将速度（而非时间）视为玩家、任务以及玩家在该任务中获得的分数的函数。一旦清理干净，遥测数据集中的有序数据将升级为3种评分方法：累积平均值；不断升级高分；并不断升级最近的分数。得到的分数将是有序的，因此第一步将使用对数变换将它们转换为区间数据，这可以在贝叶斯中下降。在一定时间内完成任务的总体难度将给出一个估计值。类似的过程将对年级分数进行，但这些分数不会有不受限制的上限和下限所引起的问题。第二个过程将产生对每个任务的认知难度值的估计。这些认知难度和速度难度估计可以在贝叶斯项目反应模型中使用，以估计学习者的能力。将得到误差的标准度量，即Infit和Outfit统计量。这是预期（我们对任务和学习者的了解）和观察到的行为之间的卡方分析的一种形式，被评估利益相关者用来判断数据集的稳定性。

Online educational games offer the potential to score complex skills that are rarely tested, and therefore rarely taught in mainstream educational settings. Demand for a way to test these skills is growing, and many would argue overdue. However, high stakes testing relies on quantitative performance data analysis techniques to improve and refine the accuracy and fairness of their tests, and these techniques incompatible with the kind of data produced in telemetry data sets. The field of Game Based Assessment is currently dominated by researchers from the games design field. Many games designers who have proposed the use of telemetry data as assessment performance data have relied on methods not recognised in many formal, or legal, settings. Correlation of gaming scores with grades from another external standardised test, or using expert judgement, both oversimplify the calibration process. Bayesian methods of data analysis are compatible with Item Response Theory, the approach to calibration favoured by assessors. In addition, Bayes also handles dynamic, conditionally dependent hypertext data to be handled. However, little research has been done to estimate the degree of bias in the quality of data selected to escalate to the Bayes; the methods of handling missing data; and finally the mathematical model to process that data. This project aims to explore ways to reduce this bias, and propose ways forward for analysing online game data that are compatible with both computing and assessment principles. Research questions:1 How can the difficulty of the game tasks and the ability of the player be estimated in the game? 2 How can missing data be dealt with fairly in games? 3 How can game-specific variables (time, iteration, and choice) be conceptualized and how might these conceptualizations affect learner ranking? Research approach: This is a quantitative data analysis to be carried out offline on extracted data sets. After the identification of suitable data sets, various data analysis models will be created. The scores that are reported to learners in high stakes assessment are usually a mathematical function of raw data scores, and these functions are sensitive to central tendency consideration. As there is no upper bound to the time limit, the first stage will be to identify the weight of the leverage that extreme scores have on the other data. This will be done through correlation of before and after scores, after capping extreme values, and deleting them. Games designers often assume that the physical measure of time has a linear relationship with ability. This study will instead take a stochastic approach, seeing speed, not time, as a function of the player, the task and possibly on the grade band that the player has achieved in that task. Once cleaned, the ordinal data from the telemetry data sets, will be escalated to 3 scoring approaches: the accumulated mean; escalating the high score; and escalating the most recent score. The resulting scores will be ordinal, and so the first step will be to transform those into interval data using a logarithmic transformation, and this can be down within Bayes. The overall difficulty of completing the tasks within a certain time will be given an estimated value. A similar process will be carried out with the grade scores, but these will not have the issues raised by unconstrained upper and lower boundaries. This second process will produce an estimation of the cognitive difficulty value to each task. These cognitive difficulty and speed difficulty estimates can then be used within the Bayesian Item Response model to estimate the ability of the learners. The standard measures of error, an Infit and Outfit statistic, will be obtained. These are a form of Chi square analysis between expected (what we know about the task and the learner) and observed behaviour, and are used by assessment stakeholders to judge the stability of the data set.