Can online educational games be scored?

在线教育游戏提供了获得复杂技能的可能性，这些技能很少经过测试，因此很少在主流教育环境中教授。对测试这些技能的需求正在增长，许多人会认为早就应该这样做了。然而，高风险测试依赖于定量性能数据分析技术来提高和改进其测试的准确性和公平性，并且这些技术与遥测数据集中产生的数据类型不兼容。目前，基于游戏的评估领域主要由游戏设计领域的研究人员主导。许多提议使用遥测数据作为评估性能数据的游戏设计者依赖于许多正式或法律的设置中不认可的方法。将游戏分数与另一个外部标准化测试的分数相关联，或者使用专家判断，都过于简化了校准过程。数据分析的贝叶斯方法与项目反应理论兼容，评估员喜欢的校准方法。此外，贝叶斯还处理动态的、有条件依赖的超文本数据。然而，很少有研究已经做了估计的程度，在质量的数据选择升级到贝叶斯;处理缺失数据的方法;最后的数学模型来处理这些数据。该项目旨在探索减少这种偏见的方法，并提出与计算和评估原则兼容的在线游戏数据分析方法。研究问题：1如何在游戏中估计游戏任务的难度和玩家的能力？2.如何在游戏中公平地处理丢失的数据？3.游戏特定变量（时间、迭代和选择）如何被概念化，这些概念化如何影响学习者的排名？研究方法：这是一种对提取的数据集离线进行的定量数据分析。在确定合适的数据集后，将创建各种数据分析模型。在高风险评估中报告给学习者的分数通常是原始数据分数的数学函数，并且这些函数对集中趋势考虑敏感。由于时间限制没有上限，第一阶段将是确定极端分数对其他数据的杠杆权重。这将通过前后评分的相关性、限制极值后的相关性以及删除极值来完成。游戏设计者经常假设时间的物理度量与能力有线性关系。这项研究将采取随机的方法，看到速度，而不是时间，作为一个功能的球员，任务，并可能在等级带，球员已经实现了在该任务。清理后，遥测数据集的有序数据将升级为3种评分方法：累积平均值;升级最高评分;升级最新评分。得到的分数将是有序的，因此第一步是使用对数变换将其转换为区间数据，这可以在贝叶斯中进行。在一定时间内完成任务的总体难度将被给出一个估计值。类似的过程将与等级分数进行，但这些将不会有不受约束的上限和下限所提出的问题。第二个过程将产生对每个任务的认知难度值的估计。这些认知难度和速度难度的估计值可以在贝叶斯项目响应模型中使用，以估计学习者的能力。将获得误差的标准测量值，即Infit和Outfit统计量。这些是预期（我们对任务和学习者的了解）和观察到的行为之间的卡方分析的一种形式，并由评估利益相关者用来判断数据集的稳定性。

Online educational games offer the potential to score complex skills that are rarely tested, and therefore rarely taught in mainstream educational settings. Demand for a way to test these skills is growing, and many would argue overdue. However, high stakes testing relies on quantitative performance data analysis techniques to improve and refine the accuracy and fairness of their tests, and these techniques incompatible with the kind of data produced in telemetry data sets. The field of Game Based Assessment is currently dominated by researchers from the games design field. Many games designers who have proposed the use of telemetry data as assessment performance data have relied on methods not recognised in many formal, or legal, settings. Correlation of gaming scores with grades from another external standardised test, or using expert judgement, both oversimplify the calibration process. Bayesian methods of data analysis are compatible with Item Response Theory, the approach to calibration favoured by assessors. In addition, Bayes also handles dynamic, conditionally dependent hypertext data to be handled. However, little research has been done to estimate the degree of bias in the quality of data selected to escalate to the Bayes; the methods of handling missing data; and finally the mathematical model to process that data. This project aims to explore ways to reduce this bias, and propose ways forward for analysing online game data that are compatible with both computing and assessment principles. Research questions:1 How can the difficulty of the game tasks and the ability of the player be estimated in the game? 2 How can missing data be dealt with fairly in games? 3 How can game-specific variables (time, iteration, and choice) be conceptualized and how might these conceptualizations affect learner ranking? Research approach: This is a quantitative data analysis to be carried out offline on extracted data sets. After the identification of suitable data sets, various data analysis models will be created. The scores that are reported to learners in high stakes assessment are usually a mathematical function of raw data scores, and these functions are sensitive to central tendency consideration. As there is no upper bound to the time limit, the first stage will be to identify the weight of the leverage that extreme scores have on the other data. This will be done through correlation of before and after scores, after capping extreme values, and deleting them. Games designers often assume that the physical measure of time has a linear relationship with ability. This study will instead take a stochastic approach, seeing speed, not time, as a function of the player, the task and possibly on the grade band that the player has achieved in that task. Once cleaned, the ordinal data from the telemetry data sets, will be escalated to 3 scoring approaches: the accumulated mean; escalating the high score; and escalating the most recent score. The resulting scores will be ordinal, and so the first step will be to transform those into interval data using a logarithmic transformation, and this can be down within Bayes. The overall difficulty of completing the tasks within a certain time will be given an estimated value. A similar process will be carried out with the grade scores, but these will not have the issues raised by unconstrained upper and lower boundaries. This second process will produce an estimation of the cognitive difficulty value to each task. These cognitive difficulty and speed difficulty estimates can then be used within the Bayesian Item Response model to estimate the ability of the learners. The standard measures of error, an Infit and Outfit statistic, will be obtained. These are a form of Chi square analysis between expected (what we know about the task and the learner) and observed behaviour, and are used by assessment stakeholders to judge the stability of the data set.