Can online educational games be scored?

点击翻译按钮获取中文摘要

Online educational games offer the potential to score complex skills that are rarely tested, and therefore rarely taught in mainstream educational settings. Demand for a way to test these skills is growing, and many would argue overdue. However, high stakes testing relies on quantitative performance data analysis techniques to improve and refine the accuracy and fairness of their tests, and these techniques incompatible with the kind of data produced in telemetry data sets. The field of Game Based Assessment is currently dominated by researchers from the games design field. Many games designers who have proposed the use of telemetry data as assessment performance data have relied on methods not recognised in many formal, or legal, settings. Correlation of gaming scores with grades from another external standardised test, or using expert judgement, both oversimplify the calibration process. Bayesian methods of data analysis are compatible with Item Response Theory, the approach to calibration favoured by assessors. In addition, Bayes also handles dynamic, conditionally dependent hypertext data to be handled. However, little research has been done to estimate the degree of bias in the quality of data selected to escalate to the Bayes; the methods of handling missing data; and finally the mathematical model to process that data. This project aims to explore ways to reduce this bias, and propose ways forward for analysing online game data that are compatible with both computing and assessment principles. Research questions:1 How can the difficulty of the game tasks and the ability of the player be estimated in the game? 2 How can missing data be dealt with fairly in games? 3 How can game-specific variables (time, iteration, and choice) be conceptualized and how might these conceptualizations affect learner ranking? Research approach: This is a quantitative data analysis to be carried out offline on extracted data sets. After the identification of suitable data sets, various data analysis models will be created. The scores that are reported to learners in high stakes assessment are usually a mathematical function of raw data scores, and these functions are sensitive to central tendency consideration. As there is no upper bound to the time limit, the first stage will be to identify the weight of the leverage that extreme scores have on the other data. This will be done through correlation of before and after scores, after capping extreme values, and deleting them. Games designers often assume that the physical measure of time has a linear relationship with ability. This study will instead take a stochastic approach, seeing speed, not time, as a function of the player, the task and possibly on the grade band that the player has achieved in that task. Once cleaned, the ordinal data from the telemetry data sets, will be escalated to 3 scoring approaches: the accumulated mean; escalating the high score; and escalating the most recent score. The resulting scores will be ordinal, and so the first step will be to transform those into interval data using a logarithmic transformation, and this can be down within Bayes. The overall difficulty of completing the tasks within a certain time will be given an estimated value. A similar process will be carried out with the grade scores, but these will not have the issues raised by unconstrained upper and lower boundaries. This second process will produce an estimation of the cognitive difficulty value to each task. These cognitive difficulty and speed difficulty estimates can then be used within the Bayesian Item Response model to estimate the ability of the learners. The standard measures of error, an Infit and Outfit statistic, will be obtained. These are a form of Chi square analysis between expected (what we know about the task and the learner) and observed behaviour, and are used by assessment stakeholders to judge the stability of the data set.