The 2020 GCSE and A-level 'exam grades fiasco': A secondary data analysis of students' grades and Ofqual's algorithm

英国2020年GCSE和A-Level考试成绩的授予被广泛认为是一场“惨败”。当COVID-19迫使考试取消时，DfE和Ofqual要求中心（学校和学院）提交中心评估等级（CAG）和排名。也就是说，教师认为学生参加考试后会达到的成绩和排名。Ofqual的任务是防止分数膨胀并确保评分的一致性，认为学生的CAG过于乐观，因此用通过他们的直接中心水平表现（DCP）算法预测的计算分数取代了他们。结果是，40%的CAG被降级一个或多个等级。公众立即强烈抗议说，学生们被“抢走”了他们应得的分数。媒体很快就报道说，计算出来的成绩对各种学生和学校都有系统性的偏见。其他人则认为，它们不够可靠，在较小的中心，预测准确性特别低。这一愤怒导致政府态度大转变，首相鲍里斯约翰逊宣布Ofqual的DCP方法是一种“突变算法”，教育部长加文威廉姆森指示Ofqual恢复原来的CAG。2021年1月，政府宣布2021年的考试也将被取消，取而代之的是CAG。学生将根据GCSE和A-level成绩被大学录取和就业。他们的成绩直接影响他们的未来。因此，社会必须了解2020年及2021年学生成绩的不公平程度，个别中心及学生和学校的特点可能存在不同的偏见。从惨败中吸取教训也是至关重要的，以帮助通知DfE和Ofqual的反应时，可能再次需要CAG代替考试成绩（例如，由于未来的流行病，教师罢工，考试抵制，泄露试卷，中心渎职，技术故障与考试评估）。更一般地说，我们的研究结果将与那些呼吁在GCSE和A级重新引入课程作业和其他非考试评估的人有关，特别是那些呼吁完全取消考试的人，因为这将意味着对学校和大学评估的永久依赖。因此，我们的总体目标是对2020年和2021年的GCSE和A进行独立和严格的二级数据分析，水平考试成绩，以探索不仅是什么出了问题的统计，但要确定什么可以改善统计预测成绩在未来几年。为了实现这一目标，我们将致力于四个主要目标：1。使用多层次模型探索不同学生和学校在2020年和2021年通过用CAG取代考试而系统性地被忽视或处于不利地位的程度，并评估Ofqual算法在2020年计算成绩是否成功地消除了这种偏见。2.研究我们可以在多大程度上使用多层次模型，更灵活的模型规范和更丰富的数据来改善基于描述性统计的Ofqual算法，以提高预测成绩的整体准确性，并减少学校之间以及学生和学校特征的差异偏差和差异预测准确性。3.通过积极参与和传播我们的研究结果，以学生考试成绩的“生产者”，惨败的关键“评论员”，以及接收和使用成绩的最终“用户”，最大限度地发挥我们的研究的影响。计划的活动包括与学校和Ofqual举行知识交流会议、技术报告、政策简报、新闻稿和互动数据可视化网站。进一步发展早期职业研究员（ECR）和Co-I Lucy Prior博士在她的上升轨迹上成为一名有才华的独立学者，专门从事二级数据分析，以解决教育研究和政策中的关键辩论。

The awarding of the 2020 GCSE and A-Level exam grades in England was widely viewed as a 'fiasco'. When COVID-19 forced the cancellation of exams, DfE and Ofqual asked centres (schools and colleges) to submit Centre Assessment Grades (CAGs) and rankings. Namely, the grades and rank orders within their centres that teachers thought students would have achieved had they sat their exams. Ofqual, tasked with preventing grade inflation and ensuring grading consistency, viewed students' CAGs as overly optimistic and so replaced them with calculated grades predicted via their Direct Centre-level Performance (DCP) algorithm. The result was that 40% of CAGs were downgraded by one or more grades. There was immediate public outcry that students were 'robbed' of the grades they deserved. The media quickly reported that the calculated grades were systematically biased against various students and schools. Others argued that they were not reliable enough, with predictive accuracy especially low in smaller centres. The furore resulted in a government U-turn, Prime Minister Boris Johnson declaring Ofqual's DCP approach a 'mutant algorithm', and Education Secretary Gavin Williamson instructing Ofqual to revert to the original CAGs. In January 2021, the government announced that the 2021 exams will also be cancelled with CAGs used in their place.Students are accepted into universities and employment based on their GCSE and A-level grades. Their grades directly impact their immediate future. It is therefore vitally important for society to understand the extent to which students' grades were unfairly awarded in 2020 and 2021 with biases potentially varying across individual centres and by student and school characteristics. It is also crucial to learn from the fiasco to help inform DfE and Ofqual responses when CAGs might again be needed in place of exam grades (e.g., due to future pandemics, teacher strikes, exam boycotts, leaked exam papers, centre malpractice, technology failures with onscreen assessments). More generally, our findings will be relevant to those calling for a reintroduction of coursework and other non-exam assessments at GCSE and A-level and especially those calling for a removal of exams altogether, since this would imply a permanent reliance on school and college assessments.Our overarching aim is to therefore conduct an independent and rigorous secondary data analysis of the 2020 and 2021 GCSE and A-level exam grades to explore not just what went wrong statistically, but to identify what could be improved statistically when predicting grades in future years. To achieve this aim, we will address four main objectives:1. Explore using multilevel models the extent to which different students and schools were systematically advantaged or disadvantaged by replacing exams with CAGs in 2020 and 2021 and evaluate how successfully or not the Ofqual algorithm calculated grades removed such biases in 2020. 2. Study the degree to which we can use multilevel models, more flexible model specifications, and richer data to improve on the descriptive statistic based Ofqual algorithm with respect to increasing the overall accuracy of predicted grades, and in reducing their differential bias and differential predictive accuracy across schools and by student and school characteristics. 3. Maximise the impact of our research by actively engaging and disseminating our findings to the 'producers' of students' exam grades, the key 'commentators' on the fiasco, and the end 'users' who received and used the grades. Planned activities include knowledge exchange meetings with schools and Ofqual, technical reports, policy briefings, press releases, and an interactive data visualisation website.4. Further develop Early Career Researcher (ECR) and Co-I Dr Lucy Prior on her upwards trajectory to becoming a talented independent academic specialising in secondary data analysis to address key debates in educational research and policy.