The 2020 GCSE and A-level 'exam grades fiasco': A secondary data analysis of students' grades and Ofqual's algorithm

2020 年 GCSE 和 A-level“考试成绩惨败”：对学生成绩和 Ofqual 算法的二次数据分析

• 批准号: ES/W000555/1
• 负责人: George Leckie
• 金额: $ 30.84万
• 依托单位: University of Bristol
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2022
• 资助国家: 英国
• 起止时间: 2022 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=ES%2FW000555%2F1
• 关键词: 2020; GCSE; level; exam; grades

The awarding of the 2020 GCSE and A-Level exam grades in England was widely viewed as a 'fiasco'. When COVID-19 forced the cancellation of exams, DfE and Ofqual asked centres (schools and colleges) to submit Centre Assessment Grades (CAGs) and rankings. Namely, the grades and rank orders within their centres that teachers thought students would have achieved had they sat their exams. Ofqual, tasked with preventing grade inflation and ensuring grading consistency, viewed students' CAGs as overly optimistic and so replaced them with calculated grades predicted via their Direct Centre-level Performance (DCP) algorithm. The result was that 40% of CAGs were downgraded by one or more grades. There was immediate public outcry that students were 'robbed' of the grades they deserved. The media quickly reported that the calculated grades were systematically biased against various students and schools. Others argued that they were not reliable enough, with predictive accuracy especially low in smaller centres. The furore resulted in a government U-turn, Prime Minister Boris Johnson declaring Ofqual's DCP approach a 'mutant algorithm', and Education Secretary Gavin Williamson instructing Ofqual to revert to the original CAGs. In January 2021, the government announced that the 2021 exams will also be cancelled with CAGs used in their place.Students are accepted into universities and employment based on their GCSE and A-level grades. Their grades directly impact their immediate future. It is therefore vitally important for society to understand the extent to which students' grades were unfairly awarded in 2020 and 2021 with biases potentially varying across individual centres and by student and school characteristics. It is also crucial to learn from the fiasco to help inform DfE and Ofqual responses when CAGs might again be needed in place of exam grades (e.g., due to future pandemics, teacher strikes, exam boycotts, leaked exam papers, centre malpractice, technology failures with onscreen assessments). More generally, our findings will be relevant to those calling for a reintroduction of coursework and other non-exam assessments at GCSE and A-level and especially those calling for a removal of exams altogether, since this would imply a permanent reliance on school and college assessments.Our overarching aim is to therefore conduct an independent and rigorous secondary data analysis of the 2020 and 2021 GCSE and A-level exam grades to explore not just what went wrong statistically, but to identify what could be improved statistically when predicting grades in future years. To achieve this aim, we will address four main objectives:1. Explore using multilevel models the extent to which different students and schools were systematically advantaged or disadvantaged by replacing exams with CAGs in 2020 and 2021 and evaluate how successfully or not the Ofqual algorithm calculated grades removed such biases in 2020. 2. Study the degree to which we can use multilevel models, more flexible model specifications, and richer data to improve on the descriptive statistic based Ofqual algorithm with respect to increasing the overall accuracy of predicted grades, and in reducing their differential bias and differential predictive accuracy across schools and by student and school characteristics. 3. Maximise the impact of our research by actively engaging and disseminating our findings to the 'producers' of students' exam grades, the key 'commentators' on the fiasco, and the end 'users' who received and used the grades. Planned activities include knowledge exchange meetings with schools and Ofqual, technical reports, policy briefings, press releases, and an interactive data visualisation website.4. Further develop Early Career Researcher (ECR) and Co-I Dr Lucy Prior on her upwards trajectory to becoming a talented independent academic specialising in secondary data analysis to address key debates in educational research and policy.

英国2020年的GCSE和A-Level考试成绩被广泛认为是一场“惨败”。当新冠肺炎强制取消考试时，DFE和Ofqual要求中心(学校和大学)提交中心评估成绩(CAG)和排名。也就是说，他们中心内的分数和排名顺序，老师们认为如果学生参加考试，他们就会取得这样的成绩。Ofqual的任务是防止分数膨胀和确保分数一致性，该公司认为学生的CAG过于乐观，因此用通过直接中心水平表现(DCP)算法预测的计算成绩取代了这些分数。结果是，40%的CAG被降级一个或多个等级。立即引起了公众的强烈抗议，认为学生们被“剥夺”了他们应得的分数。媒体迅速报道称，计算出的分数对不同的学生和学校存在系统性的偏见。其他人则认为，它们不够可靠，预测准确率在较小的中心尤其低。这场风波导致了政府的180度大转弯，英国首相鲍里斯·约翰逊宣布Ofqual的DCP方法是一种“变异算法”，教育部长加文·威廉姆森指示Ofqual恢复原来的CAG。2021年1月，政府宣布2021年的考试也将取消，取而代之的是CAG。学生根据他们的GCSE和A-Level成绩被大学录取和就业。他们的成绩直接影响到他们近期的未来。因此，对于社会来说，了解2020年和2021年学生成绩被不公平地给予的程度至关重要，各个中心以及学生和学校的特点可能存在不同的偏见。同样重要的是，从这场惨败中吸取教训，帮助DFE和Ofqual在可能再次需要CAG来取代考试成绩时(例如，由于未来的流行病、教师罢工、考试抵制、试卷泄露、中心失职、屏幕评估的技术故障)，向DFE和Ofqual提供信息。更广泛地说，我们的发现将与那些呼吁重新引入GCSE和A-Level的课程作业和其他非考试评估的人有关，特别是那些呼吁完全取消考试的人，因为这将意味着永久依赖学校和大学的评估。因此，我们的总体目标是对2020和2021年GCSE和A-Level考试的成绩进行独立和严格的二级数据分析，不仅探索统计上的错误，而且确定在预测未来几年的成绩时，哪些方面可以在统计上有所改进。为了实现这一目标，我们将解决四个主要目标：1.使用多水平模型，通过在2020年和2021年用CAG取代考试，探索不同学生和学校在系统上处于优势或劣势的程度，并评估Ofqual算法计算分数的成功与否，以消除2020年的此类偏见。2.研究我们可以在多大程度上使用多水平模型、更灵活的模型规范和更丰富的数据来改进基于描述统计的Ofqual算法，以提高预测成绩的总体准确性，并降低它们在不同学校和不同学生和学校特征之间的差异偏差和差异预测精度。3.通过积极参与并将我们的研究结果传播给学生考试成绩的“生产者”、对这场惨败的关键“评论员”以及接受和使用成绩的最终“用户”，最大限度地发挥我们研究的影响。计划的活动包括与学校和Ofqual的知识交流会、技术报告、政策简报、新闻稿和一个交互式数据可视化网站。进一步发展早期职业研究人员(ECR)和合作-I露西·普赖尔博士的上升轨道，成为一名有才华的独立学者，专门从事二次数据分析，以解决教育研究和政策中的关键辩论。

项目成果

Student Sociodemographic and School Type Differences in Teacher-Predicted vs. Achieved Grades for University Admission

学生社会人口统计和学校类型在大学入学教师预测成绩与实际成绩方面的差异

• DOI: 10.31235/osf.io/u3mz9
• 发表时间: 2023
• 作者: Leckie G

School differences on whether and where students apply to university

学生是否以及在哪里申请大学的学校差异

• 发表时间: 2023
• 作者: Prior L