The 2020 GCSE and A-level 'exam grades fiasco': A secondary data analysis of students' grades and Ofqual's algorithm

点击翻译按钮获取中文摘要

The awarding of the 2020 GCSE and A-Level exam grades in England was widely viewed as a 'fiasco'. When COVID-19 forced the cancellation of exams, DfE and Ofqual asked centres (schools and colleges) to submit Centre Assessment Grades (CAGs) and rankings. Namely, the grades and rank orders within their centres that teachers thought students would have achieved had they sat their exams. Ofqual, tasked with preventing grade inflation and ensuring grading consistency, viewed students' CAGs as overly optimistic and so replaced them with calculated grades predicted via their Direct Centre-level Performance (DCP) algorithm. The result was that 40% of CAGs were downgraded by one or more grades. There was immediate public outcry that students were 'robbed' of the grades they deserved. The media quickly reported that the calculated grades were systematically biased against various students and schools. Others argued that they were not reliable enough, with predictive accuracy especially low in smaller centres. The furore resulted in a government U-turn, Prime Minister Boris Johnson declaring Ofqual's DCP approach a 'mutant algorithm', and Education Secretary Gavin Williamson instructing Ofqual to revert to the original CAGs. In January 2021, the government announced that the 2021 exams will also be cancelled with CAGs used in their place.Students are accepted into universities and employment based on their GCSE and A-level grades. Their grades directly impact their immediate future. It is therefore vitally important for society to understand the extent to which students' grades were unfairly awarded in 2020 and 2021 with biases potentially varying across individual centres and by student and school characteristics. It is also crucial to learn from the fiasco to help inform DfE and Ofqual responses when CAGs might again be needed in place of exam grades (e.g., due to future pandemics, teacher strikes, exam boycotts, leaked exam papers, centre malpractice, technology failures with onscreen assessments). More generally, our findings will be relevant to those calling for a reintroduction of coursework and other non-exam assessments at GCSE and A-level and especially those calling for a removal of exams altogether, since this would imply a permanent reliance on school and college assessments.Our overarching aim is to therefore conduct an independent and rigorous secondary data analysis of the 2020 and 2021 GCSE and A-level exam grades to explore not just what went wrong statistically, but to identify what could be improved statistically when predicting grades in future years. To achieve this aim, we will address four main objectives:1. Explore using multilevel models the extent to which different students and schools were systematically advantaged or disadvantaged by replacing exams with CAGs in 2020 and 2021 and evaluate how successfully or not the Ofqual algorithm calculated grades removed such biases in 2020. 2. Study the degree to which we can use multilevel models, more flexible model specifications, and richer data to improve on the descriptive statistic based Ofqual algorithm with respect to increasing the overall accuracy of predicted grades, and in reducing their differential bias and differential predictive accuracy across schools and by student and school characteristics. 3. Maximise the impact of our research by actively engaging and disseminating our findings to the 'producers' of students' exam grades, the key 'commentators' on the fiasco, and the end 'users' who received and used the grades. Planned activities include knowledge exchange meetings with schools and Ofqual, technical reports, policy briefings, press releases, and an interactive data visualisation website.4. Further develop Early Career Researcher (ECR) and Co-I Dr Lucy Prior on her upwards trajectory to becoming a talented independent academic specialising in secondary data analysis to address key debates in educational research and policy.