How should we measure school performance and hold schools accountable? A study of competing statistical methods and how they compare to Progress 8

点击翻译按钮获取中文摘要

In 2016, the Department for Education radically overhauled their secondary school accountability system and introduced 'Progress 8', arguing it to be the simplest and fairest school performance measure to date. Progress 8 aims to quantify and communicate the average academic value each school adds to their pupils' learning. Specifically, Progress 8 measures how much higher each school's pupils score in their age 16 GCSE examinations than expected given their age 11 KS2 test scores when they started secondary schooling. Progress 8 scores are used to hold schools to account, with the lowest scoring schools judged 'underperforming' and 'coasting'; classifications that trigger intense scrutiny and intervention from the school inspectorate, Ofsted. Given the high-stakes involved, research is urgently needed to first evaluate the strengths and weaknesses of the statistical method underlying Progress 8, and second, to explore the potential benefits of alternative methods for measuring school performance. Our proposed research will address these needs and in doing so will advance scientific understanding about school performance measurement.The first aim of the proposed research is to study how sensitive Progress 8 and other value-added measures' scores, rankings and classifications are to the statistical modelling decisions and assumptions implicit in their design. These include fundamental decisions over whether value-added measures should additionally consider pupil non-academic outcomes and whether they should account for the substantial differences in pupils' demographic and socioeconomic characteristics between schools. We will also explore an assumption implicit in Progress 8 and many other value-added measures which is that there is no relationship between pupils' family backgrounds and the quality of the schools they attend (i.e., confounding or selection into school). We will study the extent to which alternative value-added approaches can address this (using fixed- and random-effects models). We will also explore another assumption of Progress 8 and many other value-added measures which is that the GCSE outcomes of low scoring pupils at KS2 are no more variable than those of high scoring pupils at KS2 (i.e., homoskedasticity). We will study different approaches proposed in the literature to address this concern. The second aim of the proposed research is to adapt and evaluate a range of cutting edge statistical methods not yet applied to measuring school performance. We will study their advantages and disadvantages over standard value-added approaches including Progress 8. These include methods that allow users to compare not just the average progress made by pupils in each school, but to also study the variability in pupils' progress (i.e., mixed-effects location scale models and multilevel quantile regression). These also include methods that are less reliant on extrapolating the relationships in the data and may therefore lead to fairer and more meaningful school comparisons (i.e., propensity score and other matching methods for comparing multiple treatments).Whereas the above aims will advance the academic research in the school effectiveness and methods literatures, our third aim is to make fundamental contributions to the wider non-academic understanding of alternative methods for measuring school performance in England. We will therefore disseminate our findings to the users and producers of school and public-sector institution performance measures. These groups include policy makers and their advisers, statisticians and researchers in relevant Government departments and charities, as well as schools, the media and parents. Planned activities include: knowledge exchange meetings, research briefings, a data visualization website, an end-of-grant symposium, online training materials, and a face-to-face short training course.