How should we measure school performance and hold schools accountable? A study of competing statistical methods and how they compare to Progress 8

2016年，教育部对中学问责制度进行了彻底改革，推出了“进步8”，认为这是迄今为止最简单、最公平的学校表现衡量标准。进步8旨在量化和传达每所学校为学生学习增加的平均学术价值。具体来说，进步8衡量的是，每所学校的学生在16岁的GCSE考试中的成绩比他们开始上中学时11岁的KS2考试成绩高出多少。进步8的分数被用来追究学校的责任，得分最低的学校被评为“表现不佳”和“一帆风顺”；这些分类引发了学校督学机构Ofsted的严格审查和干预。鉴于涉及的风险很高，迫切需要进行研究，首先评估作为Progress 8基础的统计方法的优点和缺点，其次探索衡量学校表现的替代方法的潜在好处。我们的拟议研究将满足这些需求，并通过这样做将促进对学校表现衡量的科学理解。拟议研究的第一个目的是研究Progress 8和其他增值衡量标准的分数、排名和分类对其设计中隐含的统计建模决策和假设的敏感性。这些问题包括关于增值措施是否应另外考虑学生的非学业成绩以及这些措施是否应考虑到不同学校之间学生的人口统计和社会经济特征的重大差异的根本决定。我们还将探讨进步8和许多其他增值措施中隐含的一个假设，即学生的家庭背景与他们就读的学校的质量之间没有关系(即混淆或选择入学)。我们将研究替代增值方法能够在多大程度上解决这个问题(使用固定效应和随机效应模型)。我们还将探讨进步8和许多其他增值措施的另一个假设，即KS2成绩低的学生的GCSE成绩并不比KS2成绩高的学生的GCSE成绩更具变数(即同质性)。我们将研究文献中提出的解决这一问题的不同方法。拟议研究的第二个目标是调整和评估一系列尚未应用于衡量学校表现的尖端统计方法。我们将研究它们与包括Progress 8在内的标准增值方法的优缺点。这些方法包括允许用户不仅比较每所学校学生的平均进步，而且还研究学生进步的变异性的方法(即混合影响位置量表模型和多级分位数回归)。这些方法也包括较少依赖于外推数据中的关系的方法，因此可能导致更公平和更有意义的学校比较(即倾向分数和其他匹配方法，用于比较多种处理)。尽管上述目的将促进学校有效性和方法文献的学术研究，但我们的第三个目标是为更广泛的非学术理解英国学校表现的替代方法做出基本贡献。因此，我们将向学校和公共部门机构绩效衡量的使用者和生产者传播我们的研究结果。这些群体包括政策制定者及其顾问、相关政府部门和慈善机构的统计学家和研究人员以及学校、媒体和家长。计划的活动包括：知识交流会议、研究简报、数据可视化网站、赠款终止研讨会、在线培训材料和面对面的短期培训课程。

In 2016, the Department for Education radically overhauled their secondary school accountability system and introduced 'Progress 8', arguing it to be the simplest and fairest school performance measure to date. Progress 8 aims to quantify and communicate the average academic value each school adds to their pupils' learning. Specifically, Progress 8 measures how much higher each school's pupils score in their age 16 GCSE examinations than expected given their age 11 KS2 test scores when they started secondary schooling. Progress 8 scores are used to hold schools to account, with the lowest scoring schools judged 'underperforming' and 'coasting'; classifications that trigger intense scrutiny and intervention from the school inspectorate, Ofsted. Given the high-stakes involved, research is urgently needed to first evaluate the strengths and weaknesses of the statistical method underlying Progress 8, and second, to explore the potential benefits of alternative methods for measuring school performance. Our proposed research will address these needs and in doing so will advance scientific understanding about school performance measurement.The first aim of the proposed research is to study how sensitive Progress 8 and other value-added measures' scores, rankings and classifications are to the statistical modelling decisions and assumptions implicit in their design. These include fundamental decisions over whether value-added measures should additionally consider pupil non-academic outcomes and whether they should account for the substantial differences in pupils' demographic and socioeconomic characteristics between schools. We will also explore an assumption implicit in Progress 8 and many other value-added measures which is that there is no relationship between pupils' family backgrounds and the quality of the schools they attend (i.e., confounding or selection into school). We will study the extent to which alternative value-added approaches can address this (using fixed- and random-effects models). We will also explore another assumption of Progress 8 and many other value-added measures which is that the GCSE outcomes of low scoring pupils at KS2 are no more variable than those of high scoring pupils at KS2 (i.e., homoskedasticity). We will study different approaches proposed in the literature to address this concern. The second aim of the proposed research is to adapt and evaluate a range of cutting edge statistical methods not yet applied to measuring school performance. We will study their advantages and disadvantages over standard value-added approaches including Progress 8. These include methods that allow users to compare not just the average progress made by pupils in each school, but to also study the variability in pupils' progress (i.e., mixed-effects location scale models and multilevel quantile regression). These also include methods that are less reliant on extrapolating the relationships in the data and may therefore lead to fairer and more meaningful school comparisons (i.e., propensity score and other matching methods for comparing multiple treatments).Whereas the above aims will advance the academic research in the school effectiveness and methods literatures, our third aim is to make fundamental contributions to the wider non-academic understanding of alternative methods for measuring school performance in England. We will therefore disseminate our findings to the users and producers of school and public-sector institution performance measures. These groups include policy makers and their advisers, statisticians and researchers in relevant Government departments and charities, as well as schools, the media and parents. Planned activities include: knowledge exchange meetings, research briefings, a data visualization website, an end-of-grant symposium, online training materials, and a face-to-face short training course.