Improving school governance and learning outcomes at scale: A randomized evaluation of the Madhya Pradesh School Quality Assessment program

我们建议对发展中国家政府最近在大规模改善教育治理方面最雄心勃勃的尝试之一进行随机对照试验(RCT)。评估将特别侧重于农村部落地区的学校治理状况、进程和学习成果，这一直是改善印度教育的最具挑战性的背景之一。中央邦(MP)学校质量评估(MPSQA)干预是一项综合方案，旨在通过以下组合改善学校治理和成果：(A)定期监测学校；(B)创建学校成绩单；(C)针对该计划的进展情况进行季度跟踪；以及(D)利用信息和通信技术工具收集和报告所有评估结果、学校改进计划、以及一个专门的在线系统的后续行动。PI与项目实施团队合作，并在2015/16学年成功地在约20,000所学校随机推出了该计划。政府目前希望普及该计划，到2019年覆盖MP的所有12万所学校。评估的时机是理想的：发现积极的结果可能有助于加快MP和印度其他地区的规模扩大，而找不到显著的积极结果(以及关于中间因素的数据以了解无影响的原因)，可能会促使在将资源用于可能对过程或结果没有预期影响的规模之前重新评估计划及其实施。评估将评估计划对关键过程指标(例如教师缺课、努力和教学方法)以及学生两年的学术结果的影响。拟议的评估将集中在2015/16年度在全州随机推出该计划，并寻求回答三个核心问题：(A)在偏远部落地区和该州其他地区的公共教育系统中，学校一级的过程、投入和学习结果有何不同？(B)MPSQA干预措施在多大程度上改善了计划学校的学校水平过程和学习结果，偏远部落地区的学校和该州其他地区的学校之间是否存在差异；以及(C)实现和抑制计划效果的途径是什么(例如，改善教师的努力、更好的学校管理、更有针对性的教学实践以及学生的出勤率和努力)？在回答这些问题时，我们将寻求使用广泛的行政数据，包括全州范围内的低风险评估结果，以及通过定期未经宣布的学校访问随机选择的小学和中学子集收集的独立管理的测试和详细的过程指标。这些独立管理的测试将防止在项目学校操纵官方测试结果的可能性，并允许通过仔细的测试设计将测试结果与印度和国际上的其他评估联系起来。核心建议和研究设计基于使用调查数据的定量影响评估方法。然而，认识到大规模的全系统改革往往具有意想不到的方面以及不同主体和不同治理水平之间的多层次互动，我们还提出了一个定性组成部分，寻求对以下方面的补充和细微差别的理解：(A)就教育系统中相关主体(学生、教师、校长和教育公务员)的互动、观念和动机而言，干预的运作背景；(B)方案效果是如何通过教育系统的这些不同层面进行调节的，并了解这种改革的“促成因素”或瓶颈可能是什么。

We propose to conduct a randomized controlled trial (RCT) of one of the most ambitious recent attempts by a developing country government to improve education governance at scale. The evaluation will have a particular focus on the state of school governance, processes and learning outcomes in rural tribal areas, which have been one of the most challenging contexts to improve education in India.The Madhya Pradesh (MP) School Quality Assessment (MPSQA) intervention is a comprehensive program that aims to improve school governance and outcomes through a combination of (a) regular monitoring of schools, (b) creation of school report cards, and a customized school improvement plan, (c) quarterly follow-ups on progress against this plan, and (d) leveraging of ICT tools to collect and report real-time records of all assessment results, school improvement plans, and follow-ups to a dedicated online system. The PI's have worked with the project implementation team and have successfully randomized the rollout of the program across ~20,000 schools in the 2015/16 school year. The government is currently hoping to universalize the program to cover all 120,000 schools in MP by 2019. The timing of the evaluation is ideal: finding positive results may help accelerate the scaling up across MP and elsewhere in India, while not finding significant positive results (along with data on intermediate factors to understand reasons for non-impact) may prompt a re-evaluation of the program and its implementation before spending resources on a scale up that may not have the desired impacts on processes or outcomes.The evaluation will assess the effects of the program on key process metrics (e.g. teacher absence, effort and pedagogy) as well as student academic outcomes for two years. The proposed evaluation will focus on the randomized state-wide rollout of the program in 2015/16 and seek to answer three core questions: (a) How do school-level processes, inputs, and learning outcomes differ within the public education system between remote tribal areas and other parts of the state?; (b) To what extent does the MPSQA intervention improve school-level processes and learning outcomes in the program schools and does this differ between schools in remote tribal areas and other parts of the state; and (c) What are the pathways through which program effects are realized and inhibited (examples include improvements in teacher effort, in better school management, in better-focused pedagogical practices, and student attendance and effort)? In answering these questions, we will seek to use extensive administrative data including state-wide low-stakes assessment results, alongside independently-administered tests and detailed process metrics collected through regular unannounced school visits to a randomly-selected subset of schools at both elementary and secondary levels. These independently-administered tests will guard against the possibility of manipulation of official test results in program schools and also allow for the linking of test results, through careful test design, to other assessments in India and internationally. The core proposal and the study design are based on the use of quantitative impact evaluation methods using survey data. However, recognizing that large system-wide reforms often have unanticipated facets and multiple layers of interaction between different agents and different levels of governance, we also propose a qualitative component which seeks a complementary nuanced understanding of (a) the context in which the intervention operates, in terms of the interactions, perceptions and motivation of the relevant agents in the education system (students, teachers, headmasters and education civil servants) (b) how the program effects are mediated through these various layers of the education system and learning about what the 'enabling factors' or bottlenecks of such reforms might be.