Rigorous Measures of Implementation: A Suite of Tools for Evaluating STEM Instructional Materials Use

芝加哥大学初等数学和科学教育中心（CEMSE）将进行一项为期两年的全面研究，通过探索这些工具的实际使用，来证明、进一步发展和测试实现保真度（FOI）工具套件所基于的理论。这项混合方法研究源于CEMSE过去由美国国家科学基金会资助的研究，重点是开发创新工具，以评估基于改革的K-8数学和科学教学材料计划的实施情况。该项目的目标是：1)使仪器适应当地发展的科学计划的操作过程；2)通过现场测试进一步完善和发展FOI仪器；3)探索评估项目实施对学生成绩影响的工具使用方法；4)通过与实践者、评估者和研究人员进行在线和基于站点的协作讨论，分享、传播和交流有关科学和数学教学材料实施的测量知识。该项目将涉及位于伊利诺伊州香槟市、伊利诺伊州内珀维尔市和马萨诸塞州牛顿市的三个中型学区，这些学区使用日常数学（EM）和基于改革的科学课程。在参与调查的地区，调查人员将采用有目的的抽样策略，选择在学校规模、学生人口统计和/或学生成绩方面最能代表该地区的17至20所小学。在第一年，所有参与学校的小学五年级教师和学校领导都将使用现有的信息自由工具。还将对这些工具进行调整、实地测试和修订，以衡量科学课程的实施情况。在二年级，将向参与的教师和学校领导发放一套经修订的科学信息自由工具。数据分析将包括探索EM和科学材料实施的现状和类型。标准化的州考试成绩和等级将在所有参与的教室中收集，以将使用的教学材料的状态和类型与学生的成绩联系起来。这项研究的基础是一种信念，即只有当评估工具和数据对目标受众可用和有用时，改进教育创新实施评估的努力才真正有效。该领域持续需要研究工具来描述、测量和解释数学和科学教学材料实施的具体方面，以了解其有效性，并为STEM教育的持续改进工作提供信息。对数学和科学教学材料进行更严格衡量的需求也与学校和地区领导人寻求将有效的课程引入他们的学校特别相关。这些信息自由工具将使利益攸关方能够获得特定程度的数据，这些数据可以为正在进行的改进提供信息，并确定其课程中导致预期实践和结果的要素。本研究将通过进一步了解如何在学校环境中应用创新的信息自由测量方法，同时创造和分享有关评估STEM教学材料实施情况的知识，最终为STEM教育的更广泛领域做出贡献。

The Center for Elementary Mathematics and Science Education (CEMSE) at the University of Chicago will conduct a full-scale 2-year study to demonstrate and further develop and test the theory upon which the suite of Fidelity of Implementation (FOI) instruments is based by exploring the practical use of these tools. This mixed-method study grows out of CEMSE's past NSF-funded studies focusing on the development of innovative tools for evaluating the implementation of reform-based K-8 mathematics and science instructional materials programs. The goals of this project are: 1) operationalizing the process of instrument adaptation to a locally-developed science program; 2) further refining and developing the FOI instruments through field testing; 3) exploring methodologies for instrument use in evaluating the effect of program implementation on student outcomes; and 4) sharing, disseminating, and communicating knowledge about measurement of implementation of science and mathematics instructional materials through both online and site-based collaborative discussions with practitioners, evaluators and researchers.This project will involve three mid-sized school districts located in Champaign, IL, Naperville, IL, and Newton, MA, that use Everyday Mathematics (EM) and a reform-based science curriculum. Within the participating districts, the investigators will use a purposeful sampling strategy to select 17 to 20 elementary schools that best represent the district in terms of school size, student demographics, and/or student achievement. In year 1, the existing FOI instruments for EM will be administered to all K-5 teachers and school leaders in the participating schools. The instruments will also be adapted, field tested, and revised for measurement of implementation of the science curriculum. In Year 2, a revised set of FOI instruments for science will be administered to participating teachers and school leaders. Data analysis will include exploring the status and types of EM and science materials implementation. Standardized state test scores and grades will be collected within all participating classrooms to link status and types of instructional materials use to student achievement. Underlying this study is a belief that efforts to improve evaluation of the implementation of educational innovations are only truly effective when evaluation tools and data are usable and useful to the audiences for which they are intended. There exists a continuing need in the field for research instruments that describe, measure, and explain specific aspects of mathematics and science instructional materials implementation in order to understand their effectiveness and to inform ongoing improvement efforts in STEM education. The need for more rigorous measurement of mathematics and science instructional materials is also particularly relevant to school and district leaders seeking to bring effective programs to their schools. These FOI instruments will enable stakeholders to obtain data at a level of specificity that can inform ongoing improvements and identify elements of their curricula that are leading to the desired practices and outcomes. This study will ultimately contribute to the broader field of STEM education by furthering an understanding of the ways in which an innovative approach to FOI measurement may be applied in school settings while creating and sharing knowledge about the evaluation of implementation of STEM instructional materials.