Rigorous Measures of Implementation: A Suite of Tools for Evaluating STEM Instructional Materials Use

芝加哥大学基础数学和科学教育中心(CEMSE)将进行一项为期两年的全面研究，通过探索这些工具的实际使用来演示、进一步开发和测试Fidelity of Implementation(FOI)工具套件所基于的理论。这项混合方法研究源于CEMSE过去由NSF资助的研究，重点是开发创新工具来评估基于改革的K-8数学和科学教材计划的实施情况。这个项目的目标是：1)使仪器适应本地开发的科学计划的过程具有可操作性；2)通过实地测试进一步完善和发展FOI仪器；3)探索在评估计划实施对学生结果的影响时使用仪器的方法；4)通过与实践者、评估者和研究人员的在线和现场协作讨论，共享、传播和交流关于科学和数学教材实施的衡量标准的知识。该项目将涉及位于伊利诺伊州香槟、伊利诺伊州内珀维尔和马萨诸塞州牛顿市的三个中型学区，它们使用日常数学(EM)和基于改革的科学课程。在参与的学区内，调查人员将使用有目的的抽样策略，选择在学校规模、学生人口统计和/或学生成绩方面最能代表该学区的17至20所小学。在第一年，现有的针对EM的FOI工具将应用于参与学校的所有K-5教师和学校领导。还将对这些工具进行改编、实地测试和修订，以衡量科学课程的实施情况。在第二年，一套修订的FOI科学仪器将被应用于参与的教师和学校领导。数据分析将包括探索EM和科学材料实施的状况和类型。标准化的州测试成绩和成绩将在所有参与的教室内收集，以将教学材料的使用状况和类型与学生的成绩联系起来。这项研究的基础是，只有在评估工具和数据对目标受众有用和有用的情况下，改进对教育创新实施情况的评估的努力才真正有效。在这一领域，仍然需要描述、衡量和解释数学和科学教材实施的具体方面的研究工具，以便了解其有效性，并为正在进行的STEM教育改进工作提供信息。对数学和科学教材进行更严格测量的需要，对于寻求将有效计划带到学校的学校和地区领导人来说也特别相关。这些FOI工具将使利益攸关方能够获得特定级别的数据，这些数据可以为正在进行的改进提供信息，并确定其课程中导致预期实践和结果的要素。这项研究最终将有助于更广泛的STEM教育领域，通过进一步了解如何在学校环境中应用创新的FOI测量方法，同时创建和共享关于STEM教材实施评估的知识。

The Center for Elementary Mathematics and Science Education (CEMSE) at the University of Chicago will conduct a full-scale 2-year study to demonstrate and further develop and test the theory upon which the suite of Fidelity of Implementation (FOI) instruments is based by exploring the practical use of these tools. This mixed-method study grows out of CEMSE's past NSF-funded studies focusing on the development of innovative tools for evaluating the implementation of reform-based K-8 mathematics and science instructional materials programs. The goals of this project are: 1) operationalizing the process of instrument adaptation to a locally-developed science program; 2) further refining and developing the FOI instruments through field testing; 3) exploring methodologies for instrument use in evaluating the effect of program implementation on student outcomes; and 4) sharing, disseminating, and communicating knowledge about measurement of implementation of science and mathematics instructional materials through both online and site-based collaborative discussions with practitioners, evaluators and researchers.This project will involve three mid-sized school districts located in Champaign, IL, Naperville, IL, and Newton, MA, that use Everyday Mathematics (EM) and a reform-based science curriculum. Within the participating districts, the investigators will use a purposeful sampling strategy to select 17 to 20 elementary schools that best represent the district in terms of school size, student demographics, and/or student achievement. In year 1, the existing FOI instruments for EM will be administered to all K-5 teachers and school leaders in the participating schools. The instruments will also be adapted, field tested, and revised for measurement of implementation of the science curriculum. In Year 2, a revised set of FOI instruments for science will be administered to participating teachers and school leaders. Data analysis will include exploring the status and types of EM and science materials implementation. Standardized state test scores and grades will be collected within all participating classrooms to link status and types of instructional materials use to student achievement. Underlying this study is a belief that efforts to improve evaluation of the implementation of educational innovations are only truly effective when evaluation tools and data are usable and useful to the audiences for which they are intended. There exists a continuing need in the field for research instruments that describe, measure, and explain specific aspects of mathematics and science instructional materials implementation in order to understand their effectiveness and to inform ongoing improvement efforts in STEM education. The need for more rigorous measurement of mathematics and science instructional materials is also particularly relevant to school and district leaders seeking to bring effective programs to their schools. These FOI instruments will enable stakeholders to obtain data at a level of specificity that can inform ongoing improvements and identify elements of their curricula that are leading to the desired practices and outcomes. This study will ultimately contribute to the broader field of STEM education by furthering an understanding of the ways in which an innovative approach to FOI measurement may be applied in school settings while creating and sharing knowledge about the evaluation of implementation of STEM instructional materials.