Dimensions of Success: Transforming Quality Assessment in Middle School Science and Engineering

点击翻译按钮获取中文摘要

This project represents a new approach to quality assessment of K-12 science and engineering learning experiences. By updating and expanding the Dimensions of Success (DoS) observation tool initially established for informal science learning settings to middle school science and engineering classrooms (DoS-MSSE), the project will create and implement a sustainable and scalable system of support for teachers who are learning how to implement the Next Generation Science Standards (NGSS) Framework for K-12 effectively and equitably. The project’s goals are as follows. (1) Develop and test the DoS-MSSE observation tool, including alignment to the NGSS and equity, to ensure scores are valid and reliable across all dimensions. (2) Establish a video scoring method to increase access to continuous improvement systems of support. (3) Create profiles of STEM quality for schools and districts to inform policy decisions, professional development opportunities, practice guidelines, and other efforts. (4) Develop and implement a training program to certify teachers and staff in the use of the tool and provide professional development. Extending DoS to middle schools will support and advancing educational practice by identifying schools’ strengths and challenges in science and engineering learning, making equity more concrete, and enhancing school and teacher capacity through established DoS systems of support. The tool will be developed to assess learning experiences for quality improvement, not used for accountability or comparison. DoS-MSSE will also support research and evaluation by providing a validated quality assessment tool, specific to middle school science and engineering, that is flexible and can be applied to a variety of proposals and projects, enhancing research infrastructure. Finally, it will improve the STEM education field by providing stakeholders with “quality profiles” to guide policy and approaches. By promoting equitable, high-quality science and engineering learning inside and outside of school, the expanded DoS framework will strengthen efforts across sectors and help communities provide the inspiration, knowledge, and skills youth need to thrive in the workforce and in life.Over four years, this project will transform the DoS framework to address multiple educational priorities for formal middle school settings by: (1) creating a new domain that aligns to science and engineering practices, disciplinary core ideas, and cross-cutting concepts from the NGSS/Framework for K-12; (2) identifying observable and measurable diversity, equity, inclusion, and access practices (and related indicators, including social-emotional learning and development) in the context of science and engineering in classrooms, and further revising rubric criteria. The project will build a validity argument for DoS-MSSE using established methods for observation protocols, and analyze the psychometric properties of all dimensions and domains using item-response theory (IRT) methods (e.g., Many-Faceted Rasch Measurement, to model the data which will allow for estimation of teachers’ effectiveness, rater severity, and the difficulty of each dimension). The rubrics will be informed by content experts and pilot observations; scores will be analyzed across dimensions, observers, disciplinary domains, and time; and scores will be compared to other data, including student and teacher self-report and observation scores collected with other established measures. The project will culminate in a larger-scale validation study, collecting observational and survey data from three geographically different school districts (N=30 schools). This effort will characterize the strength of evidence for indicators of classroom quality and provide data to identify support needs across districts, schools, and teachers. Given the ongoing need for remote learning, and the general need to increase school capacity for non-punitive assessment, the tool will be expanded to support both live and recorded modes of data collection, establishing a video scoring method. A training and certification system will be field-tested in the fourth year of the project to assure accessibility, scalability, and sustainability.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.