Dimensions of Success: Transforming Quality Assessment in Middle School Science and Engineering

该项目代表了K-12科学和工程学习经验质量评估的新方法。通过更新和扩展最初为中学科学和工程教室（DoS-MSSE）非正式科学学习环境建立的成功维度（DoS）观察工具，该项目将为正在学习如何有效和公平地实施K-12下一代科学标准（NGSS）框架的教师创建和实施可持续和可扩展的支持系统。该项目的目标如下。 (1)开发和测试DoS-MSSE观察工具，包括与NGSS和公平性的一致性，以确保所有维度的分数都有效可靠。(2)建立视频评分方法，以增加获得持续改进支持系统的机会。 (3)为学校和地区创建STEM质量档案，为政策决策、专业发展机会、实践指南和其他工作提供信息。 (4)制定和实施培训计划，以认证教师和工作人员使用该工具，并提供专业发展。将发展目标扩大到中学，将通过确定学校在科学和工程学习方面的优势和挑战，使公平更加具体，并通过既定的发展目标支助系统提高学校和教师的能力，支持和促进教育实践。开发这一工具的目的是评估学习经验，以提高质量，而不是用于问责或比较。DoS-MSSE还将通过提供一个经过验证的质量评估工具来支持研究和评估，该工具专门针对中学科学和工程，具有灵活性，可应用于各种提案和项目，增强研究基础设施。最后，它将通过向利益攸关方提供“质量概况”来指导政策和方法，从而改善STEM教育领域。通过促进学校内外公平、高质量的科学和工程学习，扩大的DoS框架将加强跨部门的努力，并帮助社区提供年轻人在劳动力和生活中茁壮成长所需的灵感、知识和技能。在四年内，该项目将通过以下方式改变DoS框架，以解决正规中学环境中的多个教育优先事项：（1）创建一个新的领域，与科学和工程实践，学科核心思想以及NGSS/K-12框架中的交叉概念保持一致;（2）确定可观察和可衡量的多样性、公平、包容和准入做法（和相关指标，包括社会情感学习和发展），并进一步修订评分标准。该项目将使用已建立的观察协议方法为DoS-MSSE建立有效性论证，并使用项目反应理论（IRT）方法（例如，多面Rasch测量，以模拟 数据，这将允许估计教师的有效性，评分员的严重性，以及每个维度的难度）。评分将由内容专家和试点观察提供信息;分数将在维度，观察员，学科领域和时间上进行分析;分数将与其他数据进行比较，包括学生和教师的自我报告以及通过其他既定措施收集的观察分数。该项目最终将进行一项更大规模的验证研究，从三个地理位置不同的学区（N=30所学校）收集观察和调查数据。这项工作将确定课堂质量指标的证据强度，并提供数据，以确定各地区、学校和教师的支持需求。鉴于对远程学习的持续需求，以及普遍需要提高学校进行非惩罚性评估的能力，该工具将得到扩展，以支持实时和录制的数据收集模式，并建立一种视频评分方法。该项目的第四年将对培训和认证系统进行实地测试，以确保可访问性，可扩展性和可持续性。Discovery Research PreK-12计划（DRK-12）旨在通过研究和开发创新资源，模型和工具，显着提高PreK-12学生和教师的科学，技术，工程和数学（STEM）的学习和教学。DRK-12项目中的项目建立在STEM教育的基础研究以及为拟议项目提供理论和经验依据的先前研究和开发工作的基础上。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project represents a new approach to quality assessment of K-12 science and engineering learning experiences. By updating and expanding the Dimensions of Success (DoS) observation tool initially established for informal science learning settings to middle school science and engineering classrooms (DoS-MSSE), the project will create and implement a sustainable and scalable system of support for teachers who are learning how to implement the Next Generation Science Standards (NGSS) Framework for K-12 effectively and equitably. The project’s goals are as follows. (1) Develop and test the DoS-MSSE observation tool, including alignment to the NGSS and equity, to ensure scores are valid and reliable across all dimensions. (2) Establish a video scoring method to increase access to continuous improvement systems of support. (3) Create profiles of STEM quality for schools and districts to inform policy decisions, professional development opportunities, practice guidelines, and other efforts. (4) Develop and implement a training program to certify teachers and staff in the use of the tool and provide professional development. Extending DoS to middle schools will support and advancing educational practice by identifying schools’ strengths and challenges in science and engineering learning, making equity more concrete, and enhancing school and teacher capacity through established DoS systems of support. The tool will be developed to assess learning experiences for quality improvement, not used for accountability or comparison. DoS-MSSE will also support research and evaluation by providing a validated quality assessment tool, specific to middle school science and engineering, that is flexible and can be applied to a variety of proposals and projects, enhancing research infrastructure. Finally, it will improve the STEM education field by providing stakeholders with “quality profiles” to guide policy and approaches. By promoting equitable, high-quality science and engineering learning inside and outside of school, the expanded DoS framework will strengthen efforts across sectors and help communities provide the inspiration, knowledge, and skills youth need to thrive in the workforce and in life.Over four years, this project will transform the DoS framework to address multiple educational priorities for formal middle school settings by: (1) creating a new domain that aligns to science and engineering practices, disciplinary core ideas, and cross-cutting concepts from the NGSS/Framework for K-12; (2) identifying observable and measurable diversity, equity, inclusion, and access practices (and related indicators, including social-emotional learning and development) in the context of science and engineering in classrooms, and further revising rubric criteria. The project will build a validity argument for DoS-MSSE using established methods for observation protocols, and analyze the psychometric properties of all dimensions and domains using item-response theory (IRT) methods (e.g., Many-Faceted Rasch Measurement, to model the data which will allow for estimation of teachers’ effectiveness, rater severity, and the difficulty of each dimension). The rubrics will be informed by content experts and pilot observations; scores will be analyzed across dimensions, observers, disciplinary domains, and time; and scores will be compared to other data, including student and teacher self-report and observation scores collected with other established measures. The project will culminate in a larger-scale validation study, collecting observational and survey data from three geographically different school districts (N=30 schools). This effort will characterize the strength of evidence for indicators of classroom quality and provide data to identify support needs across districts, schools, and teachers. Given the ongoing need for remote learning, and the general need to increase school capacity for non-punitive assessment, the tool will be expanded to support both live and recorded modes of data collection, establishing a video scoring method. A training and certification system will be field-tested in the fourth year of the project to assure accessibility, scalability, and sustainability.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.