Improving Evaluations of STEM Programs: An Empirical Investigation of Key Design Parameters

改进 STEM 项目的评估：关键设计参数的实证研究

• 批准号: 2000388
• 负责人: Andrea Beach
• 金额: $ 132.81万
• 依托单位: Western Michigan University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-06-15 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2000388&HistoricalAwards=false
• 关键词: Improving; Evaluations; STEM; Programs; Empirical

To improve science, technology, engineering, and mathematics (STEM) outcomes in K-12 classrooms, it is critical to understand the landscape of the STEM learning environment. However, the STEM learning environment is complex. Students are nested within teachers, and teachers are nested within schools (which in turn are nested within districts), which implies a multilevel structure. To date, most empirical research that uses multilevel modeling to examine the role of higher levels on variation in student outcomes does not examine the teacher level. The reason is that for many states, data linkages between students and teachers have been difficult to achieve. However, in the last five years, this situation has improved in many states, which makes this work now possible. This study seeks to further understanding of the STEM learning environment by 1) examining the extent to which mathematics and science achievement varies across students, teachers, schools, and districts and 2) examining the extent to which student, teacher, school, and district characteristics that are found in state administrative databases can be used to explain this variation at each level. This work will support advances in research and evaluation methodologies that will enable researchers to design more rigorous and comprehensive evaluations of STEM interventions and improve the accuracy of statistical power calculations. Broad dissemination of the resulting tools and techniques will provide access through freely available websites, and workshops and training opportunities to build capacity in the field for STEM researchers to design cluster randomized trials (CRTs) to answer questions beyond what works to for whom and under what conditions. This project is funded by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development. Funding is also provided by the Discovery Research K-12 program (DRK-12), which supports the research and development of innovative resources, models, and tools in order to enhance STEM learning and teaching by pre-K-12 students and teachers.This project will contribute to 1) describing and explaining the landscape of the STEM learning environment, an environment which includes students, teachers, and schools, and 2) applying this empirical information in the design of STEM intervention studies to enable researchers to extend beyond the usual questions about if the intervention works and for which types of students or schools. By adding teacher level variables, this analysis would account for key teacher characteristics that may moderate the treatment effect. The research will also increase the efficiency in the design of CRTs of STEM interventions. Specifically, the findings from this study will improve the internal validity and cost-efficiency of evaluations of STEM interventions by increasing the accuracy of estimates for the full range of parameters needed to conduct power analyses, particularly when the teacher level is included. The high cost associated with CRTs makes it critical to plan accurate trials that do not overestimate the required sample size, and hence cost more than necessary, or underestimate the sample size and thereby reduce the potential to generate high-quality evidence of program effectiveness. Including the teacher level in intervention studies, a critical level in the delivery of any intervention, will allow for more testing of teacher characteristics that may moderate intervention effects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

要提高K-12课堂的科学、技术、工程和数学(STEM)成绩，了解STEM学习环境是至关重要的。然而，STEM的学习环境是复杂的。学生嵌套在教师内部，教师嵌套在学校内(学校又嵌套在地区内)，这意味着一个多层次的结构。到目前为止，大多数使用多水平模型来检验较高水平对学生成绩差异的作用的实证研究并没有考察教师水平。原因是，对于许多州来说，学生和教师之间的数据联系一直很难实现。然而，在过去的五年里，这种情况在许多州都有所改善，这使得这项工作现在成为可能。这项研究试图通过以下方式进一步了解STEM的学习环境：1)检查学生、教师、学校和学区之间数学和科学成绩的差异程度；2)检查在国家行政数据库中发现的学生、教师、学校和地区特征在多大程度上可以用来解释每个级别的这种差异。这项工作将支持研究和评价方法方面的进展，使研究人员能够对STEM干预措施设计更严格和更全面的评价，并提高统计能力计算的准确性。所产生的工具和技术的广泛传播将通过免费提供的网站、讲习班和培训机会，为STEM研究人员提供实地能力建设的机会，以设计分组随机试验(CRT)，以回答什么对谁有效以及在什么条件下有效以外的问题。该项目由EHR核心研究(ECR)计划资助，该计划支持推进STEM学习和学习环境的基础研究、扩大STEM的参与以及STEM劳动力发展的工作。发现研究K-12计划(DRK-12)也提供了资金，该计划支持创新资源、模式和工具的研究和开发，以加强K-12之前的学生和教师的STEM学习和教学。该项目将有助于1)描述和解释STEM学习环境的景观，一个包括学生、教师和学校的环境，以及2)在STEM干预研究的设计中应用这些经验信息，使研究人员能够超越通常的问题，如干预是否有效以及针对哪些类型的学生或学校。通过增加教师水平的变量，这一分析将解释可能缓和治疗效果的关键教师特征。该研究还将提高STEM干预措施CRT的设计效率。具体地说，这项研究的结果将提高对进行权力分析所需的所有参数的估计的准确性，特别是在包括教师一级的情况下，从而提高对STEM干预措施评估的内部有效性和成本效益。与CRT相关的高成本使计划准确的试验变得至关重要，该试验不会高估所需的样本量，从而成本超过所需，或低估样本量，从而降低产生高质量计划有效性证据的可能性。在干预研究中包括教师水平，这是实施任何干预措施的关键水平，将允许对教师特征进行更多测试，以缓和干预效果。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。