Supporting Instructional Decision Making: The Potential of An Automatically Scored Three-dimensional Assessment System

该项目将研究基于机器学习的评估系统的实用性，以支持中学科学教师根据自动生成的学生报告（AutoRs）做出教学决策。这些评估针对三维（3D）科学学习，要求学生整合科学实践，交叉概念和学科核心思想，以理解现象或解决复杂问题。在来自格鲁吉亚大学、密歇根州立大学、伊利诺伊大学芝加哥分校和WestEd的合作者的领导下，该项目团队将开发计算机评分算法、一套AutoR和一系列教学内容知识支持（PCKS）。这些产品将帮助中学科学教师使用3D评估，进行信息化的教学改革，并提高学生的3D学习。该项目将产生关于教师使用3D评估的知识，并研究自动评分3D评估的潜力。该项目将通过三个阶段的混合方法设计来实现研究目标。第一阶段：开发AutoR。将使用现有数据开发3D评估任务的机器评分模型。为了支持教师对自动评分的解释和使用，项目小组将开发AutoR，并研究教师如何使用这些初始报告。根据教师的观察和反馈，AutoR将使用迭代程序进行改进，以便教师能够更有效地使用它们。第二阶段：开发和测试PCKS。第一阶段的研究结果，文献，和有经验的教师访谈将被用来开发PCKS。该项目将为教师提供如何使用AutoR和PCKS的专业学习。该项目将研究教师如何使用AutoRs和PCKS进行教学决策。研究结果将用于改进PCKS。第三阶段：课堂实施。在这一阶段，将与一组新的教师进行一项研究，探讨AutoR和PCKS在支持教师的教学决策和学生的三维学习方面的有效性和可用性。该项目将创造知识，并制定一个理论，教师如何解释和照顾学生的表现在3D评估，提供关键信息，如何支持教师的反应教学决策。合作团队将广泛传播各种产品，如3D评估评分算法，AutoR，PCKS，以及相应的专业发展计划和出版物，以促进3D教学和学习。探索研究preK-12计划（DRK-12）旨在显着提高preK-12学生和教师的科学，技术，工程和数学（STEM）的学习和教学，通过研究和开发创新资源、模式和工具。DRK-12项目中的项目建立在STEM教育的基础研究以及为拟议项目提供理论和经验依据的先前研究和开发工作的基础上。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems. Led by collaborators from University of Georgia, Michigan State University, University of Illinois at Chicago, and WestEd, the project team will develop computer scoring algorithms, a suite of AutoRs, and an array of pedagogical content knowledge supports (PCKSs). These products will assist middle school science teachers in the use of 3D assessments, making informative instructional changes, and improve students’ 3D learning. The project will generate knowledge about teachers’ uses of 3D assessments and examine the potential of automatically scored 3D assessments. The project will achieve the research goals using a mixed-methods design in three phases. Phase I: Develop AutoRs. Machine scoring models for the 3D assessment tasks will be developed using existing data. To support teachers’ interpretation and use of automatic scores, the project team will develop AutoRs and examine how teachers make use of these initial reports. Based on observations and feedback from teachers, AutoRs will be refined using an iterative procedure so that teachers can use them with more efficiency and productivity. Phase II: Develop and test PCKSs. Findings from Phase I, the literature, and interviews with experienced teachers will be employed to develop PCKSs. The project will provide professional learning with teachers on how to use the AutoRs and PCKSs. The project will research how teachers use AutoRs and PCKSs to make instructional decisions. The findings will be used to refine the PCKSs. Phase III: Classroom implementation. In this phase a study will be conducted with a new group of teachers to explore the effectiveness and usability of AutoRs and PCKSs in terms of supporting teachers’ instructional decisions and students’ 3D learning. This project will create knowledge about and formulate a theory of how teachers interpret and attend to students’ performance on 3D assessments, providing critical information on how to support teachers’ responsive instructional decision making. The collaborative team will widely disseminate various products, such as 3D assessment scoring algorithms, AutoRs, PCKSs, and the corresponding professional development programs, and publications to facilitate 3D instruction and learning.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.