Supporting Instructional Decision Making: The Potential of An Automatically Scored Three-dimensional Assessment System

本项目将研究基于机器学习的评估系统的效用，以支持中学理科教师基于自动生成的学生报告(Autors)做出教学决策。评估的目标是三维(3D)科学学习，要求学生整合科学实践、交叉概念和学科核心思想，以理解现象或解决复杂问题。在乔治亚大学、密歇根州立大学、伊利诺伊大学芝加哥分校和WestEd的合作者的带领下，该项目团队将开发计算机评分算法、一套AUTOR和一系列教学内容知识支持(PCKS)。这些产品将帮助中学理科教师使用3D评估，做出信息化的教学改变，并改善学生的3D学习。该项目将产生有关教师使用3D评估的知识，并检查自动评分的3D评估的潜力。该项目将采用混合方法设计，分三个阶段实现研究目标。第一阶段：开发自动机。将利用现有数据开发3D评估任务的机器评分模型。为了支持教师对自动评分的解释和使用，项目团队将开发自动评分器，并检查教师如何使用这些初始报告。根据观察和教师的反馈，作者将使用迭代程序进行改进，以便教师可以更有效和更有生产力地使用它们。第二阶段：开发和测试PCKS。第一阶段的调查结果、文献和对有经验的教师的访谈将被用来开发PCKS。该项目将为教师提供如何使用自动取款机和PCKS的专业学习。该项目将研究教师如何使用AUTOR和PCKS来做出教学决策。研究结果将被用来改进PCKS。第三阶段：课堂实施。在这一阶段，将与一组新的教师一起进行一项研究，以探索AUTOR和PCKS在支持教师的教学决策和学生的3D学习方面的有效性和可用性。这个项目将创造关于教师如何解释和关注学生在3D评估中的表现的知识和制定理论，为如何支持教师做出响应性的教学决策提供关键信息。合作团队将广泛传播各种产品，如3D评估评分算法、Autors、PCKSS以及相应的专业发展计划，以及促进3D教学和学习的出版物。Discovery Research PreK-12计划(DRK-12)旨在通过研究和开发创新资源、模型和工具，显著提高PreK-12学生和教师的科学、技术、工程和数学(STEM)的学习和教学。DRK-12计划中的项目建立在STEM教育的基础研究和先前的研究和开发工作的基础上，为拟议的项目提供了理论和经验上的证明。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems. Led by collaborators from University of Georgia, Michigan State University, University of Illinois at Chicago, and WestEd, the project team will develop computer scoring algorithms, a suite of AutoRs, and an array of pedagogical content knowledge supports (PCKSs). These products will assist middle school science teachers in the use of 3D assessments, making informative instructional changes, and improve students’ 3D learning. The project will generate knowledge about teachers’ uses of 3D assessments and examine the potential of automatically scored 3D assessments. The project will achieve the research goals using a mixed-methods design in three phases. Phase I: Develop AutoRs. Machine scoring models for the 3D assessment tasks will be developed using existing data. To support teachers’ interpretation and use of automatic scores, the project team will develop AutoRs and examine how teachers make use of these initial reports. Based on observations and feedback from teachers, AutoRs will be refined using an iterative procedure so that teachers can use them with more efficiency and productivity. Phase II: Develop and test PCKSs. Findings from Phase I, the literature, and interviews with experienced teachers will be employed to develop PCKSs. The project will provide professional learning with teachers on how to use the AutoRs and PCKSs. The project will research how teachers use AutoRs and PCKSs to make instructional decisions. The findings will be used to refine the PCKSs. Phase III: Classroom implementation. In this phase a study will be conducted with a new group of teachers to explore the effectiveness and usability of AutoRs and PCKSs in terms of supporting teachers’ instructional decisions and students’ 3D learning. This project will create knowledge about and formulate a theory of how teachers interpret and attend to students’ performance on 3D assessments, providing critical information on how to support teachers’ responsive instructional decision making. The collaborative team will widely disseminate various products, such as 3D assessment scoring algorithms, AutoRs, PCKSs, and the corresponding professional development programs, and publications to facilitate 3D instruction and learning.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.