Investigating How to Enhance Scientific Argumentation through Automated Feedback in the Context of Two High School Earth Science Curriculum Units

当前强调通过积极参与科学实践来学习科学，并呼吁将教学与评估相结合；正在开发新的资源、模式和技术，以改善K-12科学学习。学生评估已经成为一个全国性的教育重点，部分原因是需要相关和及时的数据，让教师、管理人员、研究人员和公众了解所有学生在学习科学时的表现和思考情况。该项目响应了对技术增强评估的需求，促进了科学论证的批判性实践——从科学问题的证据中提出和解释主张，并批判性地评估主张中的不确定性来源。它将研究如何通过自动评分和即时反馈，在两个高中课程单元——气候变化和淡水供应——的背景下，在不同学生群体的学校中加强这种做法。该项目将采用先进的自动评分工具对学生的书面科学论证进行评分，为每个学生提供定制的反馈，并为教师提供班级级别的信息，以帮助他们改进科学论证。这项工作的关键成果将是一个技术支持的评估模型，该模型将促进对论证的理解，并将多层次反馈作为有效教与学的组成部分。该项目将加强该计划目前在评估方面的资助活动，将这些努力集中在作为复杂科学实践的学生论证上。这项设计和开发研究的目标是10个州的高中生（n=1,940）和教师（n=22），为期四年。研究问题是：(1)与人工诊断相比，自动评分工具（如c-rater和c-rater- ml）在多大程度上可以诊断学生的解释和不确定性表述？(2)如何设计和传递反馈以帮助学生提高科学论证能力？(3)教师如何使用和互动班级级自动评分和反馈，以支持学生用真实数据和模型进行科学论证；(4)学生在学习气候变化和淡水资源主题的核心思想时，通过建模强化的科学论证，学生如何看待他们对自动评分和即时反馈的整体体验？在一年级和二年级，计划进行可行性研究，以建立自动化评分模型，并为两个课程单元的先前测试评估设计反馈。在第三年，该项目将实施设计研究，以便通过随机分配确定有效的反馈。在四年级，一项试点研究将调查是否应该提供有或没有分数的有效反馈。该项目将采用混合方法。数据收集策略将包括课堂观察；教师和学生互动的截屏和日志数据，并自动反馈；教师和学生的问卷调查，有选择的和开放式的问题；以及对老师和学生的深度采访。所有构造响应解释和不确定性项目将使用带有细粒度规则的自动评分引擎进行评分。数据分析策略将包括多个标准来评估自动评分的质量；描述性统计分析；方差分析以调查设计研究的前/后测试和嵌入评估结果的差异；研究学生学习轨迹的协方差分析研究班级内学生聚类的两级层次线性建模；以及对视频和日志数据的分析。

With the current emphasis on learning science by actively engaging in the practices of science, and the call for integration of instruction and assessment; new resources, models, and technologies are being developed to improve K-12 science learning. Student assessment has become a nationwide educational priority due, in part, to the need for relevant and timely data that inform teachers, administrators, researchers, and the public about how all students perform and think while learning science. This project responds to the need for technology-enhanced assessments that promote the critical practice of scientific argumentation--making and explaining a claim from evidence about a scientific question and critically evaluating sources of uncertainty in the claim. It will investigate how to enhance this practice through automated scoring and immediate feedback in the context of two high school curriculum units--climate change and fresh-water availability--in schools with diverse student populations. The project will apply advanced automated scoring tools to students' written scientific arguments, provide individual students with customized feedback, and teachers with class-level information to assist them with improving scientific argumentation. The key outcome of this effort will be a technology-supported assessment model of how to advance the understanding of argumentation, and the use of multi-level feedback as a component of effective teaching and learning. The project will strengthen the program's current set of funded activities on assessment, focusing these efforts on students' argumentation as a complex science practice.This design and development research targets high school students (n=1,940) and teachers (n=22) in up to 10 states over four years. The research questions are: (1) To what extent can automated scoring tools, such as c-rater and c-rater-ML, diagnose students' explanations and uncertainty articulations as compared to human diagnosis?; (2) How should feedback be designed and delivered to help students improve scientific argumentation?; (3) How do teachers use and interact with class-level automated scores and feedback to support students' scientific argumentation with real-data and models?; and (4) How do students perceive their overall experience with the automated scores and immediate feedback when learning core ideas in climate change and fresh-water availability topics through scientific argumentation enhanced with modeling? In Years 1 and 2, plans are to conduct feasibility studies to build automated scoring models and design feedback for previously tested assessments for the two curriculum units. In Year 3, the project will implement design studies in order to identify effective feedback through random assignment. In Year 4, a pilot study will investigate if effective feedback should be offered with or without scores. The project will employ a mixed-methods approach. Data-gathering strategies will include classroom observations; screencast and log data of teachers' and students' interaction with automated feedback; teachers' and students' surveys with selected- and open-ended questions; and in-depth interviews with teachers and students. All constructed-response explanations and uncertainty items will be scored using automated scoring engines with fine-grained rubrics. Data analysis strategies will include multiple criteria to evaluate the quality of automated scores; descriptive statistical abalyses; analysis of variance to investigate differences in outcomes from the designed studies' pre/posttests and embedded assessments; analysis of covariance to investigate student learning trajectories; two-level hierarchical linear modeling to study the clustering of students within a class; and analysis of screencasts and log data.