Investigating How to Enhance Scientific Argumentation through Automated Feedback in the Context of Two High School Earth Science Curriculum Units

当前强调通过积极参与科学实践来学习科学，并呼吁将教学和评估相结合；正在开发新的资源、模式和技术，以改善K-12科学学习。学生评估已经成为全国性的教育优先事项，部分原因是需要相关和及时的数据，让教师、管理人员、研究人员和公众了解所有学生在学习科学时的表现和思考情况。该项目响应了对促进科学论证的关键实践的技术强化评估的需要--根据关于科学问题的证据提出和解释主张，并对主张中的不确定性来源进行批判性评估。它将调查如何通过在拥有不同学生人口的学校的两个高中课程单元--气候变化和淡水供应--的背景下，通过自动评分和即时反馈来加强这一做法。该项目将把先进的自动评分工具应用于学生的书面科学论点，为学生个人提供定制的反馈，并为教师提供班级信息，以帮助他们提高科学论证能力。这一努力的主要成果将是一个由技术支持的评估模式，该模式将如何增进对辩论的理解，并将多层次反馈作为有效教学的一个组成部分。该项目将加强该项目目前资助的一系列评估活动，将这些努力集中在学生辩论这一复杂的科学实践上。这项设计和开发研究针对10个州的高中生(n=1,940)和教师(n=22)，历时四年。研究问题是：(1)与人类诊断相比，自动评分工具，如C-Rater和C-Rater-ML，能在多大程度上诊断学生的解释和不确定性表达？(2)应该如何设计和提供反馈来帮助学生改进科学论证？(3)教师如何使用和交互班级水平的自动评分和反馈来支持学生的科学论证？以及(4)当学生通过通过建模增强的科学论证来学习气候变化和淡水可获得性主题的核心思想时，学生如何感知他们对自动评分和即时反馈的总体体验？在第一年和第二年，计划进行可行性研究，以建立自动评分模型，并为以前测试过的两个课程单元的评估设计反馈。在第三年，该项目将进行设计研究，以便通过随机分配确定有效的反馈。在第四年，一项试点研究将调查是否应该提供有效的反馈，是否应该有分数。该项目将采用混合方法。数据收集战略将包括课堂观察；教师和学生与自动反馈互动的屏幕录像和记录数据；教师和学生通过精选和开放式问题进行的调查；以及与教师和学生的深入访谈。所有构建的响应解释和不确定项将使用带有细粒度Rubrics的自动评分引擎进行评分。数据分析策略将包括评估自动成绩质量的多个标准；描述性统计分析；方差分析，以调查设计研究的前测/后测和嵌入式评估结果的差异；协方差分析，以调查学生的学习轨迹；两级分层线性建模，以研究学生在班级内的聚集；以及分析截屏视频和日志数据。

With the current emphasis on learning science by actively engaging in the practices of science, and the call for integration of instruction and assessment; new resources, models, and technologies are being developed to improve K-12 science learning. Student assessment has become a nationwide educational priority due, in part, to the need for relevant and timely data that inform teachers, administrators, researchers, and the public about how all students perform and think while learning science. This project responds to the need for technology-enhanced assessments that promote the critical practice of scientific argumentation--making and explaining a claim from evidence about a scientific question and critically evaluating sources of uncertainty in the claim. It will investigate how to enhance this practice through automated scoring and immediate feedback in the context of two high school curriculum units--climate change and fresh-water availability--in schools with diverse student populations. The project will apply advanced automated scoring tools to students' written scientific arguments, provide individual students with customized feedback, and teachers with class-level information to assist them with improving scientific argumentation. The key outcome of this effort will be a technology-supported assessment model of how to advance the understanding of argumentation, and the use of multi-level feedback as a component of effective teaching and learning. The project will strengthen the program's current set of funded activities on assessment, focusing these efforts on students' argumentation as a complex science practice.This design and development research targets high school students (n=1,940) and teachers (n=22) in up to 10 states over four years. The research questions are: (1) To what extent can automated scoring tools, such as c-rater and c-rater-ML, diagnose students' explanations and uncertainty articulations as compared to human diagnosis?; (2) How should feedback be designed and delivered to help students improve scientific argumentation?; (3) How do teachers use and interact with class-level automated scores and feedback to support students' scientific argumentation with real-data and models?; and (4) How do students perceive their overall experience with the automated scores and immediate feedback when learning core ideas in climate change and fresh-water availability topics through scientific argumentation enhanced with modeling? In Years 1 and 2, plans are to conduct feasibility studies to build automated scoring models and design feedback for previously tested assessments for the two curriculum units. In Year 3, the project will implement design studies in order to identify effective feedback through random assignment. In Year 4, a pilot study will investigate if effective feedback should be offered with or without scores. The project will employ a mixed-methods approach. Data-gathering strategies will include classroom observations; screencast and log data of teachers' and students' interaction with automated feedback; teachers' and students' surveys with selected- and open-ended questions; and in-depth interviews with teachers and students. All constructed-response explanations and uncertainty items will be scored using automated scoring engines with fine-grained rubrics. Data analysis strategies will include multiple criteria to evaluate the quality of automated scores; descriptive statistical abalyses; analysis of variance to investigate differences in outcomes from the designed studies' pre/posttests and embedded assessments; analysis of covariance to investigate student learning trajectories; two-level hierarchical linear modeling to study the clustering of students within a class; and analysis of screencasts and log data.