Exploring the Validity of Three-Dimensional Assessments for Understanding Student Learning of Science

探索三维评估对于理解学生科学学习的有效性

• 批准号: 2201438
• 负责人: Cari Herrmann Abell
• 金额: $ 146.67万
• 依托单位: BSCS Science Learning
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-15 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2201438&HistoricalAwards=false
• 关键词: Exploring; Validity; Three; Dimensional; Assessments

The Next Generation Science Standards (NGSS Lead States, 2013) organizes goals for student learning along three dimensions: (1) scientific and engineering practices such as developing and using models; (2) crosscutting concepts such as cause and effect; and (3) disciplinary core ideas which includes foundational ideas from the life, physical, and earth sciences, and engineering and technology. Students in 44 states are being or will soon be held accountable to science standards based on the NGSS. High-quality assessments along these three dimensions (termed "3-D assessments") are needed to help states, teachers, and researchers measure students’ progress and the impact of educational interventions. While there is emerging agreement about the features of high-quality 3-D assessments, developers have struggled to create them for the NGSS that meet the criteria that have been set in the standards. In the process, they have developed assessments that students find more difficult to complete and score lower on than traditional ones. In addition, there is little evidence to determine if current assessments are measuring what they purport to measure. This project addresses this issue by conducting studies to explore what these assessments are actually measuring. This project will extend prior work around assessment research and development and promises to have a significant impact on how the field of science education thinks about 3-D assessments. The findings from this project will provide information about how the assessments function for English-language learners learners which could be used to increase the equity of 3-D assessments. The findings could also help assessment developers better develop instruments that measure students’ ability to demonstrate knowledge along the three dimensions articulated in the NGSS. Additionally, if the assessments used in this project are found to be valid measures of students’ 3-D science understanding, the project’s findings could be used to investigate the impact of 3-D science learning at the middle school level.In order to advance the NGSS, it is critical that advances in the assessment of learning along these three dimensions of knowing be made. Thus far, 3-D assessment development has largely focused on developing clusters of items that are built around a focal phenomenon, known as phenomena-based item clusters (PBICs). The overarching research question that this project seeks to answer is to what extent are PBICs valid measures of students’ science understanding along these three NGSS dimensions? This question will be answered by developing an evidence-based validity argument based on the variety of evidence outlined in the updated Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999/2014). First, the project team will develop a pool of extant PBICs and investigate the extent to which they meet the quality criteria established for 3-D assessments (evidence based on test content). The results of this investigation will then inform revisions to the assessments that will be made before they are used in a series of studies designed to gather validity evidence. Next, think-aloud interviews will be conducted with middle school students to study how well PBICs tap into the intended cognitive processes related to the NGSS dimensions (evidence based on response processes). Concurrently, a larger group of middle school students from across the U.S. who will be using the OpenSciEd materials will be recruited to participate in a pre-test/post-test study. Rasch modeling of the pre- and post-test data will be used to determine the dimensionality of the items that make up the PBICs (evidence based on internal structure), investigate the correlation between performance on PBICs and disciplinary core idea-focused, multiple-choice assessments (evidence based on relations to other variables), and explore whether the PBICs function differently for English learners and native English speakers (evidence based on internal structure). Hierarchical linear models will be used to uncover what proportion of variance explained by PBICs is explained by measures of reading or writing ability (evidence based on relations to other variables). Finally, measures of instructional sensitivity for both the PBICs and the disciplinary core idea-focused assessments will be calculated to investigate the extent to which these assessments are sensitive to science instruction along the three NGSS dimensions (evidence based on consequences of testing).This project is supported by NSF's EHR Core Research (ECR) program. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field. Investments are made in critical areas that are essential, broad, and enduring: STEM learning and STEM learning environments, broadening participation in STEM, and STEM workforce development. The program supports the accumulation of robust evidence to inform efforts to understand, build theory to explain, and suggest intervention and innovations to address persistent challenges in education.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

下一代科学标准（NGSS Lead States，2013）从三个维度组织学生学习目标：（1）科学和工程实践，例如开发和使用模型； (2) 因果关系等交叉概念； （3）学科核心思想，包括生命科学、物理科学、地球科学、工程技术等领域的基础思想。 44 个州的学生正在或即将对基于 NGSS 的科学标准负责。需要沿着这三个维度进行高质量评估（称为“3D评估”），以帮助各州、教师和研究人员衡量学生的进步和教育干预的影响。尽管人们对高质量 3D 评估的功能逐渐达成一致，但开发人员一直在努力为 NGSS 创建符合标准中设定标准的评估。在此过程中，他们开发了一些评估，学生发现这些评估比传统评估更难完成，得分也更低。此外，几乎没有证据可以确定当前的评估是否在衡量其声称的目标。该项目通过进行研究来探索这些评估实际测量的内容来解决这个问题。该项目将扩展之前围绕评估研究和开发的工作，并有望对科学教育领域如何看待 3D 评估产生重大影响。该项目的研究结果将提供有关评估如何为英语语言学习者发挥作用的信息，可用于提高 3D 评估的公平性。研究结果还可以帮助评估开发人员更好地开发工具，衡量学生沿着 NGSS 中阐述的三个维度展示知识的能力。此外，如果发现该项目中使用的评估是衡量学生 3D 科学理解的有效指标，则该项目的研究结果可用于调查 3D 科学学习对中学水平的影响。为了推进 NGSS，在这三个认知维度的学习评估方面取得进展至关重要。到目前为止，3D 评估开发主要集中于开发围绕焦点现象构建的项目集群，称为基于现象的项目集群 (PBIC)。该项目试图回答的首要研究问题是 PBIC 在多大程度上有效衡量学生在这三个 NGSS 维度上的科学理解？这个问题将通过根据更新的教育和心理测试标准（AERA、APA 和 NCME，1999/2014）中概述的各种证据开发基于证据的有效性论证来回答。首先，项目团队将开发现有 PBIC 池，并调查它们满足 3D 评估所建立的质量标准的程度（基于测试内容的证据）。这项调查的结果将指导对评估进行修订，然后将其用于旨在收集有效性证据的一系列研究。接下来，将对中学生进行有声思考访谈，以研究 PBIC 如何更好地利用与 NGSS 维度相关的预期认知过程（基于响应过程的证据）。同时，将招募更多来自美国各地使用 OpenSciEd 材料的中学生参加测试前/测试后研究。测试前和测试后数据的 Rasch 模型将用于确定构成 PBIC 的项目的维度（基于内部结构的证据），研究 PBIC 表现与以学科核心理念为中心的多项选择评估之间的相关性（基于与其他变量关系的证据），并探讨 PBIC 对于英语学习者和英语母语者的功能是否不同（基于内部结构的证据）。分层线性模型将用于揭示 PBIC 所解释的方差比例是通过阅读或写作能力的测量来解释的（基于与其他变量的关系的证据）。最后，将计算 PBIC 和以学科核心理念为中心的评估的教学敏感性测量，以调查这些评估对三个 NGSS 维度的科学教学的敏感程度（基于测试结果的证据）。该项目得到了 NSF 的 EHR 核心研究 (ECR) 计划的支持。 ECR 项目强调基础 STEM 教育研究，产生该领域的基础知识。对重要、广泛和持久的关键领域进行投资：STEM 学习和 STEM 学习环境、扩大 STEM 参与以及 STEM 劳动力发展。该计划支持积累强有力的证据，为理解、建立解释理论以及提出干预和创新建议以解决教育中持续存在的挑战提供信息。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。