Exploring the Validity of Three-Dimensional Assessments for Understanding Student Learning of Science

《下一代科学标准》(NGSS Lead States，2013)从三个方面组织了学生的学习目标：(1)科学和工程实践，如开发和使用模型；(2)交叉概念，如因果；(3)学科核心思想，包括来自生命、物理和地球科学以及工程和技术的基本思想。44个州的学生正在或即将被要求对基于NGSS的科学标准负责。这三个维度的高质量评估(称为“3-D评估”)是必要的，以帮助各州、教师和研究人员衡量学生的进步和教育干预的影响。尽管人们开始就高质量3-D评估的功能达成一致，但开发人员一直在努力为满足标准中设定的标准的NGSS创建这些功能。在这个过程中，他们开发了一些评估，学生们认为这些评估比传统评估更难完成，得分也更低。此外，几乎没有证据来确定当前的评估是否在衡量它们声称要衡量的东西。该项目通过开展研究来探讨这些评估实际衡量的内容，从而解决了这一问题。该项目将围绕评估、研究和开发扩展先前的工作，并有望对科学教育领域如何看待3D评估产生重大影响。该项目的调查结果将为英语学习者提供有关评估如何发挥作用的信息，这些信息可以用来增加3D评估的公平性。这些发现还可以帮助评估开发人员更好地开发工具，衡量学生在NGSS中阐明的三个维度上展示知识的能力。此外，如果本项目中使用的评估被发现是学生三维科学理解的有效测量，该项目的研究结果可以用于调查三维科学学习在中学水平上的影响。为了推进NGSS，关键是在这三个知晓维度的学习评估方面取得进展。到目前为止，3D评估的发展主要集中在开发围绕一种焦点现象建立的项目集群，称为基于现象的项目集群(PBIC)。这个项目试图回答的首要研究问题是，PBIC在多大程度上有效地衡量了学生在这三个NGSS维度上的科学理解？这一问题的答案将是根据最新的《教育和心理测试标准》(AERA，APA，&AMP；NCME，1999/2014)中概述的各种证据，提出基于证据的有效性论证。首先，项目组将开发现有的PBIC池，并调查它们在多大程度上符合为3-D评估建立的质量标准(基于测试内容的证据)。这项调查的结果将在一系列旨在收集有效性证据的研究中使用之前对评估进行修订。接下来，将对中学生进行有声思考访谈，以研究PBIC如何很好地利用与NGSS维度相关的预期认知过程(基于反应过程的证据)。与此同时，来自美国各地的一大批将使用OpenScience ED材料的中学生将被招募参加一项测试前/测试后的研究。对测试前后数据的RASCH模型将被用来确定构成PBICs的项目的维度(基于内部结构的证据)，调查PBICs的表现与学科核心理念集中的多项选择评估(基于与其他变量的关系)之间的相关性，并探索PBICs对英语学习者和以英语为母语的人是否起到不同的作用(基于内部结构的证据)。分层线性模型将被用来揭示PBIC解释的方差比例是通过阅读或写作能力(基于与其他变量的关系的证据)来解释的。最后，将计算PBICS和以学科核心理念为重点的评估的教学敏感度，以调查这些评估在三个NGSS维度(基于测试结果的证据)上对科学教学的敏感度。该项目由NSF的EHR核心研究(ECR)计划支持。ECR计划强调基础STEM教育研究，以产生该领域的基础知识。在基本、广泛和持久的关键领域进行投资：STEM学习和STEM学习环境，扩大STEM的参与，以及STEM劳动力发展。该计划支持积累强有力的证据，为理解、构建理论以解释并建议干预和创新以应对教育中持续存在的挑战的努力提供信息。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The Next Generation Science Standards (NGSS Lead States, 2013) organizes goals for student learning along three dimensions: (1) scientific and engineering practices such as developing and using models; (2) crosscutting concepts such as cause and effect; and (3) disciplinary core ideas which includes foundational ideas from the life, physical, and earth sciences, and engineering and technology. Students in 44 states are being or will soon be held accountable to science standards based on the NGSS. High-quality assessments along these three dimensions (termed "3-D assessments") are needed to help states, teachers, and researchers measure students’ progress and the impact of educational interventions. While there is emerging agreement about the features of high-quality 3-D assessments, developers have struggled to create them for the NGSS that meet the criteria that have been set in the standards. In the process, they have developed assessments that students find more difficult to complete and score lower on than traditional ones. In addition, there is little evidence to determine if current assessments are measuring what they purport to measure. This project addresses this issue by conducting studies to explore what these assessments are actually measuring. This project will extend prior work around assessment research and development and promises to have a significant impact on how the field of science education thinks about 3-D assessments. The findings from this project will provide information about how the assessments function for English-language learners learners which could be used to increase the equity of 3-D assessments. The findings could also help assessment developers better develop instruments that measure students’ ability to demonstrate knowledge along the three dimensions articulated in the NGSS. Additionally, if the assessments used in this project are found to be valid measures of students’ 3-D science understanding, the project’s findings could be used to investigate the impact of 3-D science learning at the middle school level.In order to advance the NGSS, it is critical that advances in the assessment of learning along these three dimensions of knowing be made. Thus far, 3-D assessment development has largely focused on developing clusters of items that are built around a focal phenomenon, known as phenomena-based item clusters (PBICs). The overarching research question that this project seeks to answer is to what extent are PBICs valid measures of students’ science understanding along these three NGSS dimensions? This question will be answered by developing an evidence-based validity argument based on the variety of evidence outlined in the updated Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999/2014). First, the project team will develop a pool of extant PBICs and investigate the extent to which they meet the quality criteria established for 3-D assessments (evidence based on test content). The results of this investigation will then inform revisions to the assessments that will be made before they are used in a series of studies designed to gather validity evidence. Next, think-aloud interviews will be conducted with middle school students to study how well PBICs tap into the intended cognitive processes related to the NGSS dimensions (evidence based on response processes). Concurrently, a larger group of middle school students from across the U.S. who will be using the OpenSciEd materials will be recruited to participate in a pre-test/post-test study. Rasch modeling of the pre- and post-test data will be used to determine the dimensionality of the items that make up the PBICs (evidence based on internal structure), investigate the correlation between performance on PBICs and disciplinary core idea-focused, multiple-choice assessments (evidence based on relations to other variables), and explore whether the PBICs function differently for English learners and native English speakers (evidence based on internal structure). Hierarchical linear models will be used to uncover what proportion of variance explained by PBICs is explained by measures of reading or writing ability (evidence based on relations to other variables). Finally, measures of instructional sensitivity for both the PBICs and the disciplinary core idea-focused assessments will be calculated to investigate the extent to which these assessments are sensitive to science instruction along the three NGSS dimensions (evidence based on consequences of testing).This project is supported by NSF's EHR Core Research (ECR) program. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field. Investments are made in critical areas that are essential, broad, and enduring: STEM learning and STEM learning environments, broadening participation in STEM, and STEM workforce development. The program supports the accumulation of robust evidence to inform efforts to understand, build theory to explain, and suggest intervention and innovations to address persistent challenges in education.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.