Building a Framework for Developing and Evaluating Contextualized Items in Science Assessment (DECISA)

AbstractThis合作项目涉及大学的科罗拉多在丹佛和华盛顿大学在西雅图与方面创新，将建立一个框架，解决使用的上下文项目在评估STEM学习。主要目标是系统地调查情境化项目的特征对学生表现的影响，以加强科学评估的实践，确保科学测试的公平性，并增加对评估和教学目的的支持。具有上下文的测试项目被称为上下文化项目，其包括测试项目问题之前或之后的补充信息。这些信息可能包括对实验室设置、自然现象或实际问题的描述，这些描述通常被描述为场景、背景、小插曲或封面故事。该项目的研究结果将有助于了解学生如何理解他们通常在科学评估中遇到的侧重于复杂科学概念的情境化项目。目前，语境化项目是从传统智慧或非语境化项目写作规则构建的。这些项目可能会误导学生注意到不相关的信息或干扰目标结构，从而导致对学生学习的不准确推断。本项目将开发一个框架，用于开发项目，以帮助解决这一问题。大约有70名教师和4800名中学学生将参加这项研究。该项目将提供一个理论上的语境化项目的特点和实证测试对学生的表现的影响。本文试图解决语境化项目研究中的四个不足：（1）对如何概念化构式相关的语境化项目的认识不足;（2）缺乏科学领域的语境化项目研究;（3）缺乏系统研究语境的特点及其对学生成绩的影响的研究;（4）缺乏对语境化项目的研究。以及（4）需要研究与学生分组相关的差异效应，以更清楚地了解什么类型的背景影响谁。研究设计和数据分析将由三个研究问题指导：（1）什么是可能影响学生表现的关键背景特征，因此在开发科学测试项目时应该考虑？(2)与构式相关变异相关的语境特征是什么？(3)什么样的背景特征与不同的学生表现模式有关，因为性别，ELL状态和社会经济状态变量？该项目将通过两个阶段的过程在三年内进行。在第1阶段，项目将完善一个建议的理论框架，该框架将确定项目背景特征并阐明项目开发指南。在第二阶段，目标是通过选择，修改和开发具有不同背景的科学项目来应用框架;对项目进行实地测试;并使用测试成绩执行一系列心理测量和统计程序，以及对学生认知访谈响应和教师访谈的定性分析。从这个过程中产生的项目将旨在唤起学生存储的知识相关的内容和/或过程的技能为目标。该项目将涉及一个专门从事评估开发和验证，科学教育，内容知识，语言学家和专家课堂教师的研究团队。项目开发方法和项目产生的这个项目将有直接的影响，在科学教育的研究人员和从业人员在国家和国际。

AbstractThis collaborative project involving the University of Colorado at Denver and the University of Washington at Seattle in conjunction with Facet Innovations, will build a framework for addressing the use of contextualized items in the assessment of STEM learning. The primary goal is to systematically investigate the effects of characteristics of contextualized items on student performance to strengthen practices in science assessments, ensure fairness in science testing, and increase support for both assessment and instructional purposes. Test items with contexts are called contextualized items which include supplemental information that precedes or follows a test item question. Such information may include a description of a lab setup, a natural phenomenon, or a practical problem often depicted as a scenario, background, vignette, or cover story. The project findings will help to understand how students make sense of contextualized items focusing on complex scientific concepts that they usually encounter in science assessments. Currently, contextualized items are constructed from either conventional wisdom or non-contextualized item writing rules. Such items could mislead students to attend to irrelevant information or interfere with the targeted construct, and, therefore lead to inaccurate inferences about student learning. This project will develop a framework for developing items to help address this problem. Approximately, 70 classroom teachers and 4800 students, in secondary grades, will participate in the study. The project will offer a theoretical articulation of the characteristics of contextualized items and empirically test the effects on student performance. It seeks to address four gaps in the literature on contextualized items: (1) insufficient knowledge about how to conceptualize construct-relevant contextualized items; (2) lack of research on contextualized items in science; (3) lack of research that systematically studies the characteristics of contexts and evaluates their effects on student performance; and (4) the need for studies that examine differential effects related to subgroups of students to gain a greater clarity of what types of contexts affect whom. The research design and data analyses will be guided by three research questions: (1) what are critical context characteristics that may affect student performance and should therefore be considered when developing science test items? (2) what are context characteristics associated with construct-relevant variance? (3) what context characteristics are associated with differential student performance patterns due to gender, ELL status, and socioeconomic-status variables?The project will take place over three years through a two phase process. During Phase 1 the project will refine a proposed theoretical framework that will identify the item context characteristics and articulate the item development guidelines. During Phase 2, the goal is to apply the framework by selecting, revising, and developing science items with varying profiles of contexts; conduct field tests of the items; and perform a range of psychometric and statistical procedures with test scores, and qualitative analyses of students' cognitive interview responses and teacher interviews. Items resulting from this process will aim to evoke students' stored knowledge relevant to the content and/or process skills targeted. The project will involve a team of researchers specialized in assessment development and validation, science education, content knowledge, linguists, and expert classroom teachers. The item development approach and items generated from this project will have immediate implications for researchers and practitioners in science education nationally and internationally.