Building a Framework for Developing and Evaluating Contextualized Items in Science Assessment (DECISA)

这个由科罗拉多大学丹佛分校和华盛顿大学西雅图分面创新联合参与的合作项目将建立一个框架，用于解决在STEM学习评估中使用情景项目的问题。主要目标是系统地调查情境项目的特点对学生表现的影响，以加强科学评估的实践，确保科学测试的公平性，并增加对评估和教学目的的支持。具有上下文的测试项被称为情境化项，其包括在测试项问题之前或之后的补充信息。这些信息可以包括对实验室设置、自然现象或经常被描述为场景、背景、小插曲或封面故事的实际问题的描述。该项目的发现将有助于理解学生如何理解上下文项目的意义，重点是他们在科学评估中经常遇到的复杂科学概念。目前，上下文项是根据传统智慧或非上下文项写作规则构建的。这类项目可能会误导学生注意无关的信息或干扰目标结构，从而导致对学生学习的不准确推断。该项目将开发一个框架来开发项目以帮助解决这一问题。大约有70名中学教师和4800名学生将参与这项研究。该项目将从理论上阐明语境化项目的特点，并对其对学生表现的影响进行实证检验。它试图解决关于语境化项目的文献中的四个空白：(1)关于如何概念化与结构相关的语境化项目的知识不足；(2)缺乏对科学中的语境化项目的研究；(3)缺乏系统地研究语境的特征并评估其对学生成绩的影响的研究；以及(4)需要考察与学生分组相关的差异影响的研究，以更清楚地了解哪些类型的语境影响到谁。研究设计和数据分析将以三个研究问题为指导：(1)什么是可能影响学生表现的关键背景特征，因此在制定科学测试项目时应该被考虑？(2)什么背景特征与结构相关变量有关？(3)由于性别、英语学习状况和社会经济状况变量，哪些背景特征与不同的学生表现模式有关？本项目将分两个阶段进行，历时三年。在第一阶段，该项目将完善一个拟议的理论框架，该框架将确定项目背景特征并阐明项目开发准则。在第二阶段，目标是通过选择、修订和开发具有不同背景的科学项目来应用框架；对项目进行实地测试；执行一系列心理测量和统计程序，测试分数，并对学生的认知面试反应和教师面试进行定性分析。通过这个过程产生的项目将旨在唤起学生存储的与目标内容和/或过程技能相关的知识。该项目将涉及一组专门从事评估开发和验证、科学教育、内容知识、语言学家和专家课堂教师的研究人员。该项目所产生的项目开发方法和项目将对国内和国际科学教育的研究人员和实践者产生直接影响。

AbstractThis collaborative project involving the University of Colorado at Denver and the University of Washington at Seattle in conjunction with Facet Innovations, will build a framework for addressing the use of contextualized items in the assessment of STEM learning. The primary goal is to systematically investigate the effects of characteristics of contextualized items on student performance to strengthen practices in science assessments, ensure fairness in science testing, and increase support for both assessment and instructional purposes. Test items with contexts are called contextualized items which include supplemental information that precedes or follows a test item question. Such information may include a description of a lab setup, a natural phenomenon, or a practical problem often depicted as a scenario, background, vignette, or cover story. The project findings will help to understand how students make sense of contextualized items focusing on complex scientific concepts that they usually encounter in science assessments. Currently, contextualized items are constructed from either conventional wisdom or non-contextualized item writing rules. Such items could mislead students to attend to irrelevant information or interfere with the targeted construct, and, therefore lead to inaccurate inferences about student learning. This project will develop a framework for developing items to help address this problem. Approximately, 70 classroom teachers and 4800 students, in secondary grades, will participate in the study. The project will offer a theoretical articulation of the characteristics of contextualized items and empirically test the effects on student performance. It seeks to address four gaps in the literature on contextualized items: (1) insufficient knowledge about how to conceptualize construct-relevant contextualized items; (2) lack of research on contextualized items in science; (3) lack of research that systematically studies the characteristics of contexts and evaluates their effects on student performance; and (4) the need for studies that examine differential effects related to subgroups of students to gain a greater clarity of what types of contexts affect whom. The research design and data analyses will be guided by three research questions: (1) what are critical context characteristics that may affect student performance and should therefore be considered when developing science test items? (2) what are context characteristics associated with construct-relevant variance? (3) what context characteristics are associated with differential student performance patterns due to gender, ELL status, and socioeconomic-status variables?The project will take place over three years through a two phase process. During Phase 1 the project will refine a proposed theoretical framework that will identify the item context characteristics and articulate the item development guidelines. During Phase 2, the goal is to apply the framework by selecting, revising, and developing science items with varying profiles of contexts; conduct field tests of the items; and perform a range of psychometric and statistical procedures with test scores, and qualitative analyses of students' cognitive interview responses and teacher interviews. Items resulting from this process will aim to evoke students' stored knowledge relevant to the content and/or process skills targeted. The project will involve a team of researchers specialized in assessment development and validation, science education, content knowledge, linguists, and expert classroom teachers. The item development approach and items generated from this project will have immediate implications for researchers and practitioners in science education nationally and internationally.