Building a Framework for Developing and Evaluating Contextualized Items in Science Assessment (DECISA)

构建开发和评估科学评估中情境化项目的框架 (DECISA)

• 批准号: 1432406
• 负责人: Maria Araceli Ruiz-Primo
• 金额: $ 149.87万
• 依托单位: University of Colorado at Denver-Downtown Campus
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-08-01 至 2017-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1432406&HistoricalAwards=false
• 关键词: Building; Framework; Developing; Evaluating; Contextualized

AbstractThis collaborative project involving the University of Colorado at Denver and the University of Washington at Seattle in conjunction with Facet Innovations, will build a framework for addressing the use of contextualized items in the assessment of STEM learning. The primary goal is to systematically investigate the effects of characteristics of contextualized items on student performance to strengthen practices in science assessments, ensure fairness in science testing, and increase support for both assessment and instructional purposes. Test items with contexts are called contextualized items which include supplemental information that precedes or follows a test item question. Such information may include a description of a lab setup, a natural phenomenon, or a practical problem often depicted as a scenario, background, vignette, or cover story. The project findings will help to understand how students make sense of contextualized items focusing on complex scientific concepts that they usually encounter in science assessments. Currently, contextualized items are constructed from either conventional wisdom or non-contextualized item writing rules. Such items could mislead students to attend to irrelevant information or interfere with the targeted construct, and, therefore lead to inaccurate inferences about student learning. This project will develop a framework for developing items to help address this problem. Approximately, 70 classroom teachers and 4800 students, in secondary grades, will participate in the study. The project will offer a theoretical articulation of the characteristics of contextualized items and empirically test the effects on student performance. It seeks to address four gaps in the literature on contextualized items: (1) insufficient knowledge about how to conceptualize construct-relevant contextualized items; (2) lack of research on contextualized items in science; (3) lack of research that systematically studies the characteristics of contexts and evaluates their effects on student performance; and (4) the need for studies that examine differential effects related to subgroups of students to gain a greater clarity of what types of contexts affect whom. The research design and data analyses will be guided by three research questions: (1) what are critical context characteristics that may affect student performance and should therefore be considered when developing science test items? (2) what are context characteristics associated with construct-relevant variance? (3) what context characteristics are associated with differential student performance patterns due to gender, ELL status, and socioeconomic-status variables?The project will take place over three years through a two phase process. During Phase 1 the project will refine a proposed theoretical framework that will identify the item context characteristics and articulate the item development guidelines. During Phase 2, the goal is to apply the framework by selecting, revising, and developing science items with varying profiles of contexts; conduct field tests of the items; and perform a range of psychometric and statistical procedures with test scores, and qualitative analyses of students' cognitive interview responses and teacher interviews. Items resulting from this process will aim to evoke students' stored knowledge relevant to the content and/or process skills targeted. The project will involve a team of researchers specialized in assessment development and validation, science education, content knowledge, linguists, and expert classroom teachers. The item development approach and items generated from this project will have immediate implications for researchers and practitioners in science education nationally and internationally.

摘要该合作项目由科罗拉多大学丹佛分校和华盛顿大学西雅图分校与Facet Innovations联合开展，将建立一个框架，用于解决在STEM学习评估中使用情境化项目的问题。主要目标是系统地研究情境化项目的特征对学生成绩的影响，以加强科学评估的实践，确保科学测试的公平性，并增加对评估和教学目的的支持。具有上下文的测试项目称为上下文化测试项目，它包括在测试项目问题之前或之后的补充信息。这些信息可能包括对实验室设置、自然现象或实际问题的描述，通常被描述为场景、背景、小插图或封面故事。该项目的研究结果将有助于理解学生如何理解他们在科学评估中经常遇到的复杂科学概念的情境化项目。目前，情境化的条目是根据传统智慧或非情境化的条目编写规则构建的。这些项目可能会误导学生注意不相关的信息或干扰目标结构，从而导致对学生学习的不准确推断。该项目将为开发有助于解决这一问题的项目制定一个框架。大约70名任课教师和4800名中学生将参与这项研究。该项目将提供一个理论阐述情境化项目的特点，并实证测试对学生表现的影响。它试图解决关于情境化项目的文献中的四个空白：(1)关于如何概念化与建构相关的情境化项目的知识不足；(2)科学情境化项目研究不足；(3)缺乏系统研究情境特征并评估其对学生表现影响的研究；(4)需要研究与学生分组相关的差异影响，以更清楚地了解哪种类型的环境影响谁。研究设计和数据分析将以三个研究问题为指导：(1)在开发科学测试项目时，什么是可能影响学生表现的关键环境特征？(2)与构念相关方差相关的语境特征是什么？(3)由于性别、ELL地位和社会经济地位变量，哪些情境特征与差异的学生表现模式相关？该项目将分两个阶段进行，历时三年。在第一阶段，项目将完善一个建议的理论框架，该框架将确定项目背景特征并阐明项目开发指南。在第二阶段，目标是通过选择、修改和开发具有不同背景的科学项目来应用该框架；对项目进行实地测试；并对学生的认知访谈反应和教师访谈进行定性分析，对测试成绩进行一系列心理测量和统计程序。从这个过程中产生的项目将旨在唤起学生存储的与内容和/或过程技能相关的知识。该项目将包括一个专门从事评估开发和验证、科学教育、内容知识、语言学家和专业课堂教师的研究小组。项目开发方法和项目产生的项目将对国内外科学教育的研究人员和实践者产生直接影响。