Validities of Standards-Based Science Inquiry Assessments: Implementation Study

为了收集大规模评估实践的证据，SRI国际向国家科学基金会提出了一项研究，以检查基于标准的科学探究评估的有效性。在回答可再生能源方案提出的谈判问题时，SRI将最初提议的研究工作范围调整为两个阶段：设计和实施。1999年7月，SRI获得资助进行设计研究。国家公认的专家团队研究了美国三个最有影响力的大型科学评估-NAEP、TIMSS和新标准-所使用的科学项目格式的有效性。研究团队进行了三项不同但协调的研究，包括一项对齐研究、一项认知分析研究和一项实证研究。完成后，设计研究将整合三项研究的方法和初步结果。设计研究的结果有助于拟议实施研究的设计和方法。实施研究将使我们能够深入分析在国家和州一级的大规模科学评估中使用的科学项目和任务；深入检查大样本和多样化学生的表现；以及应用在设计研究期间改进的强有力的方法。实施研究的具体目标是：(1)制定一个框架，描述旨在引发不同类型科学学习的项目/任务形式的设计特征；(2)研究现有参考和扩展调查评估中选定的评估问题和练习形式与国家科学教育标准的一致性；(3)分析解决不同类型项目/任务时对学习者的认知需求；(4)对不同类型评估形式的有效性，特别是不同评估形式的教学敏感性进行实证研究，(5)开发多层次评价系统的原型设计，为学生的科学知识和探究策略提供更全面的测量。执行研究可能带来的好处将包括强有力的方法和程序，以支持今后开展以探究为基础的科学评估研究。将确定程序陷阱以及项目和测试设计考虑因素，以指导研究社区以及州和地方教育机构设计更好的效度研究。此外，我们还将得出学生学习机会对科学探究成就研究设计的启示。实施研究将提供确凿的证据，证明哪些特定的项目和项目/任务模式打算测试学生的实际推理能力，并解释评估模式如何调用不同种类和水平的科学知识和探究。最后，我们打算整合这些发现来制定指导方针，指导教育工作者如何将参考考试结果与其他考试形式的数据结合起来，以传达科学成就水平，并就专业发展、课程和评估改革做出政策决定。

To gather evidence about large-scale assessment practices, SRI International proposed a research study to NSF to examine the validities of standards-based science inquiry assessment. In response to negotiating questions from the REPP program, SRI restructured the scope of work of the originally proposed study into two phases: Design and Implementation. SRI was funded to conduct the Design Study in July, 1999. Teams of nationally recognized experts studied the validities of science item formats used in three of the nation's most influential large-scale science assessments-NAEP, TIMSS, and New Standards. The research teams conducted three distinct, but coordinated, studies including an Alignment Study, a Cognitive Analysis Study, and an Empirical Study. When completed, the Design Study will integrate methodologies and preliminary findings from the three studies. Results of the Design Study have contributed to the design and methodologies of the proposed Implementation Study. The Implementation Study will permit an intensive analysis of science items and tasks that have been used in large-scale science assessments at the national, as well as the state level; an in-depth examination of the performances of a large and diverse sample of students; and the application of powerful methodologies that were refined during the Design Study. The specific goals of the Implementation Study are to: (1) develop a framework for characterizing the design features of item/task formats that aim to elicit different types of science learning, (2) conduct a study of the alignment of selected assessment questions and exercise formats from existing reference and extended investigation assessments with the National Science Education Standards, (3) conduct analyses of the cognitive demands placed on learners when solving different types of items/tasks, (4) conduct an empirical study of the validities of different types of assessment formats, in particular, the instructional sensitivity of the different assessment formats, and (5) develop prototype designs of multi-level assessment systems that would provide more comprehensive measures of students' science knowledge and inquiry strategies. Benefits that are likely to flow from the Implementation Study will include robust methodologies and procedures to support the conduct of future studies of inquiry-based science assessment. Procedural pitfalls and item and test design considerations will be identified that can guide the research community and state and local education agencies in the design of better validity studies. In addition, we will draw implications of student learning opportunities for the design of science inquiry achievement studies. The Implementation Study will provide confirmatory evidence of what particular items and item/task formats intend to test and student's actual reasoning and explanations of how assessment formats call on different kinds and levels of science knowledge and inquiry. Finally, we intend to integrate these findings to develop guidelines for how educators may combine reference exam results with data from other exam formats to communicate science achievement levels and make policy decisions about professional development, curriculum, and assessment reforms.