Collaborative Research: Integrating Models of Cognition and Measurement with Conceptual Knowledge: Establishing the Validity and Diagnostic Capability of Concept Inventories

这一合作研究项目对概念清单(CI)的设计和验证实施了一种综合的、多学科的方法。其目的是促进和加强STEM教育中概念清单的有效使用，以进行形成性和终结性评估。为了实现这一目标，该项目将侧重于一个具体的概念清单，即热运输概念清单(TTCI)。数据将从不同的本科生样本中收集。这项研究将为诊断性测量和报告制定一套明确的理解方面，收集关于TTCI的学生表现的多层次经验数据(包括学生礼仪和面谈研究，以及用于心理测量建模的大规模数据)，将根据这些经验数据分析修订项目和方面，并收集关于修订工具的进一步数据。关于TTCI的数据将根据其在科学和工程关键领域中利用的基本概念知识和技能进行分析。更多关于学生推理和表现的经验数据将应用侧重于诊断性建模的现代心理测量分析技术进行分析。预期的研究成果是：(1)可通过网络获得改进的TTCI，并附以供教员使用的解释性工具；(2)对TTCI的评估效度和诊断能力进行深入、多层次的分析；(3)全面清点可能形成的和终结性的信息输入，包括STEM从业者充分利用信息输入所需的内容；以及(4)设计、开发、评估、改进和/或实施其他STEM领域的信息输入的研究和开发框架。总的来说，这项研究将有助于更好地了解学生的学习情况，以及更真实的诊断性和形成性评估，从而更好地学习和教授STEM中的重要概念。TTCI的使用将导致学生在热工和运输工程科学中产生误解，如流体力学、传热学和热力学。TTCI特别关注被认为对工科本科生来说重要但困难的概念。因此，拟议的研究可能导致将TTCI和其他信息来源扩大用于形成性课堂目的以及各种总结性目的。该项目既是心理测量模型和将诊断模型应用于信息系统的方法的扩展，也是在科学和工程关键概念的测量和评估领域的评估有效性的扩展。

This collaborative research project implements a comprehensive, multidisciplinary approach to the design and validation of concept inventories (CIs). The purpose is to braoden and enhance the effective use of concept inventories in STEM education for formative and summative assessment. To accomplish this goal the project will focus on a specific concept inventory, the Thermal Transport Concept Inventory (TTCI). The data will be collected from the varied undergarduate student samples. The research would develop an explicit set of facets of understanding for diagnostic measurement and reporting, would gather multi-level empirical data of student performance on the TTCI (including student protocol and interview studies, and large scale data for psychometric modeling), would revise both the items and facets based upon these empirical data analyses, and would gather further data on the revised instrument. The data on TTCI will be analyzed in terms of the underlying conceptual knowledge and skills that it taps in critical areas of science and engineering. Additional empirical data on student reasoning and performance will be analyzed applying modern psychometric analysis techniques focused on diagnostic modeling. Research outcomes expected are: (1) an improved TTCI available through the web and accompanied by interpretive tools for use by faculty; (2) an in-depth, multi-level analysis of the assessment validity and diagnostic capacity of the TTCI; (3) a comprehensive inventory of possible formative and summative uses of CIs, including what is needed by STEM practitioners to make good use of CIs; and (4) a research and development framework for designing, developing, evaluating, improving, and/or implementing CIs in other STEM areas. In general, the study will contribute to the better understanding of student learning and more authentic diagnostic and formative assessment, leading to better learning and teaching of important concepts in STEM. The use of the TTCI will elicit student misconceptions in the thermal and transport engineering sciences such as fluid mechanics, heat transfer, and thermodynamics. The TTCI particularly focuses on concepts deemed important but difficult for undergraduate engineering students. Thus, the proposed research is likely to lead to expanded uses of the TTCI and other CIs for formative classroom purposes as well as for a variety of summative purposes. This project are both an expansion of the psychometric modeling and methods for applying diagnostic models to CIs, and an expanded assessment validity in the area of measurement and assessment of key concepts in science and engineering.