Harvesting Actionable Results for Learning and Instruction: A Novel Mixed Methods Approach to Extracting and Validating Information from Diagnostic Assessment

认知诊断模型（CDMs）是心理测量学研究的一个领域，在过去的十年中有了长足的发展。这些评估工具受到了更多的关注，因为简单的整体测试分数并不能满足教学目标，因为需要对学生的技能进行更丰富的评估，以支持量身定制的教学。研究者建议扩展现有的CDM心理测量学方法，开发基于高中物理课程的形成性评价设计的诊断方面状态模型（DFSM）。现有清洁发展机制存在一些工作上的关切。它们不能很好地扩展到高维小面空间，它们需要基于专家输入的复杂编码方案，它们不能探索复杂的小面关系或基于选项的名义响应。本项目将以现有的清洁发展机制办法为基础，克服这些限制。该项目开发了一个新的心理测量模型，DFSM，和纵向扩展，它可以同时模拟高维目标和项目选项水平的中间理解。PI开发了一种机器学习方法来识别面之间的关系，产生一个全面的“面图”，揭示了属性层次结构和物理学习的联合面。最后，研究人员进行定性研究，以验证DFSM和纵向DFSM的输出为学习和教学。（ECR：BCSER）程序，该计划旨在培养调查人员的能力，高质量的STEM教育研究。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值进行评估，被认为值得支持和更广泛的影响审查标准。

Cognitive diagnostic models (CDMs) are an area of psychometric research that has seen substantial growth in the past decade. These assessment tools have received more attention because a simple overall test score does not serve teaching goals as a richer evaluation of the student's skills is needed to support tailored instruction. The investigator proposes to extend existing CDM psychometric approaches to develop the diagnostic facet status model (DFSM) for formative assessment design based on the high school physics curriculum. There are a number of working concerns with existing CDMs. They do not scale well to high-dimensional facet spaces, they require complex coding schemes based on expert input, they cannot explore intricate facet relationships, or option-based nominal responses. This project will build on existing CDM approaches to address these limitations. The project develops a new psychometric model, the DFSM, and a longitudinal extension, which can simultaneously model high-dimensional goal and intermediate understandings at item option level. The PIs develop a machine learning method to identify relations among facets, yielding a comprehensive "facet map" that reveals both attribute hierarchies and conjoined facets for the learning of physics. Finally, the researchers conduct qualitative studies to validate DFSM and longitudinal DFSM's output for learning and instruction.The project is supported by NSF's EDU Core Research Building Capacity in STEM Education Research (ECR: BCSER) program, which is designed to build investigators' capacity to carry out high-quality STEM education research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.