Applying Multidimensional Item Response Theory Models to Generate an Interconnected Bank of Items for Earth System Science

这个项目将调查将高质量的测试发展战略应用于评估本科地球系统科学课程学习的可能性。评估学生在大学水平上的学习是困难的，这种困难源于美国学院和大学的多样性以及这些机构所教授的课程的多样性。项目团队将寻求为大学教师提供一个评估资源，使他们能够通过使用一个共同的问题库来评估他们在课程中的学习情况。这些问题的设计将允许在不同的机构环境中进行有意义的评估，从大型研究机构到部落学院。此外，从非理科生到高级本科生，不同群体的学生将被纳入研究。这项工作建立在先前开发的一系列以地球科学为重点的问题的基础上，其中一些问题是与部落大学同事合作开发的。该项目将提供：1)一个测试题库，来自不同机构的教师可以使用它来评估学生的学习情况，提供一种机制来真正了解学习是否发生以及何时发生；2)其他科学和工程领域的同事可以遵循的评估工具开发和分析模型；3)培训教师，特别是部落学院的教师，开发和使用在他们自己的机构和课程环境中有意义的评估问题。本项目将把高质量的心理测量标准应用于目前高等教育中地球系统科学定量评估实践。这项工作将应用尖端的多维项目反应理论（MIRT）技术来进行项目和规模验证，以及对地球系统科学理解的多个维度进行评分。数据将从新生、非专业课程的学生、全国学校的地球科学专业学生和部落学院的学生中收集。这种广泛的数据收集将允许准确估计能力措施和使用差异项目功能（DIF）来评估潜在的偏见。提供概念清单项目的相互关联机制对于保证学习成果的测量在教育背景下是有意义的，并且可以有效地用于高等教育政策和实践至关重要。

This project will investigate the potential for applying high quality test development strategies to assessment of learning in Earth System Science courses taught at the undergraduate level. Assessing student learning at the collegiate level is difficult, a difficulty that stems from the diversity of U.S. colleges and universities and the diversity of curriculum taught within those institutions. The project team will seek to provide college faculty with an assessment resource that will allow them to evaluate learning in their courses through use of a common bank of questions. These questions will be designed to allow meaningful assessment across different institutional settings, from large research institutions to tribal colleges. In addition, different populations of students will be included in the study, from non-­science majors through advanced undergraduates. This work builds upon prior development of a set of geoscience-­focused questions, some of which were developed in collaboration with tribal college colleagues. This project will provide: 1) a bank of test questions that faculty from diverse institutions can use to evaluate student learning, providing a mechanism for truly understanding if and when learning is occurring; 2) a model for assessment instrument development and analysis that can be followed by colleagues within other science and engineering fields; and 3) training for faculty, particularly tribal college faculty, for development and use of assessment questions that are meaningful within their own institutional and course contexts.This project will apply high quality psychometric standards to the current quantitative assessment practice utilized in higher education Earth System Science. This work will apply cutting-­edge Multidimensional Item Response Theory (MIRT) techniques to item and scale validation as well as scoring of multiple dimensions of Earth System Science understanding. Data will be collected from freshmen, students enrolled in non-­majors courses, geoscience majors enrolled in schools nationally, and students enrolled in tribal colleges. This broad data collection will allow for accurate estimation of ability measures and use of Differential Item Functioning (DIF) to evaluate potential bias. Providing mechanisms for inter-correlation of concept inventory items is vital to guarantee that measurement of learning outcomes is meaningful across educational contexts and can be effectively used to inform higher education policy and practice.