Applying Multidimensional Item Response Theory Models to Generate an Interconnected Bank of Items for Earth System Science

该项目将调查应用高质量的测试开发策略，以评估在本科阶段教授的地球系统科学课程的学习潜力。评估学生在大学一级的学习是困难的，这一困难源于美国学院和大学的多样性以及这些机构内教授的课程的多样性。该项目小组将寻求为大学教师提供一个评估资源，使他们能够通过使用一个共同的问题库来评估他们的课程学习。这些问题的设计将允许对不同的机构环境进行有意义的评估，从大型研究机构到部落学院。此外，不同的学生群体将包括在研究中，从非生物科学专业到高年级本科生。这项工作建立在先前开发的一套地球科学-地球物理为重点的问题，其中一些是与部落学院的同事合作开发的。该项目将提供：1）一个测试问题库，来自不同机构的教师可以用来评估学生的学习，提供一个真正了解学习是否发生以及何时发生的机制; 2）一个评估工具开发和分析的模型，可以被其他科学和工程领域的同事所遵循; 3）培训教师，特别是部落学院的教师，开发和使用在他们自己的机构和课程背景下有意义的评估问题。本项目将采用高质量的心理测量学标准，以目前的定量评估实践中使用的高等教育地球系统科学。这项工作将应用先进的多维项目反应理论（MIRT）技术的项目和规模的验证，以及评分的多个层面的地球系统科学的理解。数据将从大一新生、就读于非地球科学专业课程的学生、就读于全国学校的地球科学专业学生和就读于部落学院的学生中收集。这种广泛的数据收集将允许准确估计能力测量和使用差异项目功能（DIF）来评估潜在的偏倚。为概念清单项目的相互关联提供机制对于保证学习成果的测量在整个教育环境中是有意义的，并且可以有效地用于为高等教育政策和实践提供信息至关重要。

This project will investigate the potential for applying high quality test development strategies to assessment of learning in Earth System Science courses taught at the undergraduate level. Assessing student learning at the collegiate level is difficult, a difficulty that stems from the diversity of U.S. colleges and universities and the diversity of curriculum taught within those institutions. The project team will seek to provide college faculty with an assessment resource that will allow them to evaluate learning in their courses through use of a common bank of questions. These questions will be designed to allow meaningful assessment across different institutional settings, from large research institutions to tribal colleges. In addition, different populations of students will be included in the study, from non-­science majors through advanced undergraduates. This work builds upon prior development of a set of geoscience-­focused questions, some of which were developed in collaboration with tribal college colleagues. This project will provide: 1) a bank of test questions that faculty from diverse institutions can use to evaluate student learning, providing a mechanism for truly understanding if and when learning is occurring; 2) a model for assessment instrument development and analysis that can be followed by colleagues within other science and engineering fields; and 3) training for faculty, particularly tribal college faculty, for development and use of assessment questions that are meaningful within their own institutional and course contexts.This project will apply high quality psychometric standards to the current quantitative assessment practice utilized in higher education Earth System Science. This work will apply cutting-­edge Multidimensional Item Response Theory (MIRT) techniques to item and scale validation as well as scoring of multiple dimensions of Earth System Science understanding. Data will be collected from freshmen, students enrolled in non-­majors courses, geoscience majors enrolled in schools nationally, and students enrolled in tribal colleges. This broad data collection will allow for accurate estimation of ability measures and use of Differential Item Functioning (DIF) to evaluate potential bias. Providing mechanisms for inter-correlation of concept inventory items is vital to guarantee that measurement of learning outcomes is meaningful across educational contexts and can be effectively used to inform higher education policy and practice.