Defining Almost Correct: Quantifying Student Understanding Hidden in Wrong Answers

该项目旨在改变教师在本科物理课程中评估学生学习的方式。对学生物理学习的定量评估通常关注的是学生在多项选择考试中是否得到了正确的答案。这种分析无法捕捉到一个学生的理解离“正确”有多近。因此，它无法跟踪学生的理解力是否或如何提高或进步。这个项目将为多项选择测试开发复杂的评分方法，以揭示学生们富有成效但却“错误”的想法。这种方法应该会导致关于教学实践的更知情和更公平的决定。例如，这种分析可以确定学生在教学后是否提高了他们的理解力，即使他们仍然没有得到正确的答案。这些信息可能对准备不足的学生特别有帮助，因为他们在学习过程中还有更长的路要走。由于这些学生中有不成比例的人数来自物理学中代表性不足的群体，因此使用简单的对/错分析来对待这些群体是不公平的。更全面地代表学生的学习对于更好地做出关于教学实践的决策至关重要。这个项目将使用两种分析来确定一组选择中哪一个错误的答案比其他的更好。第一个过程是统计分析，确定学生在得出正确答案之前最常见的错误答案顺序。第二个过程包括定性访谈，以揭示学生对他们选择的原因的思考。更好地了解学生跨学科学习的进展应该有助于培养一支更大、更具包容性的STEM劳动力队伍，以满足美国经济日益增长的需求。该项目旨在通过开发一种新的评估工具，以新颖的方式衡量学生的学习，以满足对研究型评估工具更完整的评分标准的迫切需求。新方法基于物理中常用的研究型评估工具--力和运动概念评估的数据。本项目将在对学生回答的量化分析的基础上，对力和运动概念评估中的每个问题的错误回答进行排名。接下来，它将协调多个分析的排名，为每个问题生成统一的排名。根据这个统一的排名，该项目将定义一个衡量标准来代表学生的整体知识。然后，该项目将开发一个用户友好的评估工具(软件)来分析学生的反应数据并计算新的学习指标。接下来，通过将新的评估工具应用于现有数据，该项目将识别基于教学因素或学生人口统计数据而不同的学生反应进度模式。最后，该项目将通过采访学生为什么选择不同的回答，将统一排名与学习进展文献联系起来。结果将是一种新的评估工具，应该允许研究人员和教师更深入地分析关于学生学习的数据。这一知识可能会导致关于教学选择的更明智的决定。本项目的结果可以应用于任何学科的任何多项选择、基于研究的评估工具。因此，该项目有可能显著改善在许多不同背景下解释学习数据的方式。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to transform the way faculty assess student learning in undergraduate physics courses. Quantitative assessments of student learning in physics have generally focused on whether a student got the "right" answer on a multiple-choice exam. This kind of analysis fails to capture how close a student's understanding is to being "right." It therefore cannot track whether or how students' understanding improves or progresses. This project will develop sophisticated scoring methods for multiple-choice tests that can reveal students' productive, but "wrong" ideas. This method should result in better-informed and more equitable decisions regarding instructional practices. For instance, this kind of analysis could determine if students have improved their understanding after instruction, even if they still do not get the right answer. Such information could be particularly helpful with less-prepared students who have further to go in their learning process. Since a disproportionate number of these students are from groups that are under-represented in physics, using a simple right/wrong analysis treats these groups inequitably. More fully representing student learning is vitally important for making better decisions regarding instructional practices. This project will use two kinds of analysis to determine which wrong answers from a set of choices are better than others. The first process is a statistical analysis that determines the most common sequence of wrong answers students traverse before arriving at the correct answer. The second process involves qualitative interviews to uncover student thinking about the reasons for their choices.Better understanding the progression of student learning across multiple disciplines should help develop a larger and more inclusive STEM work force to meet the growing needs of the U.S. economy. This project aims to meet the critical need for a more complete scoring metric for research-based assessment instruments, by developing a new assessment tool to measure student learning in novel ways. The new approach is based on data from a commonly used research-based assessment instruments in physics, the Force and Motion Conceptual Evaluation. This project will begin by developing a ranking of incorrect responses to each question on the Force and Motion Conceptual Evaluation, based on quantitative analyses of student responses. Next, it will reconcile rankings from multiple analyses to generate a unified ranking for each question. From this unified ranking, the project will define a metric to represent overall student knowledge. Then, the project will develop a user-friendly assessment tool (software) to analyze student response data and calculate the new learning metric. Next, by applying the new assessment tool to existing data, the project will identify patterns in student response progressions that differ based on instructional factors or student demographics. Finally, the project will connect the unified rankings to the learning progressions literature by interviewing students about why they choose various responses. The outcome will be a new assessment tool that should allow researchers and instructors to more deeply analyze data about their students' learning. This knowledge could lead to more informed decisions regarding instructional choices. The results of this project may be applied to any multiple-choice, research-based assessment instruments in any discipline. Thus, the project has the potential to significantly improve the ways in which data about learning are interpreted in many different contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.