Defining Almost Correct: Quantifying Student Understanding Hidden in Wrong Answers

该项目旨在改变教师评估学生在本科物理课程学习的方式。对学生物理学习的定量评估通常集中在学生在多项选择题考试中是否得到了“正确”的答案。这种分析无法捕捉到学生的理解与“正确”的距离有多近。“因此，它无法跟踪学生的理解是否或如何改善或进步。 这个项目将为多项选择题开发复杂的评分方法，这些方法可以揭示学生富有成效但“错误”的想法。这种方法应导致更好地了解和更公平的决定有关的教学实践。例如，这种分析可以确定学生在教学后是否提高了理解力，即使他们仍然没有得到正确的答案。这些信息对准备不足的学生特别有帮助，因为他们在学习过程中还有很长的路要走。由于这些学生中有不成比例的人数来自物理学代表性不足的群体，因此使用简单的对/错分析不公平地对待这些群体。更充分地代表学生的学习对于做出更好的教学实践决策至关重要。这个项目将使用两种分析方法来确定一组选择中哪些错误的答案比其他答案更好。第一个过程是统计分析，确定学生在得出正确答案之前最常见的错误答案序列。第二个过程包括定性访谈，以揭示学生对他们选择的原因的思考。更好地了解学生在多个学科中的学习进展，应有助于培养更大，更包容的STEM劳动力，以满足美国经济不断增长的需求。本项目旨在通过开发一种新的评估工具，以新颖的方式来衡量学生的学习，以满足对基于研究的评估工具的更完整的评分指标的迫切需要。新的方法是基于一个常用的研究为基础的评估工具，在物理，力和运动概念评估的数据。这个项目将开始，根据对学生反应的定量分析，对力和运动概念评估中的每个问题的不正确反应进行排名。接下来，它将协调来自多个分析的排名，为每个问题生成统一的排名。从这个统一的排名，该项目将定义一个指标来代表学生的整体知识。然后，该项目将开发一个用户友好的评估工具（软件）来分析学生的反应数据，并计算新的学习指标。接下来，通过将新的评估工具应用于现有数据，该项目将确定基于教学因素或学生人口统计数据而不同的学生反应进展模式。最后，该项目将通过采访学生，了解他们为什么选择不同的答案，将统一的排名与学习进展文献联系起来。其结果将是一个新的评估工具，应该允许研究人员和教师更深入地分析有关学生学习的数据。这种知识可以导致更明智的决定，关于教学的选择。这个项目的结果可以应用于任何学科的任何多项选择，以研究为基础的评估工具。因此，该项目有可能显着改善学习数据在许多不同的上下文中解释的方式。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估。

This project aims to transform the way faculty assess student learning in undergraduate physics courses. Quantitative assessments of student learning in physics have generally focused on whether a student got the "right" answer on a multiple-choice exam. This kind of analysis fails to capture how close a student's understanding is to being "right." It therefore cannot track whether or how students' understanding improves or progresses. This project will develop sophisticated scoring methods for multiple-choice tests that can reveal students' productive, but "wrong" ideas. This method should result in better-informed and more equitable decisions regarding instructional practices. For instance, this kind of analysis could determine if students have improved their understanding after instruction, even if they still do not get the right answer. Such information could be particularly helpful with less-prepared students who have further to go in their learning process. Since a disproportionate number of these students are from groups that are under-represented in physics, using a simple right/wrong analysis treats these groups inequitably. More fully representing student learning is vitally important for making better decisions regarding instructional practices. This project will use two kinds of analysis to determine which wrong answers from a set of choices are better than others. The first process is a statistical analysis that determines the most common sequence of wrong answers students traverse before arriving at the correct answer. The second process involves qualitative interviews to uncover student thinking about the reasons for their choices.Better understanding the progression of student learning across multiple disciplines should help develop a larger and more inclusive STEM work force to meet the growing needs of the U.S. economy. This project aims to meet the critical need for a more complete scoring metric for research-based assessment instruments, by developing a new assessment tool to measure student learning in novel ways. The new approach is based on data from a commonly used research-based assessment instruments in physics, the Force and Motion Conceptual Evaluation. This project will begin by developing a ranking of incorrect responses to each question on the Force and Motion Conceptual Evaluation, based on quantitative analyses of student responses. Next, it will reconcile rankings from multiple analyses to generate a unified ranking for each question. From this unified ranking, the project will define a metric to represent overall student knowledge. Then, the project will develop a user-friendly assessment tool (software) to analyze student response data and calculate the new learning metric. Next, by applying the new assessment tool to existing data, the project will identify patterns in student response progressions that differ based on instructional factors or student demographics. Finally, the project will connect the unified rankings to the learning progressions literature by interviewing students about why they choose various responses. The outcome will be a new assessment tool that should allow researchers and instructors to more deeply analyze data about their students' learning. This knowledge could lead to more informed decisions regarding instructional choices. The results of this project may be applied to any multiple-choice, research-based assessment instruments in any discipline. Thus, the project has the potential to significantly improve the ways in which data about learning are interpreted in many different contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.