Defining Almost Correct: Quantifying Student Understanding Hidden in Wrong Answers

定义几乎正确：量化隐藏在错误答案中的学生理解

• 批准号: 1836470
• 负责人: Trevor Smith
• 金额: $ 20.69万
• 依托单位: Rowan University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1836470&HistoricalAwards=false
• 关键词: Defining; Almost; Correct; Quantifying; Student

This project aims to transform the way faculty assess student learning in undergraduate physics courses. Quantitative assessments of student learning in physics have generally focused on whether a student got the "right" answer on a multiple-choice exam. This kind of analysis fails to capture how close a student's understanding is to being "right." It therefore cannot track whether or how students' understanding improves or progresses. This project will develop sophisticated scoring methods for multiple-choice tests that can reveal students' productive, but "wrong" ideas. This method should result in better-informed and more equitable decisions regarding instructional practices. For instance, this kind of analysis could determine if students have improved their understanding after instruction, even if they still do not get the right answer. Such information could be particularly helpful with less-prepared students who have further to go in their learning process. Since a disproportionate number of these students are from groups that are under-represented in physics, using a simple right/wrong analysis treats these groups inequitably. More fully representing student learning is vitally important for making better decisions regarding instructional practices. This project will use two kinds of analysis to determine which wrong answers from a set of choices are better than others. The first process is a statistical analysis that determines the most common sequence of wrong answers students traverse before arriving at the correct answer. The second process involves qualitative interviews to uncover student thinking about the reasons for their choices.Better understanding the progression of student learning across multiple disciplines should help develop a larger and more inclusive STEM work force to meet the growing needs of the U.S. economy. This project aims to meet the critical need for a more complete scoring metric for research-based assessment instruments, by developing a new assessment tool to measure student learning in novel ways. The new approach is based on data from a commonly used research-based assessment instruments in physics, the Force and Motion Conceptual Evaluation. This project will begin by developing a ranking of incorrect responses to each question on the Force and Motion Conceptual Evaluation, based on quantitative analyses of student responses. Next, it will reconcile rankings from multiple analyses to generate a unified ranking for each question. From this unified ranking, the project will define a metric to represent overall student knowledge. Then, the project will develop a user-friendly assessment tool (software) to analyze student response data and calculate the new learning metric. Next, by applying the new assessment tool to existing data, the project will identify patterns in student response progressions that differ based on instructional factors or student demographics. Finally, the project will connect the unified rankings to the learning progressions literature by interviewing students about why they choose various responses. The outcome will be a new assessment tool that should allow researchers and instructors to more deeply analyze data about their students' learning. This knowledge could lead to more informed decisions regarding instructional choices. The results of this project may be applied to any multiple-choice, research-based assessment instruments in any discipline. Thus, the project has the potential to significantly improve the ways in which data about learning are interpreted in many different contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在改变教师在本科物理课程中评估学生学习的方式。对学生物理学习的定量评估通常集中在学生是否在多项选择题中得到“正确”答案上。这种分析没有捕捉到学生的理解离“正确”有多近。因此，它无法跟踪学生的理解是否或如何提高或进步。该项目将为多项选择题测试开发复杂的评分方法，以揭示学生富有成效但“错误”的想法。这种方法应该会导致关于教学实践的更明智和更公平的决定。例如，这种分析可以确定学生在接受指导后是否提高了理解能力，即使他们仍然没有得到正确的答案。这些信息对那些在学习过程中还有很长的路要走的准备不足的学生特别有帮助。由于这些学生中不成比例的一部分来自物理学科代表性不足的群体，使用简单的对/错分析对这些群体是不公平的。更充分地反映学生的学习情况对于制定更好的教学实践决策至关重要。这个项目将使用两种分析来确定从一组选择中哪些错误的答案比其他的更好。第一个过程是统计分析，确定学生在得出正确答案之前最常见的错误答案顺序。第二个过程包括定性访谈，以揭示学生思考他们选择的原因。更好地了解学生跨学科学习的进展，有助于培养一支更大、更具包容性的STEM劳动力队伍，以满足美国经济日益增长的需求。该项目旨在通过开发一种新的评估工具，以新颖的方式衡量学生的学习情况，以满足对基于研究的评估工具更完整的评分指标的迫切需求。新方法是基于物理学中常用的基于研究的评估工具，力和运动概念评估的数据。这个项目将首先根据学生回答的定量分析，对力和运动概念评估中每个问题的错误回答进行排名。接下来，它将协调多个分析的排名，为每个问题生成统一的排名。从这个统一的排名中，该项目将定义一个度量来表示学生的整体知识。然后，该项目将开发一个用户友好的评估工具（软件）来分析学生的反应数据和计算新的学习指标。接下来，通过将新的评估工具应用于现有数据，该项目将根据教学因素或学生人口统计数据确定学生反应进展的模式。最后，该项目将通过采访学生，了解他们为什么选择不同的答案，将统一的排名与学习进展文献联系起来。结果将是一种新的评估工具，使研究人员和教师能够更深入地分析有关学生学习的数据。这些知识可以使学生在选择教学时做出更明智的决定。这个项目的结果可以应用于任何学科的任何选择题、研究型评估工具。因此，该项目有可能显著改善在许多不同背景下解释学习数据的方式。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Quantitatively ranking incorrect responses to multiple-choice questions using item response theory

使用项目反应理论对多项选择题的错误反应进行定量排名

• DOI: 10.1103/physrevphyseducres.16.010107
• 发表时间: 2020
• 期刊: Physical Review Physics Education Research
• 影响因子: 3.1
• 作者: Smith, Trevor I.;Louis, Kyle J.;Ricci, Bartholomew J.;Bendjilali, Nasrine
• 通讯作者: Bendjilali, Nasrine

Motivations for using the item response theory nominal response model to rank responses to multiple-choice items

使用项目反应理论名义反应模型对多项选择项目的反应进行排名的动机

• DOI: 10.1103/physrevphyseducres.18.010133
• 发表时间: 2022
• 期刊: Physical Review Physics Education Research
• 影响因子: 3.1
• 作者: Smith, Trevor I.;Bendjilali, Nasrine
• 通讯作者: Bendjilali, Nasrine

Replicating analyses of item response curves using data from the Force and Motion Conceptual Evaluation

使用力和运动概念评估的数据重复分析项目响应曲线

• DOI: 10.1103/physrevphyseducres.17.020127
• 发表时间: 2021
• 期刊: Physical Review Physics Education Research
• 影响因子: 3.1
• 作者: Richardson, Connor J.;Smith, Trevor I.;Walter, Paul J.
• 通讯作者: Walter, Paul J.

Comparing pre/post item response curves to identify changes in misconceptions

比较前/后项目反应曲线以识别误解的变化

• DOI: 10.1119/perc.2021.pr.walter
• 发表时间: 2021
• 期刊: 2021 PERC Proceedings
• 作者: Walter, Paul J.;Smith, Trevor I.
• 通讯作者: Smith, Trevor I.