Collaborative Research: Machine Learning for Student Reasoning during Challenging Concept Questions

人工智能(AI)，更具体地说，语言模型，已经极大地改变了学生和教师对学习和评估的看法。虽然人们有理由担心这些工具的使用可能会对学习不利，但该研究项目旨在利用语言模型更好地为21世纪的工程学习者做好准备。该研究项目将使用现代人工智能和机器学习(ML)工具来自动分析学生对挑战性概念问题的书面回答。这些定性问题经常用于STEM大班，以支持主动学习教学法；它们需要最少的计算，并侧重于将潜在的物理和化学现象应用于各种情况。在之前的国家科学基金会的资助下，我们开发了概念仓库(国家科学基金会截止日期1023099,1821439,2135190)，这是一个课堂回答系统，学生可以在其中提供书面理由来回答概念问题。提供书面理由的目标是发展学生的推理和理解能力，还可以使他们更好地为与同龄人的讨论做好准备，从而使积极学习的教学更有效。然而，说明性散文也为教师提供了大量需要处理的信息。在这个项目中，我们将利用机器学习工具和自然语言处理技术的最新进展来开发自动化过程，以分析学生对挑战性概念问题的书面理由。该项目将加入塔夫茨大学的工程教育研究人员和马萨诸塞大学洛厄尔大学的AI/ML研究人员。我们将集中于以下研究问题：(1)在人类编码的基础上，学生在解释静力学中具有挑战性的概念问题时使用了什么想法？这些在所研究的具有挑战性的概念问题中有何不同？(2)基于Transformer的ML模型能够多好地复制人类编码员所做的编码？对于同构的问题对，根据第一个问题的解释训练的ML模型在第二个问题上的表现如何？更广泛地说，基于一个问题的基于ML的编码可以成功地应用于为其他问题编码数据吗？这种通用性的限制是什么？我们将完成三项研究任务：(1)收集来自不同机构的9名或更多工程静力学教师对同一概念问题的书面回答的数据；(2)对学生的书面解释的子集进行人工编码；(3)开发和评估ML编码方法，然后对收集的完整书面解释进行ML编码。虽然该项目的重点是工程静力学，但预计结果将转移到其他工程和科学主题的挑战性问题上。最终，机器学习的成功实施将支持具有挑战性的概念的学习和教学。预期的结果包括对不同ML方法的优点和缺点的发展理解，包括它们的准确性，确定应用算法的最小数据大小要求，以及将学习从一个问题转移到需要类似推理模式的同构问题的能力。对于教师来说，生成的数据可以提供有关学生通过常见案例的例子进行推理的不同方式的实时信息。对于工程教育研究人员来说，在不同的环境中进行个性化的解释将有助于对学生思维如何与教学实践和环境相关的调查。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial Intelligence (AI), and more specifically, language models, have been drastically changing how students and instructors think about learning and assessment. While there are legitimate concerns about how the use of these tools could be detrimental to learning, this research project aims to leverage language models to better prepare engineering learners of the 21st Century. The research project will use modern AI and machine learning (ML) tools to automate analysis of student-written responses to challenging concept questions. These qualitative questions are often used in large STEM classes to support active learning pedagogies; they require minimum calculations and focus on the application of underlying physical and chemical phenomena to various situations. With previous NSF funding, we have developed the Concept Warehouse (NSF DUE 1023099, 1821439, 2135190), a classroom response system where students provide written justifications to concept questions. Providing written justifications targets development of reasoning and sense-making skills in students and can also better prepare them for discussions with peers resulting in broader effectiveness of active learning pedagogies. However, expository prose also presents a daunting amount of information for instructors to process. In this project, we will leverage recent advancements in machine learning tools and natural language processing technologies to develop automated processes to analyze student-written justifications to challenging concept questions.This project will join engineering education researchers at Tufts University and AI/ML researchers at the University of Massachusetts Lowell. We will focus on the following research questions: (1) Based on human coding, what ideas do students use in explaining challenging concept questions in statics? How do these vary among challenging concept questions studied? (2) How well can Transformer-based ML models replicate the coding done by the human coders? For isomorphic question pairs, how well do ML models trained on the first question’s explanations perform on the second question? More generally, can ML-based coding based on one question be applied successfully to code the data for other questions, and what are the limits to this generalizability? We will complete three research tasks: (1) data collection of written responses for the same concept questions from nine or more engineering statics instructors at different institutions; (2) manual coding of a subset of students’ written explanations, and (3) developing and evaluating ML coding methods, followed by ML coding of the complete set of collected written explanations. While the project focuses on engineering statics, it is expected that findings will transfer to challenging questions in other engineering and science topics. Ultimately, successful implementation of machine learning will support learning and instruction of challenging concepts. Expected outcomes include a developing understanding of advantages and disadvantages of different ML approaches including their accuracy, determination of minimum data size requirements to apply the algorithms, and the ability to transfer learning from one question to isomorphic questions that require similar reasoning patterns. For instructors, data generated can provide real-time information about the different ways students are reasoning with examples of common cases. For engineering education researchers, characterizing explanations in different settings will support investigations of how student thinking relates to instructional practices and environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.