Research on Automated Formative Feedback of Problem-Solving Strategy Writing in Introductory Physics using Natural Language Processing

解决问题是一个基本的21世纪世纪干劳动力技能，但在大多数本科入门STEM课程的学生使用无效的解决问题的策略。教学上的挑战是，这些课程通常有非常大的招生，这就禁止教师提供实时反馈给学生对他们的非生产性策略，考虑到劳动密集型提供这种个性化的反馈。研究人员建议使用自然语言处理（NLP）来开发一个自动化系统，该系统将为本科物理课程的学生提供关于他们解决问题策略的个性化形成性反馈。有两个研究问题：（1）NLP分类器对这些文章的评分有多准确，以及它在不同类型的问题中的泛化能力如何？（2）为学生提供这种反馈在多大程度上提高了他们解决问题的策略和实际解决问题的能力？该项目利用了STEM教育中的现有研究，这些研究表明，使用策略写作和实时形成性评估可以提高学生解决问题的能力。从大量现有数据开始，研究人员将使用最先进的监督和无监督学习来训练机器学习算法（MLA），以分类学生表达的策略。他们的研究将跨越三个阶段，随着泛化的增加，首先概括学生用来描述他们解决特定问题的策略的词汇，然后概括使用相同策略的同构问题集，最后概括具有相同基本原则但不同表面特征的迁移问题类。他们的目标是确定MLA的准确性，以便对学生撰写的描述解决问题策略的短文提供反馈。他们还将调查在何种程度上实时形成反馈给学生，根据他们自己的策略论文，可以帮助他们迭代地完善自己的策略，以解决日益普遍的问题。在开发和训练他们的机器学习算法时，研究人员将实施经过验证的策略来解决人工智能（AI）中的公平性问题。该项目的最终目标是改变STEM本科生中专家式问题解决策略的发展，从而有可能提高他们在STEM专业的保留率。该项目由NSF的EDU核心研究（ECR）计划支持。ECR计划强调基础STEM教育研究，产生该领域的基础知识。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响力审查标准进行评估，被认为值得支持。

Problem solving is an essential 21st century STEM workforce skill, but in most undergraduate introductory STEM courses students use ineffective problem-solving strategies. The pedagogical challenge is that these courses typically have very large enrollments which prohibits instructors' providing of real-time feedback to students on their unproductive strategies, given how labor intensive providing such individualized feedback is. The investigators propose to use natural language processing (NLP) to develop an automated system that will provide personalized formative feedback to students in undergraduate physics courses about their problem-solving strategies. There are two research questions: (1) How accurately can an NLP classifier score such essays and how well can it generalize across types of problems?, and (2) How well does providing students with such feedback improve their problem-solving strategies and actual ability to solve problems? This project leverages existing research in STEM education that has shown that the use of strategy writing, and real-time formative assessment can improve students' problem-solving skills. Starting with a vast corpus of existing data, the investigators will use state-of-the-art supervised and unsupervised learning to train a machine learning algorithm (MLA) to classify strategies articulated by students. Their study will span three phases with increasing generalization, first generalizing over vocabulary used by students to describe their strategies for solving specific problems, then generalizing over sets of isomorphic problems using the same strategies, and finally generalizing over classes of transfer problems that have the same underlying principles but different surface features. Their goal is to determine the accuracy with which an MLA can be trained to provide feedback on short essays written by students describing their strategies for solving a problem. They will also investigate the extent to which real-time formative feedback to students, based on their own strategy essays, can help them to refine their strategies iteratively in order to solve problems with increasing generality. In developing and training their machine learning algorithm, the researchers will implement proven strategies to address issues of fairness in artificial intelligence (AI). The ultimate goal of the project is to transform the development of expert-like problem-solving strategies in STEM undergraduates and thereby potentially to increase their retention in STEM majors.This project is supported by NSF's EDU Core Research (ECR) program. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field. Investments are made in critical areas that are essential, broad, and enduring: STEM learning and STEM learning environments, broadening participation in STEM, and STEM workforce development.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.