Research on Automated Formative Feedback of Problem-Solving Strategy Writing in Introductory Physics using Natural Language Processing

解决问题是21世纪STEM劳动力的基本技能，但在大多数本科STEM入门课程中，学生使用无效的解决问题策略。教学上的挑战是，这些课程通常有非常大的注册人数，这使得教师无法向学生提供关于他们非生产性策略的实时反馈，因为提供这种个性化反馈是多么的劳动密集型。研究人员建议使用自然语言处理（NLP）来开发一个自动化系统，该系统将为本科物理课程的学生提供关于他们解决问题策略的个性化形成性反馈。有两个研究问题：(1)NLP分类器对这些文章的评分有多准确，它对不同类型的问题的概括有多好？(2)为学生提供这样的反馈在多大程度上提高了他们解决问题的策略和实际解决问题的能力？该项目利用了STEM教育的现有研究，这些研究表明，使用策略写作和实时形成性评估可以提高学生解决问题的能力。从大量现有数据开始，研究人员将使用最先进的监督和无监督学习来训练机器学习算法（MLA），以对学生所表达的策略进行分类。他们的研究将跨越三个阶段，逐渐泛化，首先泛化学生用来描述他们解决特定问题的策略的词汇，然后泛化使用相同策略的同构问题集，最后泛化具有相同基本原理但表面特征不同的迁移问题类。他们的目标是确定MLA可以训练的准确性，以便对学生写的描述他们解决问题策略的短文提供反馈。他们还将调查在何种程度上实时形成反馈给学生，基于他们自己的策略论文，可以帮助他们迭代地完善他们的策略，以解决日益普遍的问题。在开发和训练机器学习算法的过程中，研究人员将实施经过验证的策略，以解决人工智能（AI）中的公平性问题。该项目的最终目标是改变STEM本科生专家型问题解决策略的发展，从而有可能提高他们在STEM专业的保留率。本项目由美国国家科学基金会EDU核心研究（ECR）项目支持。ECR项目强调在该领域产生基础知识的基础STEM教育研究。投资在至关重要、广泛和持久的关键领域：STEM学习和STEM学习环境，扩大STEM参与，以及STEM劳动力发展。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Problem solving is an essential 21st century STEM workforce skill, but in most undergraduate introductory STEM courses students use ineffective problem-solving strategies. The pedagogical challenge is that these courses typically have very large enrollments which prohibits instructors' providing of real-time feedback to students on their unproductive strategies, given how labor intensive providing such individualized feedback is. The investigators propose to use natural language processing (NLP) to develop an automated system that will provide personalized formative feedback to students in undergraduate physics courses about their problem-solving strategies. There are two research questions: (1) How accurately can an NLP classifier score such essays and how well can it generalize across types of problems?, and (2) How well does providing students with such feedback improve their problem-solving strategies and actual ability to solve problems? This project leverages existing research in STEM education that has shown that the use of strategy writing, and real-time formative assessment can improve students' problem-solving skills. Starting with a vast corpus of existing data, the investigators will use state-of-the-art supervised and unsupervised learning to train a machine learning algorithm (MLA) to classify strategies articulated by students. Their study will span three phases with increasing generalization, first generalizing over vocabulary used by students to describe their strategies for solving specific problems, then generalizing over sets of isomorphic problems using the same strategies, and finally generalizing over classes of transfer problems that have the same underlying principles but different surface features. Their goal is to determine the accuracy with which an MLA can be trained to provide feedback on short essays written by students describing their strategies for solving a problem. They will also investigate the extent to which real-time formative feedback to students, based on their own strategy essays, can help them to refine their strategies iteratively in order to solve problems with increasing generality. In developing and training their machine learning algorithm, the researchers will implement proven strategies to address issues of fairness in artificial intelligence (AI). The ultimate goal of the project is to transform the development of expert-like problem-solving strategies in STEM undergraduates and thereby potentially to increase their retention in STEM majors.This project is supported by NSF's EDU Core Research (ECR) program. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field. Investments are made in critical areas that are essential, broad, and enduring: STEM learning and STEM learning environments, broadening participation in STEM, and STEM workforce development.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.