Excellence in Research: Exploring Effectiveness of Automatic Assessment of Cognitive and Metacognitive Processes in Engineering Learning through Natural Language Processing Models

及时评估学生的学习情况对于满足他们的需求至关重要。然而，目前的评估方法，如多项选择或计算测试，可能无法真正揭示学生的内部思维过程，因为学生可能会猜测答案或遵循一步一步的例子程序，而没有概念的理解。已有的研究表明，让学生写下自己的答案的理由，并计划和反思他们的学习，可以促进他们发展更深层次的概念理解和应用认知和元认知策略在他们的学习。然而，由于评估自由文本答复的耗时性，这种方法没有得到广泛采用。该项目旨在开发一个基于规则的自动评估工具，从学生的自由文本回答中识别个别学生的误解和学习缺陷，使教师和学生能够立即调整他们的教学和学习，并最终实现针对个别学习者需求的个性化指导，这特别有利于HBCU的学生。该项目还将为本科生和教师提供研究经验，并为包括公立学校学生在内的各种受众提供基础，提高公众对人工智能的认识。该研究提出了创新策略，通过自我监督学习和少量学习来微调和校准预先训练的语言模型，用于基于用户指定的标题自动分类文本，特别关注学习者的认知和元认知过程。该模型集成了三个新颖的属性，用于提高文本和标题关键词之间的相似性比较：（1）被比较文本之间的交叉关注，以提高比较的灵敏度;（2）词、短语和句子的联合嵌入，以提高比较的准确性;（3）纳入主题相关性，以增强比较的广度。在给定评价规则的情况下，该模型可以对文本进行分类，从而从多个更细粒度的角度揭示学生的思维或其他特征。它可以灵活地添加新的评估视角，并且比整体评估更透明，并且允许涉及人类判断，从而产生更可靠的可接受的评估实践，并推进关于适应个性化指导的预训练语言模型的知识。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Timely assessment of students' learning is crucial for addressing their needs. However, current assessment methods such as multiple-choice or calculation tests may not truly reveal students’ internal thinking processes, because students may guess answers or follow the step-by-step procedures of examples without conceptual understanding. Existing research shows that asking students to write down justifications for their answers, and plan and reflection on their learning can promote them to develop deeper conceptual understanding and apply cognitive and metacognitive strategies in their learning. Nonetheless, this approach is not widely adopted due to the time-consuming nature of assessing free-text responses. This project aims to develop a rubrics-based automatic assessment tool to identify individual students' misconceptions and learning deficiencies from their free-text responses, allowing instructors and students to instantly adjust their teaching and learning, and eventually enabling personalized instructions tailored to individual learners’ needs, which particularly benefits students at HBCUs. The project will also provide the base for delivering research experience for both undergraduates and teachers and outreach various audiences, including public school students, increasing the literacy of the public on artificial intelligence.This research proposes innovative strategies to fine-tune and calibrate the pre-trained language model, through self-supervised learning and few-shot learning, for automatic classification of texts based on user-specified rubrics, with a particular focus on learners' cognitive and metacognitive processes. The proposed model integrates three novel attributes for improving similarity comparison between texts and rubric keywords: (1) across-attention between compared texts for increasing the sensitivity of comparison; (2) joint embedding of words, phrases, and sentences for improving the accuracy of comparison; (3) incorporation of thematic relevance for enhancing the breadth of comparison. Given the assessment rubrics, the model can classify texts to reveal students’ thinking or other traits in multiple finer granular perspectives. It is flexible for adding new assessment perspectives and more transparent than the overall assessment, and allows involving human judgment, leading to more reliable assessment acceptable for practice, and advancing knowledge on adapting pre-trained language models for personalized instructions. The dataset collected from diverse students and specific strategies will be adopted to mitigate the potential biases of the proposed model.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.