Excellence in Research: Exploring Effectiveness of Automatic Assessment of Cognitive and Metacognitive Processes in Engineering Learning through Natural Language Processing Models

及时评估学生的学习对满足他们的需求至关重要。然而，目前的评估方法，如选择题或计算题，可能并不能真正揭示学生的内在思维过程，因为学生可能会猜测答案或按照例子的一步一步的过程，而没有概念上的理解。现有研究表明，要求学生写下答案的理由，并对自己的学习进行计划和反思，可以促进他们在学习中形成更深层次的概念理解，并在学习中应用认知和元认知策略。尽管如此，由于评估自由文本响应的耗时特性，这种方法并没有被广泛采用。该项目旨在开发一种基于规则的自动评估工具，从学生的自由文本回答中识别出个体学生的误解和学习缺陷，使教师和学生能够立即调整他们的教学和学习，并最终实现针对个体学习者需求的个性化指导，这对hbcu的学生尤其有利。该项目还将为本科生和教师提供研究经验，并向包括公立学校学生在内的各种受众提供服务，提高公众对人工智能的认识。本研究提出了创新策略，通过自我监督学习和少量学习来微调和校准预训练的语言模型，以基于用户指定的规则对文本进行自动分类，特别关注学习者的认知和元认知过程。该模型集成了三个新属性，以提高文本和标题关键词之间的相似性比较：(1)被比较文本之间的交叉注意，提高比较的敏感性；(2)单词、短语和句子的联合嵌入，提高比较的准确性；(3)结合主题相关性，提高比较的广度。在给定评估规则的情况下，该模型可以对文本进行分类，从而从多个更细粒度的角度揭示学生的思维或其他特征。它可以灵活地添加新的评估视角，并且比整体评估更透明，并且允许涉及人类判断，从而导致更可靠的评估，为实践所接受，并在适应个性化指导的预训练语言模型方面提高知识。将采用从不同学生和特定策略收集的数据集来减轻所提出模型的潜在偏差。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Timely assessment of students' learning is crucial for addressing their needs. However, current assessment methods such as multiple-choice or calculation tests may not truly reveal students’ internal thinking processes, because students may guess answers or follow the step-by-step procedures of examples without conceptual understanding. Existing research shows that asking students to write down justifications for their answers, and plan and reflection on their learning can promote them to develop deeper conceptual understanding and apply cognitive and metacognitive strategies in their learning. Nonetheless, this approach is not widely adopted due to the time-consuming nature of assessing free-text responses. This project aims to develop a rubrics-based automatic assessment tool to identify individual students' misconceptions and learning deficiencies from their free-text responses, allowing instructors and students to instantly adjust their teaching and learning, and eventually enabling personalized instructions tailored to individual learners’ needs, which particularly benefits students at HBCUs. The project will also provide the base for delivering research experience for both undergraduates and teachers and outreach various audiences, including public school students, increasing the literacy of the public on artificial intelligence.This research proposes innovative strategies to fine-tune and calibrate the pre-trained language model, through self-supervised learning and few-shot learning, for automatic classification of texts based on user-specified rubrics, with a particular focus on learners' cognitive and metacognitive processes. The proposed model integrates three novel attributes for improving similarity comparison between texts and rubric keywords: (1) across-attention between compared texts for increasing the sensitivity of comparison; (2) joint embedding of words, phrases, and sentences for improving the accuracy of comparison; (3) incorporation of thematic relevance for enhancing the breadth of comparison. Given the assessment rubrics, the model can classify texts to reveal students’ thinking or other traits in multiple finer granular perspectives. It is flexible for adding new assessment perspectives and more transparent than the overall assessment, and allows involving human judgment, leading to more reliable assessment acceptable for practice, and advancing knowledge on adapting pre-trained language models for personalized instructions. The dataset collected from diverse students and specific strategies will be adopted to mitigate the potential biases of the proposed model.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.