CAREER: Generative Item, Response, and Feedback Models in Assessment and Learning

个性化辅导和对表现或知识掌握的反馈是两种教学策略，已被证明在提高学生学习成果方面是有效的。但是，就有效提供这些战略所需的人力资源而言，执行这些战略，特别是大规模执行这些战略，代价高昂。利用人工智能（AI）在数字学习平台上为学生提供反馈和个性化辅导，有可能减少提供这些服务所需的人力资本，并有效地为越来越多的学习者提供服务。该CAREER项目将利用人工智能机器学习领域的最新创新——生成语言模型（GLMs）来估计学习者的知识水平，并从开放式学习者的回答中识别具体错误。由此产生的系统将能够自动生成个性化的项目和反馈，以支持教师和学习者。这个CAREER项目主要以中学数学教育为基础，由ASSISTments和OpenStax支持数据收集和评估，有可能使许多教师和学习者受益。他的CAREER项目的其他潜在成果包括扩大少数民族学习者接触人工智能现实应用的活动，以及一门关于人工智能教育的新课程。这个CAREER项目包括三个主要的研究线索。首先，项目团队将为开放式数学项目开发一系列开放式项目反应理论和知识追踪框架。关键的技术挑战将是注入学习者的知识状态，引导glm根据每个学习者对不同技能的知识生成个性化的反应预测。这些模型将在辅导活动中为教师仪表板工具和学习者错误检测工具提供动力。其次，项目团队将开发基于glm的自动数学题生成方法，以满足每个学习者的需求和兴趣，并在随机对照试验中对其进行评估。关键的技术挑战将是根据人类对项目上下文的规范以及数学和语言复杂性来控制生成的项目。第三，项目团队将开发一个基于glm的自动反馈生成框架，并探索其在常见错误答案反馈和辅导对话回合生成中的应用。关键的技术挑战将是学习如何利用有效的教师撰写的反馈信息，并将其作为glm的输入示例。该团队还将探索“师从编辑”的方法，以随着时间的推移不断提高生成反馈的质量。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Personalized tutoring and feedback on performance or knowledge mastery, are two instructional strategies that have been shown to be effective at improving student learning outcomes. However, implementing these strategies, especially at scale, is costly in terms of the human resources required to provide them effectively. The use of Artificial Intelligence (AI) to provide students with feedback and personalized tutoring in digital learning platforms has the potential to reduce the human capital required to provide these services and to service growing numbers of learners effectively. This CAREER project will leverage generative language models (GLMs), a recent innovation in AI machine learning, to estimate learner knowledge levels and identify specific errors from open-ended learner responses. The resulting system will then be able to automatically generate personalized items and feedback, to support teachers and learners. Primarily grounded in middle-school math education with data collection and evaluation supported by ASSISTments and OpenStax, this CAREER project has the potential to benefit many teachers and learners. Other potential outcomes of his CAREER project include activities that expand the access of minority learners to real-world applications of AI and a new course on AI for education. This CAREER project includes three major research threads. First, the project team will develop a family of open-ended item response theory and knowledge tracing frameworks for open-ended math items. The key technical challenge will be to inject learner knowledge states to steer GLMs towards generating personalized response predictions according to each learner’s knowledge on different skills. These models will power teacher dashboard tools and learner error detection tools during tutoring activities. Second, the project team will develop GLM-based automated math item generation methods to meet the needs and interests of each learner and evaluate them in a randomized controlled trial. The key technical challenge will be to control the generated items according to human specifications on item context and both mathematical and language complexity. Third, the project team will develop a GLM-based automated feedback generation framework and explore its usage in both common wrong answer feedback and tutoring dialogue turn generation. The key technical challenge will be to learn how to leverage effective teacher-written feedback messages and use them as input examples for GLMs. The team will also explore learning-from-teacher-edit methods to constantly improve the quality of generated feedback over time.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.