Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning
Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning
复制标题
DOI：
--
复制
发表时间：
2023-08
期刊：
影响因子：
--
通讯作者：
Hunter McNichols;Wanyong Feng;Jaewook Lee;Alexander Scarlatos;Digory Smith;Simon Woodhead;Andrew S. Lan
中科院分区：
文献类型：
--
作者：
Hunter McNichols;Wanyong Feng;Jaewook Lee;Alexander Scarlatos;Digory Smith;Simon Woodhead;Andrew S. Lan
多项选择题（MCQ）在几乎所有的教育层次中都是普遍存在的，因为它们易于管理，评分，并且是一种可靠的评估形式。MCQ的一个重要方面是干扰物，即，针对学生的具体误解或知识不足而设计的不正确选项。到目前为止，制作高质量干扰项的任务在很大程度上仍然是教师和学习内容设计师的劳动密集型过程，其可扩展性有限。在这项工作中，我们探讨了任务的自动分心和相应的反馈信息生成的数学MCQs使用大型语言模型。我们建立了这两个任务的制定，并提出了一个简单的，在上下文学习为基础的解决方案。此外，我们提出了基于生成式AI的指标来评估反馈消息的质量。我们使用真实世界的MCQ数据集对这些任务进行了广泛的实验。我们的研究结果表明，自动干扰器和反馈生成还有很大的改进空间;根据这些研究结果，我们概述了未来工作的几个方向。
Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and are a reliable form of assessment. An important aspect of MCQs is the distractors, i.e., incorrect options that are designed to target specific misconceptions or insufficient knowledge among students. To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers, which has limited scalability. In this work, we explore the task of automated distractor and corresponding feedback message generation in math MCQs using large language models. We establish a formulation of these two tasks and propose a simple, in-context learning-based solution. Moreover, we propose generative AI-based metrics for evaluating the quality of the feedback messages. We conduct extensive experiments on these tasks using a real-world MCQ dataset. Our findings suggest that there is a lot of room for improvement in automated distractor and feedback generation; based on these findings, we outline several directions for future work.