Can Crowds Customize Instructional Materials with Minimal Expert Guidance?: Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor
Can Crowds Customize Instructional Materials with Minimal Expert Guidance?: Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor
复制标题
群体可以在最少的专家指导下定制教学材料吗？：探索教师引导的众包以改进基于人工智能的导师的提示
DOI：
10.1145/3449193
复制
发表时间：
2021
影响因子：
--
通讯作者：
Aleven, Vincent
中科院分区：
文献类型：
--
作者：
Yang, Kexin Bella;Nagashima, Tomohiro;Yao, Junhui;Williams, Joseph Jay;Holstein, Kenneth;Aleven, Vincent
当基于人工智能的教育技术与教师的目标、偏好和教学实践相一致时，它们可能在课堂上最受欢迎。然而，教师们很少有时间自己做这样的定制。如何利用群众来帮助时间紧张的教师？传统上，众包管道专注于内容生成。如何设计管道是一个开放的问题，以便大众能够成功地完成修订/定制任务。在本文中，我们探索了一个教师引导的众包管道的初始版本，旨在改进基于人工智能的辅导系统的自适应数学提示，使其符合教师的偏好，同时需要最少的专家指导。在涉及144名数学教师和481名众包工作者的两个实验中，我们发现这种专家引导的复习管道比两种比较条件节省了专家的时间，并且产生了更好的众包复习提示（就教师满意度而言）。然而，修改后的提示并没有改善AI导师中现有的提示，这些提示写得很仔细，但仍有改进和定制的空间。进一步分析表明，众包工作者面临的主要挑战可能在于理解教师简短的书面评论，并以有效的编辑形式实施这些评论，而不引入新的问题。我们还发现，教师更喜欢自己的修改而不是其他来源的提示，并且对提示表现出不同的偏好。总体而言，结果证实，有明确的需要定制提示，以个别教师的喜好。他们还强调需要更复杂的框架，这样人们就可以对教师对提示的要求有具体的了解。该研究首次探索了如何在修订和定制教学材料时，以最少的专家指导来支持人群。
AI-based educational technologies may be most welcome in classrooms when they align with teachers' goals, preferences, and instructional practices. Teachers, however, have scarce time to make such customizations themselves. How might the crowd be leveraged to help time-strapped teachers? Crowdsourcing pipelines have traditionally focused on content generation. It is an open question how a pipeline might be designed so the crowd can succeed in a revision/customization task. In this paper, we explore an initial version of a teacher-guided crowdsourcing pipeline designed to improve the adaptive math hints of an AI-based tutoring system so they fit teachers' preferences, while requiring minimal expert guidance. In two experiments involving 144 math teachers and 481 crowdworkers, we found that such an expert-guided revision pipeline could save experts' time and produce better crowd-revised hints (in terms of teacher satisfaction) than two comparison conditions. The revised hints however, did not improve on the existing hints in the AI tutor, which were carefully-written but still have room for improvement and customization. Further analysis revealed that the main challenge for crowdworkers may lie in understanding teachers' brief written comments and implementing them in the form of effective edits, without introducing new problems. We also found that teachers preferred their own revisions over other sources of hints, and exhibited varying preferences for hints. Overall, the results confirm that there is a clear need for customizing hints to individual teachers' preferences. They also highlight the need for more elaborate scaffolds so the crowd can have specific knowledge of the requirements that teachers have for hints. The study represents a first exploration in the literature of how to support crowds with minimal expert guidance in revising and customizing instructional materials.