Collaborative Research: Advancing STEM Online Learning by Augmenting Accessibility with Explanatory Captions and AI

视频是一种流行的在线学习媒体，其中的字幕对于增加学生有效学习的可及性至关重要。这项研究确定了两种类型的视频字幕：典型的闭路字幕和解释性字幕。隐藏字幕是视频语音部分的文本表示。创建说明性字幕是为了让学生深入了解视频的视觉、文本和音频内容。现有技术侧重于自动生成或改进闭路字幕的质量。对于STEM学习来说，说明性字幕有可能在学习中发挥新的作用。该项目将致力于设计有效的问答机制和有效的互动设计，使学生和教师能够以合作的方式为STEM视频生成说明性字幕。拟议的技术将增强服务不足人群的可及性和学习体验，包括由4800万美国人组成的聋人和听力障碍(DHH)社区，同时还将提高非英语母语者的理解力，即使是那些没有听力障碍的人。评估网站包括加拉德特大学(Gallaudet University)和伊利诺伊大学香槟分校(University of Illinois at Urbana-Champaign)，前者是世界上唯一一所专门培养DHH学习者的文科大学，后者是美国公共机构中国际学生人数最多的，在包容性学习环境中为残疾学生提供支持。这项跨学科的研究借鉴并促进了计算机科学和学习科学，以及以下领域的可访问性实践。第一步是发现新的知识，了解支持无障碍的视频(带有说明性和隐藏字幕)如何扩大服务不足的人群参与STEM学习的程度。这将为开发说明性字幕如何有助于学习的理论和基于众包人类贡献和机器学习算法的有效机制提供基础，以便在不同的学习阶段(例如，准备、跟踪、故障排除和反思)为STEM视频创建这些说明性字幕。然后，研究人员将利用这一理论创建一种新型聊天机器人，使不同背景的学生能够分享知识。理论框架--ICAP(交互式、建设性、主动性和被动性)和调查社区将指导对说明性字幕和聊天机器人如何有助于学习的评估。最后，该团队将获得关于人工智能代理(例如聊天机器人)的增强可访问性如何影响学生和教师实践的经验理解。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Videos are a popular medium for online learning, in which captions are essential for increasing accessibility to students for effective learning. This research identifies two types of video captions: typical closed captions and explanatory captions. Closed captions are a text representation of the spoken part of a video. Explanatory captions are created to give students insights into the visual, textual, and audio content of a video. Existing technologies have focused on automatically generating or improving the quality of closed captions. For STEM learning, explanatory captions have the potential to play a new role in learning. This project will work to devise effective Q/A mechanisms and effective interaction designs that enable students and instructors to generate explanatory captions for STEM videos in a collaborative manner. The proposed technologies will augment accessibility and learning experiences for under-served populations, including the Deaf and Hard-of-Hearing (DHH) community, made up of 48 million Americans, while also improving comprehension for non-native English speakers, even those without hearing impairments. Evaluation sites include both Gallaudet University, the world’s only liberal arts university dedicated exclusively to educating DHH learners, and the University of Illinois at Urbana-Champaign, which has the largest international student population amongst U.S. public institutions and supports students with disabilities in inclusive learning environments. This interdisciplinary research draws from and contributes to both computer science and learning science, and accessibility practices in the following areas. The first step is discovering new knowledge about how accessibility-enabled videos (with explanatory and closed captions) broaden the participation of under-served populations in STEM learning. This will provide the foundation for developing a theory of how explanatory captions can contribute to learning and effective mechanisms, based on crowdsourced human contributions and machine learning algorithms, to create these explanatory captions for STEM videos at different learning stages (e.g., preparing, tracking, trouble-shooting, and reflecting). The investigators will then use the theory to create a novel chatbot that enables knowledge sharing for students with diverse backgrounds. Theoretical frameworks--ICAP (interactive, constructive, active, and passive) and Community of Inquiry will guide the evaluation of how explanatory captions and chatbots can contribute to learning. Finally, the team will acquire empirical understanding of how augmented accessibility with AI agents (e.g., chatbots) impacts students' and instructors’ practices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.