Collaborative Research: Advancing STEM Online Learning by Augmenting Accessibility with Explanatory Captions and AI

视频是一种流行的在线学习媒介，其中的字幕对于提高学生有效学习的可及性至关重要。本研究确定了两种类型的视频字幕：典型的封闭式字幕和解释性字幕。封闭字幕是视频中口语部分的文本表示。解释性字幕的创建是为了让学生深入了解视频的视觉、文本和音频内容。现有的技术主要集中在自动生成或提高封闭字幕的质量。对于STEM学习，解释性说明可能在学习中发挥新的作用。该项目将致力于设计有效的问答机制和有效的交互设计，使学生和教师能够以协作的方式为STEM视频生成解释性字幕。拟议的技术将增加服务不足人群的可及性和学习体验，包括由4800万美国人组成的聋人和听力障碍（DHH）社区，同时也提高非英语母语人士的理解能力，甚至包括那些没有听力障碍的人。评估地点包括加劳德特大学（Gallaudet University）和伊利诺伊大学厄巴纳-香槟分校（University of Illinois at Urbana-Champaign）。加劳德特大学是世界上唯一一所专门致力于培养DHH学习者的文理大学，伊利诺伊大学厄巴纳-香槟分校是美国公立机构中国际学生人数最多的大学，并为残疾学生提供包容性学习环境。这项跨学科的研究从计算机科学和学习科学以及以下领域的可访问性实践中汲取并做出了贡献。第一步是发现新的知识，了解无障碍视频（带有解释性和封闭式字幕）如何扩大服务不足人群对STEM学习的参与。这将为开发解释性字幕如何有助于学习和有效机制的理论提供基础，基于众包的人类贡献和机器学习算法，为不同学习阶段（例如，准备，跟踪，故障排除和反思）的STEM视频创建这些解释性字幕。然后，研究人员将利用这一理论创造一种新型聊天机器人，使不同背景的学生能够分享知识。理论框架——ICAP（互动、建设性、主动和被动）和探究社区将指导对解释性字幕和聊天机器人如何有助于学习的评估。最后，团队将获得人工智能代理（如聊天机器人）增强可访问性如何影响学生和教师实践的经验理解。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Videos are a popular medium for online learning, in which captions are essential for increasing accessibility to students for effective learning. This research identifies two types of video captions: typical closed captions and explanatory captions. Closed captions are a text representation of the spoken part of a video. Explanatory captions are created to give students insights into the visual, textual, and audio content of a video. Existing technologies have focused on automatically generating or improving the quality of closed captions. For STEM learning, explanatory captions have the potential to play a new role in learning. This project will work to devise effective Q/A mechanisms and effective interaction designs that enable students and instructors to generate explanatory captions for STEM videos in a collaborative manner. The proposed technologies will augment accessibility and learning experiences for under-served populations, including the Deaf and Hard-of-Hearing (DHH) community, made up of 48 million Americans, while also improving comprehension for non-native English speakers, even those without hearing impairments. Evaluation sites include both Gallaudet University, the world’s only liberal arts university dedicated exclusively to educating DHH learners, and the University of Illinois at Urbana-Champaign, which has the largest international student population amongst U.S. public institutions and supports students with disabilities in inclusive learning environments. This interdisciplinary research draws from and contributes to both computer science and learning science, and accessibility practices in the following areas. The first step is discovering new knowledge about how accessibility-enabled videos (with explanatory and closed captions) broaden the participation of under-served populations in STEM learning. This will provide the foundation for developing a theory of how explanatory captions can contribute to learning and effective mechanisms, based on crowdsourced human contributions and machine learning algorithms, to create these explanatory captions for STEM videos at different learning stages (e.g., preparing, tracking, trouble-shooting, and reflecting). The investigators will then use the theory to create a novel chatbot that enables knowledge sharing for students with diverse backgrounds. Theoretical frameworks--ICAP (interactive, constructive, active, and passive) and Community of Inquiry will guide the evaluation of how explanatory captions and chatbots can contribute to learning. Finally, the team will acquire empirical understanding of how augmented accessibility with AI agents (e.g., chatbots) impacts students' and instructors’ practices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.