Collaborative Research: Advancing STEM Online Learning by Augmenting Accessibility with Explanatory Captions and AI

协作研究：通过解释性字幕和人工智能增强可访问性，推进 STEM 在线学习

• 批准号: 2119531
• 负责人: Meng Jiang
• 金额: $ 19.01万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-15 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2119531&HistoricalAwards=false
• 关键词: Collaborative; Research; Advancing; STEM; Online

Videos are a popular medium for online learning, in which captions are essential for increasing accessibility to students for effective learning. This research identifies two types of video captions: typical closed captions and explanatory captions. Closed captions are a text representation of the spoken part of a video. Explanatory captions are created to give students insights into the visual, textual, and audio content of a video. Existing technologies have focused on automatically generating or improving the quality of closed captions. For STEM learning, explanatory captions have the potential to play a new role in learning. This project will work to devise effective Q/A mechanisms and effective interaction designs that enable students and instructors to generate explanatory captions for STEM videos in a collaborative manner. The proposed technologies will augment accessibility and learning experiences for under-served populations, including the Deaf and Hard-of-Hearing (DHH) community, made up of 48 million Americans, while also improving comprehension for non-native English speakers, even those without hearing impairments. Evaluation sites include both Gallaudet University, the world’s only liberal arts university dedicated exclusively to educating DHH learners, and the University of Illinois at Urbana-Champaign, which has the largest international student population amongst U.S. public institutions and supports students with disabilities in inclusive learning environments. This interdisciplinary research draws from and contributes to both computer science and learning science, and accessibility practices in the following areas. The first step is discovering new knowledge about how accessibility-enabled videos (with explanatory and closed captions) broaden the participation of under-served populations in STEM learning. This will provide the foundation for developing a theory of how explanatory captions can contribute to learning and effective mechanisms, based on crowdsourced human contributions and machine learning algorithms, to create these explanatory captions for STEM videos at different learning stages (e.g., preparing, tracking, trouble-shooting, and reflecting). The investigators will then use the theory to create a novel chatbot that enables knowledge sharing for students with diverse backgrounds. Theoretical frameworks--ICAP (interactive, constructive, active, and passive) and Community of Inquiry will guide the evaluation of how explanatory captions and chatbots can contribute to learning. Finally, the team will acquire empirical understanding of how augmented accessibility with AI agents (e.g., chatbots) impacts students' and instructors’ practices.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

视频是一种流行的在线学习媒体，其中字幕对于提高学生有效学习的可访问性至关重要。本研究将视频字幕分为两类：典型隐藏式字幕和解释性字幕。隐藏式字幕是视频语音部分的文本表示。创建解说字幕是为了让学生深入了解视频的视觉，文本和音频内容。现有技术集中于自动生成或提高隐藏字幕的质量。对于STEM学习，解释性字幕有可能在学习中发挥新的作用。该项目将致力于设计有效的Q/A机制和有效的互动设计，使学生和教师能够以合作的方式为STEM视频生成解释性字幕。拟议的技术将增加服务不足人群的可访问性和学习体验，包括由4800万美国人组成的聋人和听力困难（DHH）社区，同时还提高了非英语母语者的理解能力，即使是那些没有听力障碍的人。评估地点包括Gallaudet大学，这是世界上唯一一所专门致力于教育DHH学习者的文科大学，以及伊利诺伊大学厄巴纳-香槟分校，该大学在美国公立机构中拥有最多的国际学生人口，并在包容性学习环境中支持残疾学生。这种跨学科的研究借鉴并有助于计算机科学和学习科学，以及以下领域的无障碍实践。第一步是发现关于无障碍视频（带有解释性和隐藏字幕）如何扩大服务不足人群参与STEM学习的新知识。这将为开发解释性字幕如何有助于学习的理论和有效机制提供基础，基于众包的人类贡献和机器学习算法，为不同学习阶段的STEM视频创建这些解释性字幕（例如，准备、跟踪、故障排除和反思）。然后，研究人员将利用这一理论创建一个新颖的聊天机器人，为不同背景的学生提供知识共享。 理论框架--ICAP（交互式、建设性、主动和被动）和探究社区将指导对解释性字幕和聊天机器人如何有助于学习的评估。最后，该团队将获得经验性的理解如何增强与AI代理的可访问性（例如，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

IfQA：反事实预设下的开放域问答数据集

• DOI: 10.18653/v1/2023.emnlp-main.515
• 发表时间: 2023
• 期刊: EMNLP
• 作者: Yu, Wenhao;Jiang, Meng;Clark, Peter;Sabharwal, Ashish
• 通讯作者: Sabharwal, Ashish

Completing Taxonomies with Relation-Aware Mutual Attentions

• 发表时间: 2023
• 作者: Qingkai Zeng;Zhihan Zhang;Jinfeng Lin;Meng Jiang

Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts

知识图专家混合的常识推理内容生成多样化

• DOI: 10.18653/v1/2022.findings-acl.149
• 发表时间: 2022
• 期刊: Findings of the Association for Computational Linguistics: ACL 2022
• 作者: Yu, Wenhao;Zhu, Chenguang;Qin, Lianhui;Zhang, Zhihan;Zhao, Tong;Jiang, Meng
• 通讯作者: Jiang, Meng

Generate rather than Retrieve: Large Language Models are Strong Context Generators

• DOI: 10.48550/arxiv.2209.10063
• 发表时间: 2022-09
• 期刊: ArXiv
• 作者: W. Yu;Dan Iter;Shuohang Wang;Yichong Xu;Mingxuan Ju;Soumya Sanyal;Chenguang Zhu;Michael Zeng;Meng Jiang

Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions

探索开放域问答系统对最少编辑问题的对比度一致性

• 发表时间: 2023
• 期刊: Transactions of the Association for Computational Linguistics
• 影响因子: 10.9
• 作者: Zhang, Zhihan;Yu, Wenhao;Ning, Zheng;Ju, Mingxuan;Jiang, Meng
• 通讯作者: Jiang, Meng