SBIR Phase I: VideoPoints: A Companion for Classroom Learning

SBIR第一阶段项目将开发文本、语音和图像分析技术，将一套教育讲座视频转化为互动学习伙伴。课堂视频将以一系列主题连贯的部分呈现，每个部分由文本摘要、视觉摘要和视频片段表示。学生将能够查询一系列讲座视频，答案以文本、图像、文字记录的演讲和视频的组合形式呈现，以及指向其他相关资源的链接。这些技术改进将推动讲座视频管理系统的发展，其功能远远超出商业最先进水平。学生将获得在一学期的课程中立即访问任何信息的能力，而教师几乎没有开销。这份商业计划的重点是一种免费增值模式，旨在广泛采用，对教师来说是免费的，对学生来说是非常低的成本。最终的潜在社会效益是为高等教育学生提供显著更好的学习体验和学习成果。该项目的关键创新是视频片段的摘要和针对一系列讲座视频定制的问答系统。尽管在文本挖掘和图像分析方面已有大量相关研究，但这些都是艰巨的挑战。讲座视频的信息内容跨越多种形式，具体地说是屏幕文本、图像和语音。此外，每种模式都有独特的特点。屏幕文本通常是非结构化的，并且包括光学字符识别错误。课堂授课视频的文字记录包含非正式的课堂互动，并存在语音识别错误。讲座视频中的图像可以代表各种概念，包括说明性示例、图形和图表以及摄像机图像。将开发创新的主题建模方法，以处理与大多数文件相比的输入的独特性质。该项目将采用表征学习方法，将每种情态映射到一个共同的语义空间，以推动学生问题与授课视频内容之间的匹配。由于学生问题的最佳答案可能不完全包含在讲座视频中，因此还将分析包括教科书、留言板、实际测验和考试在内的外部资源来驱动QA模块。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This SBIR Phase I project will develop text, speech and image analysis technologies to transform a set of educational lecture videos into an interactive learning companion. A lecture video will be presented as a series of topically cohesive sections, each represented by a textual summary, a visual summary, and a video segment. Students will be able to query a series of lecture videos with the answers presented as a combination of text, images, speech with transcript, and video, along with links to additional relevant resources. These technological enhancements will drive the development of the lecture video management system with capabilities well beyond commercial state of the art. The students will gain the ability to instantly access any information in a semester long course, with little overhead for the instructor. The business plan focuses on a freemium model designed for wide adoption with no cost to instructors and a very low cost to students. The ultimate potential societal benefit is a significantly better learning experience and learning outcomes for higher education students.The key innovations in this project are summarization of video segments and a Questions & Answers (QA) system customized for a series of lecture videos. These are formidable challenges despite existing substantial related research in text mining and image analysis. The information content of a lecture video spans multiple modalities, specifically, screen text, images, and speech. Further, each modality has unique characteristics. Screen text is typically unstructured and includes Optical Character Recognition errors. Transcripts from classroom lecture videos contain informal classroom interactions and suffer from speech recognition errors. The images in a lecture video can represent a variety of concepts including illustrative examples, graphs and charts, and camera images. Innovative approaches to topic modeling will be developed to handle the unique nature of the input compared to most documents. The project will employ representation learning approaches to map from each modality to a common semantic space that will drive the matching between student questions and the content of lecture videos. Since the best answer to a student question may not be fully contained in a lecture video, external resources including textbooks, message boards, and actual quizzes and exams will also be analyzed to drive the QA module.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.