RI: Medium: Collaborative Research: Learning to Summarize User-Generated Video

今天，消费者、科学家、国防分析师和其他人捕获的视频远远超过了观看的数量。随着视频数据的爆炸性增长，迫切需要开发自动视频摘要算法。视频摘要以长视频作为输入，产生短视频作为输出，同时尽可能地保留其信息内容。因此，摘要技术具有巨大的潜力，可以使大型视频集合更高效地浏览、搜索、传播和促进通信。这种效率的提高将在许多重要的应用领域发挥至关重要的作用。例如，有了可靠的摘要系统，灵长类动物学家收集她的动物受试者的长视频后，可以快速浏览他们一周的活动，然后决定在哪里最密切地检查数据。一名年轻的学生在YouTube上搜索黄石国家公园，可以一目了然地看到公园的内容，比今天简单的缩略图描述的要好得多。情报人员可以快速筛选大量的航空视频，减少分析监控数据以识别可疑活动所需的资源。该项目开发了用于视频摘要的新的机器学习和计算机视觉算法。作为几乎所有现有方法的基石的非监督方法，由于依赖于手工制作的启发式方法，已经变得越来越有限。这个项目不是把视频摘要作为一个有监督的学习问题，而是研究了一个明显不同的任务公式。研究团队正在研究四个关键的新想法：用于学习选择最佳视频帧子集进行摘要的强大概率模型，用于利用丰富的多个相关视频的半监督学习模型和共同摘要算法，用于利用Web上的照片来改进摘要的算法，以及以符合人类理解的方式评估摘要的评估协议。拟议研究的更广泛影响包括视频摘要的实用工具、广泛吸引几个社区的科学进步、公开传播研究成果、跨学科培训的研究生以及吸引年轻学生参与STEM教育和职业道路的外联活动。

Today there is far more video being captured - by consumers, scientists, defense analysts, and others - than can ever be watched. With this explosion of video data comes a pressing need to develop automatic video summarization algorithms. Video summarization takes a long video as input and produces a short video as output, while preserving its information content as much as possible. As such, summarization techniques have great potential to make large video collections substantially more efficient to browse, search, disseminate, and facilitate communication. Such increased efficiency will play a vital role in many important application areas. For example, with reliable summarization systems, a primatologist gathering long videos of her animal subjects could quickly browse a week's worth of their activity before deciding where to inspect the data most closely. A young student searching YouTube to learn about Yellowstone National Park could see at a glance what content exists, much better than today's simple thumbnail images can depict. An intelligence agent could rapidly sift through reams of aerial video, reducing the resources required to analyze surveillance data to identify suspicious activities.This project develops new machine learning and computer vision algorithms for video summarization. Unsupervised methods, which are the cornerstone of nearly all existing approaches, have become increasingly limiting due to their reliance on hand-crafted heuristics. By instead posing video summarization as a supervised learning problem, this project investigates a markedly different formulation of the task. The research team is investigating four key new ideas: powerful probabilistic models for learning to select the optimal subset of video frames for summarization, semi-supervised learning models and co-summarization algorithms for leveraging the abundance of multiple related videos, algorithms for exploiting photos on the Web to improve summarization, and evaluation protocols that assess summaries in a way that aligns well with human comprehension. The broader impact of the proposed research includes practical tools for video summarization, scientific advances that appeal broadly to several communities, publicly disseminated research results, inter-disciplinarily trained graduate students, and outreach activities to engage young students in STEM education and career paths.