Collaborative Research: Automatic Video Interpretation and Description

数字信息处理已成为现代生活的重要组成部分。如今，它经常以多媒体的形式表现出来，包括带有图像、字幕和音频的视频。鉴于此类多媒体数据的爆炸式增长，对其进行准确的汇总和组织，以便在人工智能中进行自动处理，这一点至关重要。一个重要但具有挑战性的问题是视频内容的自动解释和总结，在视频广告、在线视频搜索和浏览、基于个人偏好的电影推荐以及基本上任何电子商务平台中都有大量应用。在这个项目中，研究小组计划开发统计工具，以提高我们处理数字信息的能力，以应对现实世界应用中快速增长的视频内容。主要目标是创建一个学习系统来解读观众所感知的视觉表达的含义，重点是理解视频所传达的语义。本项目旨在开发视频自动解读和描述的方法，通过理解视频所表达的视觉思想，生成视频内容的语义表达。特别是，它将在多标签和分层分类的框架中利用实体的条件依赖结构以及实体与其相关动作之间的条件依赖结构。它将集中在三个领域：1)实体和动作学习，2)长视频和基于内容的分割的语义学习，以及3)自动视频描述生成，每个领域都以新颖的方式开发技术。在每个领域，分类将基于视频帧的两两条件标签依赖关系和时间依赖关系协同执行，以图形和隐马尔可夫模型为特征。将特别努力从多个来源学习并提取与视频场景对应的潜在结构。pi还计划将开发的软件作为开源发布，并通过确保感兴趣的研究人员能够为开发的软件的代码库做出贡献，从而围绕该语言建立一个用户社区。这将使该项目得到更广泛的发展。高级网络基础设施办公室的软件集群对这方面特别感兴趣，该办公室为该奖项提供了共同资金。

Digital information processing has become an essential part of modern life. It is nowadays often expressed in a form of multimedia, involving videos accompanied with images, captions, and audio. Given the explosive growth of such multimedia data, it is extremely critical that it is accurately summarized and organized for automatic processing in artificial intelligence. One important yet challenging problem is automatic interpretation and summarization of video content, having enormous applications in video advertisements, online video searching and browsing, movie recommendation based on personal preference, and essentially any electronic commerce platform. In this project, the research team plans to develop statistical tools to raise our capacity of processing digital information to respond to a rapid growth of video content in real-world applications. The primary objective is to create a learning system to decipher the meaning of visual expressions as perceived by the audience, with a focus on understanding semantic meaning conveyed by a video.This project aims to develop methods of automatic video interpretation and description, which understands visual thoughts expressed by a video and generates semantic expressions of the content of a video. Particularly, it will utilize conditional dependence structures of entities as well as between entities and their pertinent actions, in a framework of multi-label and hierarchical classification. It will focus on three areas: 1) entity and action learning, 2) semantic learning for long videos and content-based segmentation, and 3) automatic video description generation, each of which develops techniques in novel ways. In each area, classification will be performed collaboratively based on pairwise conditional label dependencies and temporal dependencies of video frames, characterized by graphical and hidden Markov models. Special effort will be devoted to learning from multiple sources and extracting latent structures corresponding to scenes of a video. The PIs also plan to release the software developed as open source and build a user community around the language by ensuring that interested researchers are able to contribute to the codebase of the software developed. This will allow a wider growth of the project. This aspect is of special interest to the software cluster in the Office of Advanced Cyberinfrastructure, which has provided co-funding for this award.