Collaborative Research: Automatic Video Interpretation and Description

数字信息处理已成为现代生活的重要组成部分。如今，它通常以多媒体的形式表达，包括伴随着图像，字幕和音频的视频。鉴于这种多媒体数据的爆炸性增长，它是非常关键的，它是准确地总结和组织的人工智能自动处理。视频内容的自动解释和摘要是一个重要而具有挑战性的问题，在视频广告、在线视频搜索和浏览、基于个人偏好的电影推荐以及基本上任何电子商务平台中具有巨大的应用。在这个项目中，研究小组计划开发统计工具，以提高我们处理数字信息的能力，以应对现实世界应用中视频内容的快速增长。本项目的主要目的是建立一个学习系统，以理解视频所传达的语义为重点，破译观众所感知的视觉表达的含义。本项目旨在开发自动视频解释和描述的方法，该方法理解视频所表达的视觉思想，并生成视频内容的语义表达。特别是，它将利用实体的条件依赖结构，以及实体和它们的相关动作之间，在多标签和层次分类的框架。它将专注于三个领域：1）实体和动作学习，2）长视频的语义学习和基于内容的分割，以及3）自动视频描述生成，每一个都以新颖的方式开发技术。在每个区域中，将基于成对条件标签依赖性和视频帧的时间依赖性协作执行分类，其特征在于图形和隐马尔可夫模型。特别的努力将致力于从多个源学习和提取对应于视频场景的潜在结构。PI还计划发布作为开源开发的软件，并通过确保感兴趣的研究人员能够为所开发软件的代码库做出贡献，围绕该语言建立一个用户社区。这将使该项目得到更广泛的发展。高级网络基础设施办公室的软件集群对此特别感兴趣，该办公室为该奖项提供了共同资助。

Digital information processing has become an essential part of modern life. It is nowadays often expressed in a form of multimedia, involving videos accompanied with images, captions, and audio. Given the explosive growth of such multimedia data, it is extremely critical that it is accurately summarized and organized for automatic processing in artificial intelligence. One important yet challenging problem is automatic interpretation and summarization of video content, having enormous applications in video advertisements, online video searching and browsing, movie recommendation based on personal preference, and essentially any electronic commerce platform. In this project, the research team plans to develop statistical tools to raise our capacity of processing digital information to respond to a rapid growth of video content in real-world applications. The primary objective is to create a learning system to decipher the meaning of visual expressions as perceived by the audience, with a focus on understanding semantic meaning conveyed by a video.This project aims to develop methods of automatic video interpretation and description, which understands visual thoughts expressed by a video and generates semantic expressions of the content of a video. Particularly, it will utilize conditional dependence structures of entities as well as between entities and their pertinent actions, in a framework of multi-label and hierarchical classification. It will focus on three areas: 1) entity and action learning, 2) semantic learning for long videos and content-based segmentation, and 3) automatic video description generation, each of which develops techniques in novel ways. In each area, classification will be performed collaboratively based on pairwise conditional label dependencies and temporal dependencies of video frames, characterized by graphical and hidden Markov models. Special effort will be devoted to learning from multiple sources and extracting latent structures corresponding to scenes of a video. The PIs also plan to release the software developed as open source and build a user community around the language by ensuring that interested researchers are able to contribute to the codebase of the software developed. This will allow a wider growth of the project. This aspect is of special interest to the software cluster in the Office of Advanced Cyberinfrastructure, which has provided co-funding for this award.