Video Understanding and Retrieval with Complex Natural Language Queries

该提案的目标是开发集成的语言和视觉解决方案，以实现人类与自治系统之间以及人类与大规模视觉内容之间的日常交互中的自然通信，从而实现具有自然文本查询的智能浏览。语言是高级语义概念与低级视觉感知之间的重要纽带。一个成功的机器人平台需要能够理解视觉世界和人类（语言）指令，并以自然的方式将其理解传达给用户。我的主要工作兴趣领域是为老年人和视障人士提供语言辅助视觉感知的解决方案。这是一个影响很大的研究领域，目前还没有好的解决方案。

The goal of this proposal is to develop integrated language and vision solutions to enable natural communication between humans and autonomous systems in everyday interaction, as well as between humans and large-scale visual content, enabling intelligent browsing with natural textual queries. Language is an important link between high level semantic concepts and more low level visual perception. A successful robotic platform needs to be able to understand both the visual world and the human's (lingual) instructions and communicate its understanding back to the user in a natural way. The main domain of interest of my work are solutions for language assisted visual perception for the elderly and visually impaired. This is a high-impact area of research where no good solutions exist yet. The focus of the proposal will be on three different aspects of language and vision integration, as part of a large language assisted vision solution. The first effort will be on efficient modeling of visual scenes both in term of objects as well as activities. I will continue my current work on object detection, where I plan to incorporate multi-modal information such as appearance, semantic segmentation, contextual information, motion as well as depth into a deep learning framework. I expect this work to significantly push forward visual recognition performance in both static imaginary as well as video data. My second effort will be on linking the semantic content of natural descriptive text with visual content. I plan to explore two very important settings, one for online visual retrieval and one for offline large-scale visual content browsing. In the first setting, the human conveys to the system a descriptive instruction that the system needs to pay attention to in a live video stream, and inform the user in time. This is particularly important for a solution for the visually impaired, where the automatic system needs to serve as active eyes of the user. In a related setting the proposal will tackle, the user gives a complex descriptive query, and the system needs to retrieve the most relevant video clips from a large video collection. This is important to both analyze large amounts of recorded data, learn user-specific interests, as well as to quickly browse web-scale visual content. This is in contrast to existing search engines which can only handle one or two textual tags. The third goal in this proposal is to develop a system that can describe their internal semantic understanding of the visual scene via language. While solutions for text generation exist, none of them tries to match the structure and saliency with which us, humans, describe the world. This is, however, of critical importance, if an automatic system is to communicate with the human in a natural way. Describing the visual understanding of the world back to the user closes the loop of human-robot interaction and would enable a new generation of robotic systems.