RI-Medium: From Actors To Actions: Analysis And Alignment Of Images, Video And Text

RI-Medium：从演员到行动：图像、视频和文本的分析和对齐

• 批准号: 0803538
• 负责人: Jianbo Shi
• 金额: --
• 依托单位: University of Pennsylvania
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2008
• 资助国家: 美国
• 起止时间: 2008-09-01 至 2011-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0803538&HistoricalAwards=false
• 关键词: RI; Medium; Actors; Actions; Analysis

Video clips and corresponding narrations together provide much richer information than either in isolation, yet most current recognition systems process visual and textual information separately. The PIs focus on the task of learning how to recognize corresponding actions in videos and textual narrative accurately and robustly. In particular, they focus on semantic descriptions of human actions. This research will have broad impact on applications including video retrieval in digital libraries, human behavior modeling, and video surveillance.The PIs' research will tightly couple methods in computer vision, natural-language processing, and machine learning through robust, automatically learned correspondences. With a collection of loosely aligned video-text annotation pairs (such as movies or TV shows with their associated screenplays), the task is to learn how to associate action descriptions in text with actions, objects and actors in videos. This correspondence is essential for semantic grounding of text using visual action appearance. The fundamental challenge is bridging the semantic gap of images and of text: images depict geometrical relationships and properties of image regions, while natural language encodes abstract semantic relationships in grammatical structures. Bridging this semantic gap in the context of action understanding is the focus of our research effort.The eventual goal is to be able to recognize actions in videos and create text description for actions in videos. While this goal challenges both computer vision and natural language processing, it also opens up an exciting new and very fruitful collaboration between the two research areas where the task of recognition is achieved by simultaneous learning and inference in both domains.Information on this project, including papers, results, database and open source codes, will be available at http://www.seas.upenn.edu/~jshi/#research

视频片段和相应的旁白一起提供的信息比单独提供的信息要丰富得多，然而大多数当前的识别系统分别处理视觉和文本信息。PI专注于学习如何准确而稳健地识别视频和文本叙事中的相应动作。特别是，它们侧重于对人类行为的语义描述。这项研究将对数字图书馆中的视频检索、人类行为建模和视频监控等应用产生广泛的影响。PI的研究将通过稳健的、自动学习的对应关系将计算机视觉、自然语言处理和机器学习的方法紧密结合在一起。有了一组松散排列的视频-文本注释对(例如电影或电视剧及其关联的剧本)，任务是学习如何将文本中的动作描述与视频中的动作、对象和演员相关联。这种对应关系对于使用视觉动作外观的文本的语义基础是必不可少的。最根本的挑战是弥合图像和文本的语义鸿沟：图像描述图像区域的几何关系和属性，而自然语言则以语法结构编码抽象的语义关系。在动作理解的背景下弥合这种语义鸿沟是我们研究的重点，最终目标是能够识别视频中的动作并为视频中的动作创建文本描述。虽然这一目标挑战了计算机视觉和自然语言处理，但它也开启了这两个研究领域之间令人兴奋的新的和非常富有成效的合作，其中识别任务是通过两个领域的同时学习和推理来完成的。关于这个项目的信息，包括论文、结果、数据库和开放源代码，将在http://www.seas.upenn.edu/~jshi/#research上获得