Affect-Based Video Retrieval

基于情感的视频检索

• 批准号: 1539012
• 负责人: Qiang Ji
• 金额: $ 45万
• 依托单位: Rensselaer Polytechnic Institute
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1539012&HistoricalAwards=false
• 关键词: Affect; Based; Video; Retrieval

This project develops advanced machine learning and computer vision technologies for affect-based video retrieval to retrieve videos according to their emotional content. Introducing such a personal touch into video retrieval can improve user's interaction experiences with videos by allowing user to retrieve and organize videos based on their specific emotional needs. In addition, the project also has impacts on a wide range of fields including advertisement and education, allowing the video creators such as advertisers, and educators to effectively customize the videos to best serve the users' emotional needs. The project also contributes to education and student training. The project is integrated with education by (a) introducing a course on computer vision for affective computing; (b) involving undergraduate and graduate students in this project, especially those from under-represented groups; and (c) organizing workshops and tutorials in major computer vision and affective computing conferences on topics related to this research for further dissemination of the research ideas and results.This research addresses problems in video affective content analysis. Affect-based video retrieval faces two major challenges. First, there exists a significant semantic gap between the low level video features and the high level affective content of the video. Second, due to the subjective nature of user's perception of emotion, user's emotional responses to videos vary significantly with people. For the first challenge, the PI develops a novel generative deep model to automatically learn an affect-sensitive multi-modal middle level video representation from the raw video data. To further improve the characterization of the video's affective content, the PI augments it with semantic video attributes derived from well-established video production knowledge to produce a hybrid multi-modal middle level video representation. The hybrid multi-modal middle level representation can effectively bridge the gap between the raw video and its affective content. For the second challenge, the PI employs a multi-task deep learning method to tailor the middle level representation to each user's specific affective preferences in order to maximize their experience with videos.

该项目开发了先进的机器学习和计算机视觉技术，用于基于情感的视频检索，以根据情感内容检索视频。 将这样的个人触摸引入到视频检索中可以通过允许用户基于其特定的情感需求来检索和组织视频来改善用户与视频的交互体验。 此外，该项目还对包括广告和教育在内的广泛领域产生了影响，使广告商和教育工作者等视频创作者能够有效地定制视频，以最好地满足用户的情感需求。该项目还有助于教育和学生培训。 该项目通过以下方式与教育相结合：（a）开设一门关于情感计算的计算机视觉课程;（B）让本科生和研究生参与该项目，特别是那些来自代表性不足群体的学生;以及（c）在主要的计算机视觉和情感计算会议上组织与本研究相关主题的研讨会和教程，以进一步传播研究思想和成果。视频情感内容分析基于情感的视频检索面临两个主要挑战。首先，在低层视频特征和高层视频情感内容之间存在显著的语义鸿沟。其次，由于用户对情感感知的主观性，用户对视频的情感反应因人而异。对于第一个挑战，PI开发了一种新的生成式深度模型，以从原始视频数据中自动学习情感敏感的多模态中级视频表示。 为了进一步改善视频的情感内容的表征，PI用从完善的视频制作知识导出的语义视频属性对其进行增强，以产生混合多模态中间级视频表示。混合多模态中间层表示可以有效地弥合原始视频与其情感内容之间的差距。 对于第二个挑战，PI采用多任务深度学习方法来根据每个用户的特定情感偏好定制中间层表示，以最大限度地提高他们的视频体验。