Affect-Based Video Retrieval

该项目开发先进的机器学习和计算机视觉技术，用于基于情感的视频检索，根据视频的情感内容检索视频。在视频检索中引入这种个性化的元素，可以让用户根据自己特定的情感需求来检索和组织视频，从而改善用户与视频的交互体验。此外，该项目还对广告、教育等广泛领域产生影响，使广告主、教育工作者等视频创作者能够有效地定制视频，以最好地服务于用户的情感需求。该项目还有助于教育和学生培训。该项目通过以下方式与教育相结合：(a)引入一门关于情感计算的计算机视觉课程；(b)让本科生和研究生参与这个项目，特别是那些来自代表性不足群体的学生；(c)在与本研究相关的主要计算机视觉和情感计算会议上组织研讨会和教程，以进一步传播研究思想和成果。本研究针对视频情感内容分析中的问题。基于影响的视频检索面临两大挑战。首先，低层次视频特征与高层次视频情感内容之间存在显著的语义差距。第二，由于用户情感感知的主观性，用户对视频的情感反应因人而异。对于第一个挑战，PI开发了一种新的生成深度模型，从原始视频数据中自动学习影响敏感的多模态中级视频表示。为了进一步改善视频情感内容的表征，PI通过从成熟的视频制作知识中获得的语义视频属性对其进行增强，以产生混合多模态中级视频表示。混合多模态中间层表示可以有效地弥合原始视频与其情感内容之间的鸿沟。对于第二个挑战，PI采用多任务深度学习方法，根据每个用户的特定情感偏好定制中级表示，以最大限度地提高他们的视频体验。

This project develops advanced machine learning and computer vision technologies for affect-based video retrieval to retrieve videos according to their emotional content. Introducing such a personal touch into video retrieval can improve user's interaction experiences with videos by allowing user to retrieve and organize videos based on their specific emotional needs. In addition, the project also has impacts on a wide range of fields including advertisement and education, allowing the video creators such as advertisers, and educators to effectively customize the videos to best serve the users' emotional needs. The project also contributes to education and student training. The project is integrated with education by (a) introducing a course on computer vision for affective computing; (b) involving undergraduate and graduate students in this project, especially those from under-represented groups; and (c) organizing workshops and tutorials in major computer vision and affective computing conferences on topics related to this research for further dissemination of the research ideas and results.This research addresses problems in video affective content analysis. Affect-based video retrieval faces two major challenges. First, there exists a significant semantic gap between the low level video features and the high level affective content of the video. Second, due to the subjective nature of user's perception of emotion, user's emotional responses to videos vary significantly with people. For the first challenge, the PI develops a novel generative deep model to automatically learn an affect-sensitive multi-modal middle level video representation from the raw video data. To further improve the characterization of the video's affective content, the PI augments it with semantic video attributes derived from well-established video production knowledge to produce a hybrid multi-modal middle level video representation. The hybrid multi-modal middle level representation can effectively bridge the gap between the raw video and its affective content. For the second challenge, the PI employs a multi-task deep learning method to tailor the middle level representation to each user's specific affective preferences in order to maximize their experience with videos.