EAGER: Joint Modeling and Querying of Social Media and Video Data

随着用户生成的数据量的增加，有效地搜索这些数据以获得有用信息变得更具挑战性。人们一直在研究如何搜索文本社交媒体帖子，如推文或视频。 然而，使用单独的工具搜索这些来源是无效的，因为它们之间的信息链接丢失了;例如，人们不能自动将社交网络帖子与视频上看到的活动相匹配。举个例子，考虑一组在骚乱期间生成的推文和视频（可能发布在Twitter或其他媒体上）。一名警探希望联合搜索这些数据，以找到与汽车火灾等特定事件相关的材料。一些推文（没有包含视频）可能会对汽车火灾发表评论，而另一条推文的视频片段则显示了火灾期间或之后的汽车。将视频与相关的社交媒体帖子链接，这是该项目的重点，可以大大减少搜索有用信息的工作。该项目的成功完成有可能提高在社交媒体上搜索的人的生产力，如警察侦探，灾害管理当局的记者。该项目还将加强和扩大正在进行的本科生研究和高中外展活动的调查人员。该项目的目标是集中在基础研究任务，将允许社会网络和视频数据的联合建模。然后，给定一组帖子，系统将通过定义社交媒体和视频数据的公共特征空间来找到相关的视频片段，反之亦然。这个概念验证项目将在Twitter平台上分享的帖子和视频中进行评估。 鉴于深度学习和大数据管理技术的最新进展，现在是解决这个问题的正确时机。一个关键的风险是，推文中的语义可能不足以将其映射到视频片段;为此，上下文（例如，来自密切相关用户的推文）可能需要被利用。

As the amount of user generated data increases, it becomes more challenging to effectively search this data for useful information. There has been work on how to search text social media posts, such as Tweets, or videos. However, searching on these sources using separate tools is ineffective because the information links between them are lost; for instance, one cannot automatically match social network posts with activities seen on a video. As an example, consider a set of tweets and videos (which may be posted on Twitter or other media) generated during a riot. A police detective would like to jointly search this data to find material related to a specific incident like a car fire. Some tweets (with no contained video) may comment on the car fire, while a video segment from another tweet shows the car during or after the fire. Linking the videos with the relevant social media posts, which is the focus of this project, can greatly reduce the effort in searching for useful information. The successful completion of this project has the potential to improve the productivity of people who search in social media, such as police detectives, journalists of disaster management authorities. This project will also strengthen and extend the ongoing undergraduate research and high school outreach activities of the investigators.The objective of this project is to focus on the fundamental research tasks that would allow for joint modeling of social network and video data. Then, given a set of posts, the system would find relevant video segments and vice versa, by defining a common feature space for social media and video data. This proof-of-concept project will be evaluated on posts and videos shared on the Twitter platform. This is the right time to tackle this problem given the recent advances in deep learning and big data management technologies. A key risk is that the semantics in a tweet may not be enough to map it to a video segment; for that, the context (e.g., tweets from closely related users) of the tweet may need to be leveraged.