Using Video Information Retrieval (VIR) for "decoding" video-based content

在过去十年左右的时间里，媒体广播的需求一直很高。即使在大流行之前，人们过去也会花相当多的时间观看他们最喜欢的节目。电视广播公司和视频点播(VOD)平台大幅增加了内容，以满足观众的期望。在Channel4，我们的战略支柱之一是尽可能最好地了解客户的行为。观众的期望已经发生了巨大的变化，我们需要把观众的动机放在我们决策的核心(《未来4》，2021年)。为了更好地了解我们的客户是谁，我们需要关注他们与我们内容的互动。因此，深入理解我们的内容是解码观众偏好的前提。音乐信息检索(MIR)在过去得到了广泛的研究。音乐的各个方面“不仅以旋律、和声、节拍和音色的形式存在于音乐本身中，而且通过听者在文化认同的规则框架内的认知过程来解释，如流派、风格和情绪”(Inksip，2011)。另一方面，关于从基于视频的内容中检索信息的研究非常有限。这项研究方案侧重于基于内容的可比性信息检索原则，但内容将不是音乐，而是视频。整个广播界都可以从这项工作中受益。视频点播平台和电视广播公司应该让他们的观众在他们的决策系统中发挥关键作用，而不仅仅是接受基于人口统计的推荐和交流。观众与基于视频的内容的互动可以成为最丰富的信息来源，如果机器学习技术通过检查每天数百万次观看产生的海量数据和元数据来检索洞察力。Lee等人(2016)强调了模型准确性的一些潜在改进，以便预测电影的成功。从引入“未探索的功能”等角度，可以提高模型的准确性(Lee等人，2016)。对于这个项目，我将集中在三个主要路线图上。首先，我建议对基于视频的内容进行微流派的细分。除了非常广泛的流派仍然有用这一事实之外，Channel4还需要遵循类似于Netflix的方法，Netflix已经推出了超过7.6万种微流派(Janelle K，2020)。我认为这是每个广播公司的一项非常重要的任务，无论是线性的还是按需的，有效地分割他们的内容，以便为他们的机器学习模型提供有助于他们提高准确性的功能。我建议使用计算机视觉和尖端机器学习来定义描述S频道基于视频的内容的微观体裁。第二，通过将机器学习应用于内容的脚本和对话以及视频本身，可以实现信息检索。我建议使用自然语言处理和计算机视觉，以提取视频传递给观众的真实感受。脚本、对话和视频的情感分析可以是非常有用的信息来源，并可能揭示内容的一些额外方面，这将为机器学习模型提供额外的功能。第三，我需要确定用户观看基于视频的内容的不同观看模式。通过应用尖端的机器学习算法，我建议对观看的节目进行分割，无论它们是在通勤时、下班后还是午休期间播出的。挑战是管理和预处理用户在观看后留下的所有元数据。应该应用高级编码，以便将所有这些数量的非结构化数据转换为有助于机器学习模型生成有价值的见解的维度。

Media broadcasting has been in very high demand during the last decade or so. Even before the pandemic, people used to spend quite a lot of time watching their favourite programs. TV broadcasters and VideoOnDemand (VoD) Platforms have increased their content dramatically, in order to meet viewers' expectations.At Channel4, one of our strategic pillars is to have the best possible understanding of our customers' behaviour. Viewers' expectations have changed dramatically, and we need to put viewers' motivations at the heart of our decision-making (Future4, 2021). To better understand who our customers are, we need to focus on their interactions with our content. As a result, a deep understanding of our content is a prerequisite to decoding viewers' preferences.Music Information Retrieval (MIR) has been researched extensively in the past. Music has facets that "are not just to be found within the music itself, in the form of melody, harmony, tempo, and timbre, but are interpreted by the cognitive processes of the listener within frameworks of culturally agreed rules, such as genre, style, and mood" (Inksip, 2011). On the other hand, there is very limited research around retrieving information from video-based content. This research proposal focuses on a comparable Content-based Information Retrieval principle, however, the content will not be music, but video.The whole broadcasting community can benefit from this work. VoD Platforms and TV Broadcasters should let their viewers play a key role within their decision-making systems, and not only to receive recommendations and communications based on demographics. Viewers' interactions with video-based content can become the richest source of information, if Machine Learning techniques retrieve insights, by checking the enormous amounts of data and metadata that are generated by the millions of viewings on a daily basis. Lee et al (2016) highlighted some potential improvements for a model's accuracy, in order to predict the success of a movie. Taking perspectives such as introducing "unexplored features", can elevate the accuracy of the model (Lee et al, 2016).For this project, I will focus on three main roadmaps. Firstly, I am proposing a segmentation of video-based content on micro-genres. Besides the fact that the very broad genres are still useful, Channel4 needs to follow an approach similar to Netflix, which has come up with over 76,000 micro-genres (Janelle K, 2020). I consider it a very important task of every Broadcaster, either linear or OnDemand, to segment their content efficiently, in order to feed their Machine Learning models with features that will help them improve their accuracy. I am proposing the use of Computer Vision and cutting-edge Machine Learning in order to define micro-genres that describe Channel4's video-based content.Secondly, Information Retrieval can be achieved by applying Machine Learning to the scripts and the dialogues of the content, and the video itself. I am proposing the use of Natural Language Processing and Computer Vision, in order to extract the real feelings that a video is passing to its viewers. Sentiment analysis of scripts, dialogues, and videos can be a very useful source of information, and might reveal some extra aspects for the content that will boost a Machine Learning model with extra power.Thirdly, I need to identify the different viewing modes that users watch for video-based content. By applying cutting-edge Machine Learning algorithms, I am suggesting a segmentation of the viewings whether they are broadcasted during a commute, or after work, or maybe during a lunch break. The challenge is to manage and pre-process all the metadata that users leave after their viewings. Advanced coding should be applied in order to transform all this amount of unstructured data, into dimensions that will help a Machine Learning model generate valuable insights.