Using Video Information Retrieval (VIR) for "decoding" video-based content

在过去十年左右的时间里，媒体广播的需求非常高。即使在疫情爆发之前，人们也会花很多时间看自己喜欢的节目。为了满足观众的期望，电视广播公司和视频点播（VoD）平台大幅增加了内容。在Channel4，我们的战略支柱之一就是尽可能地了解客户的行为。观众的期望已经发生了巨大的变化，我们需要把观众的动机放在我们决策的核心（Future4, 2021）。为了更好地了解我们的客户是谁，我们需要关注他们与我们的内容的交互。因此，深刻理解我们的内容是解码观众偏好的先决条件。音乐信息检索（MIR）在过去得到了广泛的研究。音乐的方方面面“不仅以旋律、和声、节奏和音色的形式存在于音乐本身，而且是由听者在文化商定的规则框架内的认知过程来解释的，比如流派、风格和情绪”（Inksip, 2011）。另一方面，关于从基于视频的内容中检索信息的研究非常有限。本研究计划的重点是一个类似的基于内容的信息检索原则，然而，内容将不是音乐，而是视频。整个广播界都可以从这项工作中受益。VoD平台和电视广播公司应该让他们的观众在他们的决策系统中发挥关键作用，而不仅仅是根据人口统计数据接受建议和沟通。如果机器学习技术通过检查每天数百万次观看产生的大量数据和元数据来检索见解，观众与视频内容的互动可以成为最丰富的信息来源。Lee等人（2016）强调了模型准确性的一些潜在改进，以预测电影的成功。采用引入“未探索特征”等视角可以提高模型的准确性（Lee et al, 2016）。对于这个项目，我将重点关注三个主要路线图。首先，我提出了基于微体裁的视频内容细分。除了非常广泛的类型仍然有用这一事实之外，Channel4还需要遵循类似于Netflix的方法，Netflix已经提出了超过76,000个微类型（Janelle K, 2020）。我认为这是每个广播公司的一个非常重要的任务，无论是线性的还是点播的，有效地分割他们的内容，以便为他们的机器学习模型提供有助于他们提高准确性的特征。我建议使用计算机视觉和尖端机器学习来定义描述Channel4视频内容的微类型。其次，通过将机器学习应用于内容的脚本和对话，以及视频本身，可以实现信息检索。我建议使用自然语言处理和计算机视觉，以便提取视频传递给观众的真实感受。对脚本、对话和视频的情感分析可能是非常有用的信息来源，并可能揭示内容的一些额外方面，从而增强机器学习模型的额外功能。第三，我需要识别用户观看视频内容的不同观看模式。通过应用尖端的机器学习算法，我建议对观看内容进行细分，无论这些内容是在上下班途中、下班后还是在午休时间播出的。挑战在于管理和预处理用户查看后留下的所有元数据。为了将所有这些非结构化数据转换为有助于机器学习模型产生有价值见解的维度，应该应用高级编码。

Media broadcasting has been in very high demand during the last decade or so. Even before the pandemic, people used to spend quite a lot of time watching their favourite programs. TV broadcasters and VideoOnDemand (VoD) Platforms have increased their content dramatically, in order to meet viewers' expectations.At Channel4, one of our strategic pillars is to have the best possible understanding of our customers' behaviour. Viewers' expectations have changed dramatically, and we need to put viewers' motivations at the heart of our decision-making (Future4, 2021). To better understand who our customers are, we need to focus on their interactions with our content. As a result, a deep understanding of our content is a prerequisite to decoding viewers' preferences.Music Information Retrieval (MIR) has been researched extensively in the past. Music has facets that "are not just to be found within the music itself, in the form of melody, harmony, tempo, and timbre, but are interpreted by the cognitive processes of the listener within frameworks of culturally agreed rules, such as genre, style, and mood" (Inksip, 2011). On the other hand, there is very limited research around retrieving information from video-based content. This research proposal focuses on a comparable Content-based Information Retrieval principle, however, the content will not be music, but video.The whole broadcasting community can benefit from this work. VoD Platforms and TV Broadcasters should let their viewers play a key role within their decision-making systems, and not only to receive recommendations and communications based on demographics. Viewers' interactions with video-based content can become the richest source of information, if Machine Learning techniques retrieve insights, by checking the enormous amounts of data and metadata that are generated by the millions of viewings on a daily basis. Lee et al (2016) highlighted some potential improvements for a model's accuracy, in order to predict the success of a movie. Taking perspectives such as introducing "unexplored features", can elevate the accuracy of the model (Lee et al, 2016).For this project, I will focus on three main roadmaps. Firstly, I am proposing a segmentation of video-based content on micro-genres. Besides the fact that the very broad genres are still useful, Channel4 needs to follow an approach similar to Netflix, which has come up with over 76,000 micro-genres (Janelle K, 2020). I consider it a very important task of every Broadcaster, either linear or OnDemand, to segment their content efficiently, in order to feed their Machine Learning models with features that will help them improve their accuracy. I am proposing the use of Computer Vision and cutting-edge Machine Learning in order to define micro-genres that describe Channel4's video-based content.Secondly, Information Retrieval can be achieved by applying Machine Learning to the scripts and the dialogues of the content, and the video itself. I am proposing the use of Natural Language Processing and Computer Vision, in order to extract the real feelings that a video is passing to its viewers. Sentiment analysis of scripts, dialogues, and videos can be a very useful source of information, and might reveal some extra aspects for the content that will boost a Machine Learning model with extra power.Thirdly, I need to identify the different viewing modes that users watch for video-based content. By applying cutting-edge Machine Learning algorithms, I am suggesting a segmentation of the viewings whether they are broadcasted during a commute, or after work, or maybe during a lunch break. The challenge is to manage and pre-process all the metadata that users leave after their viewings. Advanced coding should be applied in order to transform all this amount of unstructured data, into dimensions that will help a Machine Learning model generate valuable insights.