Multi-modal content similarity for predicting audience behaviour

最近的自然语言处理（NLP）的多模态方法表明，结合不同模态表达的信息，如图像及其标题，单词及其典型声音或发音方式，音乐作品的音频及其描述，甚至从兴奋的大脑区域提取的信息，都可以提高NLP的分类，相似性和推理任务。这表明考虑多模态数据提供了更好的理解内容的方式，本项目的目标是推进这一领域的理论和应用。BBC的新闻、戏剧和事实类节目是很好的数据来源，也是本项目的一个很好的案例研究，它们有元数据（例如类型、格式、服务）和多种形式的数据（例如音频、视频、文本）。一般的研究方法是使用机器学习算法（如经典的k-means和神经网络）分别和/或通过联合目标函数来学习多个模式的向量表示。对于训练和应用领域，我们建议使用内容近似观众行为的任务，例如节目受欢迎程度和内容推荐。我们将在不同的社会人口群体和这些群体的观众行为数据中工作。学术导师为Mehrnoosh Sadrzadeh，机构为伦敦大学学院计算机科学系。BBC的主管是Chris Newell和Andrew McParland。Sadrzadeh通过两次英国皇家工程学院工业计划奖学金（2017年1月至2018年1月，2019年9月至2020年9月）研究了导致该项目的初步想法。BBC研发部门雇佣了两名实习生，对这个项目的某些方面进行小规模的探索（将145个电视剧节目的音频、体裁和字幕结合起来）。初步的发现是有希望的，这导致了BBC经常使用的基于元数据的推荐的准确性和多样性的提高。

Recent multi-modal approaches to Natural Language Processing (NLP) have shown that combining the information expressed by different modalities, such as images with their captions, a word with its typical sound or the way it is pronounced, the audio of a musical piece and its description, even information extracted from excited brain areas, improves NLP tasks of classification, similarity, and inference. This is taken as an indication that considering multi-modal data provides a better way of understanding content and the goal of this project is to advance the theory and applications in this area. The BBC news, drama and factual programmes are an excellent source of data and a great case study for this project they come with metadata (e.g. genre, format, service) and multiple modalities of data (e.g. audio, video, text). The general research methodology would be to learn vector representations for the multiple modes separately and/or via joint objective functions using machine learning algorithms such as the classical k-means and neural networks. For training and as an application domain, we suggest tasks that approximate viewers' behaviour with content, such as programme popularity and content recommendations. We will work within different socio-demographic groups and the viewers behaviour data of these groups. The academic supervisor will be Mehrnoosh Sadrzadeh, the institution is the CS Department of UCL. The BBC supervisors are Chris Newell and Andrew McParland. Sadrzadeh has worked on the preliminary ideas that led to this project via two Royal Academy of Engineering Industrial Scheme Fellowships with the BBC R&D (Jan 2017-2018, Sept 2019-2020). The BBC R&D has hired two interns to work on small scale explorations of some aspects of this project (combining audio, genre, subtitles in a dataset of 145 drama programmes). The preliminary findings have been promising, leading to improvements in precision and diversity of metadata-based recommendations of the kind frequently used by the BBC.