Multi-modal content similarity for predicting audience behaviour

最近的多模式自然语言处理(NLP)方法表明，将不同模式表达的信息(如图像及其字幕、单词及其典型发音或其发音方式、音乐作品的音频及其描述，甚至从兴奋的大脑区域提取的信息)相结合，可以改善NLP的分类、相似性和推理任务。这表明，考虑多模式数据提供了一种更好的理解内容的方式，本项目的目标是促进这一领域的理论和应用。BBC新闻、电视剧和纪实节目是一个很好的数据来源，也是该项目的一个很好的案例研究。它们包括元数据(如流派、格式、服务)和多种形式的数据(如音频、视频、文本)。一般的研究方法是使用机器学习算法，例如经典的k-均值和神经网络，分别和/或通过联合目标函数来学习多个模式的矢量表示。对于培训和应用领域，我们建议执行与内容近似的观众行为任务，例如节目受欢迎程度和内容推荐。我们将在不同的社会人口群体中工作，并收集这些群体的观众行为数据。学术导师将是Mehrnoosh Sadrzadeh，该机构是伦敦大学学院的CS系。BBC的主管是克里斯·纽威尔和安德鲁·麦克帕兰。Sadrzadeh通过英国皇家工程院与BBC研发部门的两项工业计划奖学金(2017年1月至2018年1月，2019年9月至2020年)，致力于导致该项目的初步想法。BBC研发部门聘请了两名实习生对该项目的某些方面进行小规模探索(将音频、流派、字幕结合在一个包含145个电视剧节目的数据库中)。初步调查结果令人振奋，提高了BBC经常使用的基于元数据的推荐的精确度和多样性。

Recent multi-modal approaches to Natural Language Processing (NLP) have shown that combining the information expressed by different modalities, such as images with their captions, a word with its typical sound or the way it is pronounced, the audio of a musical piece and its description, even information extracted from excited brain areas, improves NLP tasks of classification, similarity, and inference. This is taken as an indication that considering multi-modal data provides a better way of understanding content and the goal of this project is to advance the theory and applications in this area. The BBC news, drama and factual programmes are an excellent source of data and a great case study for this project they come with metadata (e.g. genre, format, service) and multiple modalities of data (e.g. audio, video, text). The general research methodology would be to learn vector representations for the multiple modes separately and/or via joint objective functions using machine learning algorithms such as the classical k-means and neural networks. For training and as an application domain, we suggest tasks that approximate viewers' behaviour with content, such as programme popularity and content recommendations. We will work within different socio-demographic groups and the viewers behaviour data of these groups. The academic supervisor will be Mehrnoosh Sadrzadeh, the institution is the CS Department of UCL. The BBC supervisors are Chris Newell and Andrew McParland. Sadrzadeh has worked on the preliminary ideas that led to this project via two Royal Academy of Engineering Industrial Scheme Fellowships with the BBC R&D (Jan 2017-2018, Sept 2019-2020). The BBC R&D has hired two interns to work on small scale explorations of some aspects of this project (combining audio, genre, subtitles in a dataset of 145 drama programmes). The preliminary findings have been promising, leading to improvements in precision and diversity of metadata-based recommendations of the kind frequently used by the BBC.