Audio-visual object-based dynamic scene representation from monocular video

这项研究将研究单目视听视频到基于对象的空间局部化视听表示的转换。将研究自监督和弱监督深度学习，以将一般场景转换为具有自我标记和局部定位的对象。这将建立在基于深度学习的一般动态场景和具有已知语义标签的对象(例如人)的单目重建方面的最新进展。包括音频和文本字幕在内的多模式信息源将被用于支持语义标注和基于对象的重建的弱监督学习。这项研究的目标是将具有多个交互人员的复杂现实世界场景的无约束视频序列概括为无约束视频序列。研究将研究多模式或附加信息的传输方法，以支持基于对象的场景重建，并评估不同信息源的相对重要性。该方法应该能够实现未知或未建模对象类的似是而非的重建，以及建模对象类的完全重建。将研究野外和BBC档案数据集的学习，以支持对复杂场景的泛化。还将对体育和节目推荐等具体使用案例进行调查，以便在有限的情况下进行评估。该方法将在现场和传统内容上进行评估。

This research will investigate the transformation of monocular audio and visual video into a spatially localised object-based audio-visual representation. Self supervised and weakly supervised deep learning will be investigated for the transformation of general scenes intosemantically labelled and localised objects. This will build on recent advances in deep-learning based monocular reconstruction of general dynamic scenes and objects with known semantic labels, such as people. Multi-modal information sources including audio and text subtitles will be employed to support weakly supervised learning for semantic labelling and object-based reconstruction. The goal of this research is to generalise to unconstrained video sequences of complex real world scenes with multiple interacting people. Research will investigate approaches for the transfer of multi-modal or additional information to support the object-based scene reconstruction and evaluate the relative importance of differentinformation sources. The approach should be able to achieve plausible reconstruction of unknown or unmodelled object classes, together with complete reconstruction for modelled object classes. Learning on in-the-wild and BBC archive datasets will be investigated to support the generalisation to complex scenes. Specific use-cases such as sports and programme recommendation will also be investigated for evaluation in constrained contexts. The approach will be evaluated on both live and legacy content.