Unsupervised cataloguing of objects in a 3D scene

研究背景的简要描述，包括潜在影响：很难或有时不可能获得足够大量的标记数据或原始数据来训练3D感知算法。另一方面，视频已经成为应对这一挑战的一种有效解决方案，因为它们作为丰富的信息来源，提供了毫不费力地捕捉完整3D复杂场景和对象的潜力。在这种背景下，本研究的重点是自动提取和索引的对象在3D场景中，利用视频数据的方便性和普遍性的关键任务。该项目旨在构建算法，可以推理高级概念，如“对象性”，以识别场景中的对象，同时还能够重建这些对象并预测它们在场景的3D模型中的精确布局。如果成功的话，我们的系统可以用来从单目视频中生成3D场景的虚拟模型，这可以用来创建逼真的增强现实体验。此外，该系统还具有直接的现实应用-例如，在零售系统中跟踪库存或帮助仓库进行库存管理。主要目标是开发一个系统，可以自动识别物体在三维场景和唯一索引他们.除了唯一地索引对象，我们的系统还应该能够准确地重建对象，并预测它们在场景的3D模型中的精确布局.我们的目标是以无监督的方式训练这个系统，而不明确使用3D数据或注释，因为它们的获取成本很高。相反，我们将从更便宜的来源，如视频捕捉场景或图像的场景从多个视点提取信息.该项目还将探索使用文本作为另一种形式，以提供关于场景或对象的语义属性的弱监督。另一种可能性是研究动态场景，其中运动可以成为识别场景中对象的有用线索。研究方法的新奇：1.识别和编目场景中的对象是一个具有挑战性的问题，因为我们没有一个预定义的对象列表，因此我们将开发算法来训练系统，可以推理“对象”来发现场景中的对象。另一个困难来自编目方面，因为我们还需要验证新发现的对象是否已经与场景中的编目对象相对应。因此，我们的研究也将集中在三维形状感知的实例检索算法，可以有效地用于编目.由于3D注释很难获得，我们将探索如何将其他监督来源（如视频中的文本或动作线索）纳入模型训练。（该项目涉及EPSRC的哪个研究领域）图像和视觉计算（EPSRC链接）任何参与的公司或合作者：这个博士项目的顾问将是来自牛津大学视觉几何小组的AndreaVedaldi，AndrewZisserman，JoaoHenriques和IroLaina

Brief description of the context of the research including potential impact:It is difficult or sometimes impossible to obtain a sufficiently large amounts of labelled or evenraw data to train 3D perception algorithms. Videos, on the other hand, have emerged as apragmatic solution to this challenge as they serve as a rich source of information, offeringthe potential to effortlessly capture scenes and objects in their full 3D complexity. In this context,the focus of this research is the pivotal task of automated extraction and indexing of objectswithin 3D scenes, leveraging the convenience and ubiquity of video data. This project aims atbuilding algorithms that can reason about high-level concepts such as "objectness" to discoverobjects in a scene, while also being able to reconstruct these objects and predict their preciselayout in the 3D model of the scene. If successful, our system can be used to generate virtualmodels of 3D scenes from monocular videos, which could be used to create realistic augmentedreality experiences. Furthermore, this system also has direct real-world applications -- e.g., tokeep track of inventory in retail systems or to help with stock management in warehouses.Aims and Objectives1. The primary objective is to develop a system that can automatically identify objects in a3D scene and uniquely index them.2. In addition to uniquely indexing the objects, our system should also be able to accuratelyreconstruct the objects and predict their precise layout in the 3D model of the scene.3. We aim to train this system in an unsupervised manner and without explicitly using 3Ddata or annotations as they are expensive to obtain. We will instead extract informationfrom cheaper sources such as videos capturing the scene or images of the scene frommultiple viewpoints.4. The project will also explore using text as another modality to provide weak supervisionabout the semantic properties of the scene or objects. Another possibility is to work ondynamic scenes where motion can be a useful cue for identifying objects in the scene.Novelty of the research methodology:1. Identifying and cataloguing objects in a scene is a challenging problem because we donot possess a predefined list of objects, hence we will develop algorithms to train asystem that can reason about "objectness" to discover objects in the scene.2. Another difficulty comes from the cataloguing aspect as we also need to verify if a newlyfound object already corresponds to a catalogued object in the scene(s). Hence, ourresearch will also focus on 3D shape-aware instance retrieval algorithms that can beefficiently used for cataloguing.3. Since 3D annotations are difficult to obtain, we will explore ways to incorporate othersources of supervision such as text or motion cues from videos to train our models.Alignment to EPSRC's strategies and research areas (which EPSRC research area theproject relates to)Image and vision computing (EPSRC link)Any companies or collaborators involved: The advisors for this DPhil project will be AndreaVedaldi, Andrew Zisserman, João Henriques and Iro Laina from the Visual Geometry Group atUniversity of Oxford