Intelligent scene understanding for collaborative mobile augmented reality

增强现实（AR）是目前最受期待的新兴技术之一，它将虚拟对象与现实世界相结合，为最终用户提供自然的交互式环境。AR已经在许多领域得到了应用，包括数字媒体、娱乐、安全、医疗保健等****目前的AR技术限制了AR的可能性，因为它只对用户周围的场景有初步的了解。与AWE公司合作，我们将开发新的算法，实现对智能AR场景的语义理解，例如理解用户周围的物体。这种程度的理解将有助于AWE提供更强大的AR体验。我们特别感兴趣的是实现协作式AR体验，即多个用户可以同时享受相同的AR体验并与之互动。具体来说，我们将：1)开发一种自动化的方法来语义地理解用户的环境。该方法将使用预训练对象库搜索用户数据（2D图像和3D点云），并计算识别对象的位置和大小。分割数据和定位检测到的对象将利用最先进的深度学习技术来解释对象和环境外观的变化。2. 通过使用从第一个贡献中获得的语义信息，实现基于视觉的最先进的定位（VBL），并用一个共同的坐标系统表示所有用户，以实现协作AR。****总之，所提出的方法将形成一个强大的框架，用于创建智能和协作的移动AR应用程序。拟议的研究将有助于将加拿大定位为多媒体技术的领导者，而由此产生的技术转移到加拿大工业将加强加拿大的全球竞争力，并对加拿大经济和社会产生积极影响。********

Augmented Reality (AR) is currently one of the most anticipated emerging technology, where the combination of virtual objects with the real world provide the end-user with a natural and interactive environment. AR has found its application in many fields, including digital media, entertainment, security, healthcare, etc.****Current AR technology limits the possibilities of AR through only having a rudimentary understanding of the scene surrounding a user. Partnering with AWE Company Ltd., we will develop new algorithms that will enable semantic understanding of the scene for intelligent AR, such as understanding what objects are around the user. This level of understanding will help AWE in delivering more robust AR experiences. We are especially interested in enabling collaborative AR experiences, where multiple users can enjoy and interact with the same AR experience at the same time. Specifically, we will : 1)Develop an automated method to semantically understand the user's environment. The proposed method will search the user's data (2D image and 3D point cloud) using a library of pre-trained objects, and compute locations and sizes of the identified objects. Segmenting the data and localizing detected objects will take advantage of state-of-the-art deep learning techniques to account for changes in object and environment appearance. 2. Achieve state-of-the-art in Visual-Based Localization (VBL) by using the semantic information obtained from the first contribution, and represent all the users with a common coordinate system to achieve collaborative AR. ****Together, the proposed methods will result in a robust framework for creating intelligent and collaborative mobile AR applications. The proposed research will help to position Canada as a leader in multimedia technologies, while the resultant technology transfer to Canadian industry will strengthen Canada's global competitiveness and create positive impacts to Canadian economy and society.********