Research and development of a cloud-based context-aware API for semantic scene understanding

增强现实（AR）已成为提高数字媒体产品体验质量的一种有前途的工具。然而，可用于AR的技术工具缺乏快速有效地设计交互式数字体验所需的丰富理解。与AWE Company Ltd合作，本研究项目将尝试开发一个上下文感知的应用程序编程接口（API），该接口可以通过智能地将来自用户设备的摄像机馈送数据分发到三个独立的子模块上以进行语义理解来提供对用户周围环境的语义理解：建筑立面分割，平面图创建和对象定位。通过理解用户的上下文（即，理解上述语义的要求），所提出的API可以减少在云上运行基于神经网络的系统所需的计算和数据开销。拟议的项目将采用分布式深度神经网络（DDNN）设计原则，在客户端和云之间仔细分配计算任务，同时在现有技术工具（如ARCore/ARKit）之上构建语义理解堆栈，以便AWE的软件开发人员可以轻松地将拟议的解决方案集成到他们的产品开发管道中。 拟议的API将为AWE计划在未来两年内为其许多客户推出的数字孪生项目的快速分发提供一个强大的框架。拟议的研究将有助于将加拿大定位为多媒体技术的领导者，而由此产生的对加拿大工业的技术转让将加强加拿大的全球竞争力，并对加拿大经济和社会产生积极影响。

Augmented Reality (AR) has become a promising tool for enhancing the quality of experience in digital media products. However, the technical tools available for AR lacks in providing the rich understanding required to design interactive digital experiences quickly and efficiently. In partnership with AWE Company Ltd, this research project will attempt at developing a context-aware Application Programming Interface (API) that can provide a semantic understanding of a user's surrounding by intelligently distributing the camera feed data from a user's device on to three separate sub-modules for semantic understanding: building facade segmentation, floor plan creation, and object localization. Through understanding the user's context (i.e. the requirement of the understanding of the aforementioned semantics), the proposed API can reduce the computational and data overhead required to run the neural network-based systems on the cloud. The proposed project will employ Distributed Deep Neural Network (DDNN) design principles to carefully distribute the computational task between the clients and the cloud, while building the semantic understanding stack on top of available technology tools such as ARCore/ARKit, so that AWE's software developers can easily integrate the proposed solution into their product development pipeline. The proposed API will result in a robust framework for quick distribution of digital twins projects that AWE plans to roll out for many of its clients within the next two years. The proposed research will help to position Canada as a leader in multimedia technologies, while the resultant technology transfer to Canadian industry will strengthen Canada's global competitiveness and create positive impacts to Canadian economy and society.