Research and development of a cloud-based context-aware API for semantic scene understanding

点击翻译按钮获取中文摘要

Augmented Reality (AR) has become a promising tool for enhancing the quality of experience in digital media products. However, the technical tools available for AR lacks in providing the rich understanding required to design interactive digital experiences quickly and efficiently. In partnership with AWE Company Ltd, this research project will attempt at developing a context-aware Application Programming Interface (API) that can provide a semantic understanding of a user's surrounding by intelligently distributing the camera feed data from a user's device on to three separate sub-modules for semantic understanding: building facade segmentation, floor plan creation, and object localization. Through understanding the user's context (i.e. the requirement of the understanding of the aforementioned semantics), the proposed API can reduce the computational and data overhead required to run the neural network-based systems on the cloud. The proposed project will employ Distributed Deep Neural Network (DDNN) design principles to carefully distribute the computational task between the clients and the cloud, while building the semantic understanding stack on top of available technology tools such as ARCore/ARKit, so that AWE's software developers can easily integrate the proposed solution into their product development pipeline. The proposed API will result in a robust framework for quick distribution of digital twins projects that AWE plans to roll out for many of its clients within the next two years. The proposed research will help to position Canada as a leader in multimedia technologies, while the resultant technology transfer to Canadian industry will strengthen Canada's global competitiveness and create positive impacts to Canadian economy and society.