Towards Seamless Interaction and Navigation in Virtual Worlds using Multiple 3D Sensors

加拿大媒体基金（CMF）的主要趋势报告将Transmedia（增强和虚拟现实（VR））确定为“准备广泛的消费者发布”。虽然“VR距离主流应用还有5到10年的时间，但内容企业如果希望竞争，现在就必须做好准备”[CMF报告摘录]。拟议的研究包括两个相关的问题，涉及跨媒体应用程序，3D内容（虚拟世界）的创建和人类与虚拟对象的交互，一个完全沉浸式的VR体验。 虚拟世界，即，大规模城市的多边形网格越来越多地用于基于VR的应用，例如，谷歌地球VR. 2018年，面积50平方米。芬兰赫尔辛基200公里的区域由Umbra技术数字化，目标是在真实的城市环境中进行AR、VR游戏[www.forbes.com]。我们定位我们的研究，以支持图形和游戏行业的持续努力，利用数字化城市的跨媒体应用。大规模城市数字化的当前实践包括使用LiDAR传感器获取物理世界的原始测量值以及从LiDAR数据中提取粗略模型。然而，这种粗糙的建筑物模型缺乏几何细节，因此缺乏真实感，并且不代表3D导航、虚拟现实和视频游戏创建的良好选择。因此，我们研究了一个全自动的计算框架，可以在大规模上创建高度详细和语义丰富的城市模型。这项工作是我在2015年开始的基于LiDAR的城市建模研究的后续工作。 VR空间中的交互的核心使能器是真实的时间全身跟踪，例如，完全沉浸式VR游戏在真实的时间中将人类交互带到虚拟对象，并且这样的交互需要可无缝部署的鲁棒且高度精确的全身跟踪系统。我们的目标是在存在干扰几何结构（例如，操纵杆）使用多相机系统。将设计和建立一个多摄像头3D数据采集系统，这将为未来的应用开辟进一步的研究途径，例如在线商店的虚拟试衣，以及运动和健身中的生物力学研究。我们计划收集带注释的人体运动数据，并将其公开提供给几何深度学习社区。 这项研究的结果将对行业从业者和学术界在开发虚拟世界创建和AR/VR应用程序的更好实践和工具方面具有价值。帮助加拿大VR行业成为全球VR市场的关键参与者将对当地经济产生积极影响，资本流入以及蓬勃发展的VR/AR行业对高科技工人的相关需求。所提出的基于LiDAR的重建框架预计将在数字化地球仪周围的现有城市中发挥关键作用，这些城市可以用作基于VR的游戏或图尔斯的3D内容以及用于导航的3D地图。

The key trends report from the Canada Media Fund (CMF) identifies Transmedia (Augmented and Virtual Reality (VR)) as “poised for wide consumer release”. While “VR is still 5 to 10 years from mainstream adoption, content businesses must prepare now if they hope to compete” [excerpts from the CMF report]. The proposed research comprises two relevant problems pertaining to transmedia applications, 3D content (virtual worlds) creation and human interaction with virtual objects for a fully immersive VR experience. Virtual worlds, i.e., polygonal meshes of cities at scale, are increasingly used in VR based applications, e.g., Google Earth VR. In 2018, an area of 50 sq. kms of Helsinki, Finland, was digitized by Umbra technologies targeting at AR, VR gaming int real city settings [www.forbes.com]. We position our research to support the ongoing efforts of the graphics and gaming industry to utilize digitized cities in transmedia applications. The current practices in large scale city digitization include acquiring raw measurements of the physical world using LiDAR sensors and abstracting coarse models from the LiDAR data. However, such coarse building models lack geometric details and hence realism, and do not represent a good choice for 3D navigation, virtual reality and video game creation. So, we investigate into a fully automatic computational framework that creates highly detailed and semantically rich urban models at large scale. The work is a follow up to my research in LiDAR based urban modeling which began in 2015. A core enabler of interaction in VR spaces is real time full body tracking, e.g., a fully-immersive VR game brings human interaction to virtual objects in real time and such interactions call for robust and highly accurate full body tracking systems which are seamlessly deployable. We target highly robust and low-latency full-body motion capture in the presence of interfering geometry (e.g., joystick) using a multi camera system. A multi-camera 3D data acquisition system will be designed and set up, which would open up further avenues of research in futuristic applications such as virtual cloth fitting in online stores, and bio-mechanical studies in sports and fitness. We plan to collect annotated human motion data and make it publicly available for the geometric deep learning community. The result of this research will be valuable to industry practitioners as well as the academic community in developing better practices and tools for virtual world creation and AR/VR applications. Helping the Canadian VR industry to become a key player in the global VR market will result in a positive impact on the local economy, with capital inflow and the associated requirement for high tech workers in the booming VR/AR industry. The proposed LiDAR based reconstruction framework is expected to play a key role in digitizing existing cities around the globe which may be used as 3D content for VR based games or tours and 3D maps for navigation.