Towards Seamless Interaction and Navigation in Virtual Worlds using Multiple 3D Sensors

加拿大媒体基金（CMF）的主要趋势报告将跨媒体（增强和虚拟现实（VR））确定为“广泛的消费者发布”。虽然“VR距离主流应用还有5到10年的时间，但内容企业如果想要竞争，现在就必须做好准备”（摘自CMF报告）。建议的研究包括两个相关问题，涉及跨媒体应用，3D内容（虚拟世界）创建和人类与虚拟对象的交互，以实现完全沉浸式的VR体验。******虚拟世界，即大规模的城市多边形网格，越来越多地用于基于VR的应用程序，例如谷歌Earth VR。2018年，面积为50平方米。Umbra技术将芬兰赫尔辛基方圆公里的区域数字化，目标是在真实的城市环境中进行AR、VR游戏[www.forbes.com]。我们的研究旨在支持图形和游戏行业在跨媒体应用中利用数字化城市的持续努力。目前大规模城市数字化的实践包括使用激光雷达传感器获取物理世界的原始测量数据，并从激光雷达数据中提取粗模型。然而，这种粗糙的建筑模型缺乏几何细节和真实感，并不是3D导航、虚拟现实和电子游戏创作的好选择。因此，我们研究了一个全自动的计算框架，它可以在大范围内创建高度详细和语义丰富的城市模型。这项工作是我在2015年开始的基于激光雷达的城市建模研究的后续工作。******虚拟现实空间中交互的一个核心促成因素是实时全身跟踪，例如，一个完全沉浸式的虚拟现实游戏将人类互动实时带到虚拟对象中，这种互动需要强大和高度精确的全身跟踪系统，这些系统可以无缝部署。我们的目标是在存在干扰几何（例如，操纵杆）的情况下，使用多摄像头系统实现高度鲁棒和低延迟的全身运动捕捉。将设计并建立一个多摄像头3D数据采集系统，这将为未来的应用开辟进一步的研究途径，如在线商店的虚拟试衣，以及运动和健身中的生物力学研究。我们计划收集带注释的人体运动数据，并将其公开提供给几何深度学习社区。******这项研究的结果将对行业从业者和学术界为虚拟世界创建和AR/VR应用开发更好的实践和工具有价值。帮助加拿大VR产业成为全球VR市场的关键参与者，将对当地经济产生积极影响，并带来资本流入和蓬勃发展的VR/AR行业对高科技工人的相关需求。拟议的基于激光雷达的重建框架预计将在全球现有城市数字化方面发挥关键作用，这些城市可能被用作基于VR的游戏或旅游的3D内容，以及用于导航的3D地图。

The key trends report from the Canada Media Fund (CMF) identifies Transmedia (Augmented and Virtual Reality (VR)) as “poised for wide consumer release”. While “VR is still 5 to 10 years from mainstream adoption, content businesses must prepare now if they hope to compete” [excerpts from the CMF report]. The proposed research comprises two relevant problems pertaining to transmedia applications, 3D content (virtual worlds) creation and human interaction with virtual objects for a fully immersive VR experience. ******Virtual worlds, i.e., polygonal meshes of cities at scale, are increasingly used in VR based applications, e.g., Google Earth VR. In 2018, an area of 50 sq. kms of Helsinki, Finland, was digitized by Umbra technologies targeting at AR, VR gaming int real city settings [www.forbes.com]. We position our research to support the ongoing efforts of the graphics and gaming industry to utilize digitized cities in transmedia applications. The current practices in large scale city digitization include acquiring raw measurements of the physical world using LiDAR sensors and abstracting coarse models from the LiDAR data. However, such coarse building models lack geometric details and hence realism, and do not represent a good choice for 3D navigation, virtual reality and video game creation. So, we investigate into a fully automatic computational framework that creates highly detailed and semantically rich urban models at large scale. The work is a follow up to my research in LiDAR based urban modeling which began in 2015.******A core enabler of interaction in VR spaces is real time full body tracking, e.g., a fully-immersive VR game brings human interaction to virtual objects in real time and such interactions call for robust and highly accurate full body tracking systems which are seamlessly deployable. We target highly robust and low-latency full-body motion capture in the presence of interfering geometry (e.g., joystick) using a multi camera system. A multi-camera 3D data acquisition system will be designed and set up, which would open up further avenues of research in futuristic applications such as virtual cloth fitting in online stores, and bio-mechanical studies in sports and fitness. We plan to collect annotated human motion data and make it publicly available for the geometric deep learning community.******The result of this research will be valuable to industry practitioners as well as the academic community in developing better practices and tools for virtual world creation and AR/VR applications. Helping the Canadian VR industry to become a key player in the global VR market will result in a positive impact on the local economy, with capital inflow and the associated requirement for high tech workers in the booming VR/AR industry. The proposed LiDAR based reconstruction framework is expected to play a key role in digitizing existing cities around the globe which may be used as 3D content for VR based games or tours and 3D maps for navigation.