Towards Seamless Interaction and Navigation in Virtual Worlds using Multiple 3D Sensors

使用多个 3D 传感器实现虚拟世界中的无缝交互和导航

• 批准号: RGPIN-2019-05092
• 负责人: Poovvancheri, Jiju
• 金额: $ 1.68万
• 依托单位: Saint Mary's University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=758749
• 关键词: Towards; Seamless; Interaction; Navigation; Virtual

The key trends report from the Canada Media Fund (CMF) identifies Transmedia (Augmented and Virtual Reality (VR)) as "poised for wide consumer release". While "VR is still 5 to 10 years from mainstream adoption, content businesses must prepare now if they hope to compete" [excerpts from the CMF report]. The proposed research comprises two relevant problems pertaining to transmedia applications, 3D content (virtual worlds) creation and human interaction with virtual objects for a fully immersive VR experience. Virtual worlds, i.e., polygonal meshes of cities at scale, are increasingly used in VR based applications, e.g., Google Earth VR. In 2018, an area of 50 sq. kms of Helsinki, Finland, was digitized by Umbra technologies targeting at AR, VR gaming int real city settings [www.forbes.com]. We position our research to support the ongoing efforts of the graphics and gaming industry to utilize digitized cities in transmedia applications. The current practices in large scale city digitization include acquiring raw measurements of the physical world using LiDAR sensors and abstracting coarse models from the LiDAR data. However, such coarse building models lack geometric details and hence realism, and do not represent a good choice for 3D navigation, virtual reality and video game creation. So, we investigate into a fully automatic computational framework that creates highly detailed and semantically rich urban models at large scale. The work is a follow up to my research in LiDAR based urban modeling which began in 2015. A core enabler of interaction in VR spaces is real time full body tracking, e.g., a fully-immersive VR game brings human interaction to virtual objects in real time and such interactions call for robust and highly accurate full body tracking systems which are seamlessly deployable. We target highly robust and low-latency full-body motion capture in the presence of interfering geometry (e.g., joystick) using a multi camera system. A multi-camera 3D data acquisition system will be designed and set up, which would open up further avenues of research in futuristic applications such as virtual cloth fitting in online stores, and bio-mechanical studies in sports and fitness. We plan to collect annotated human motion data and make it publicly available for the geometric deep learning community. The result of this research will be valuable to industry practitioners as well as the academic community in developing better practices and tools for virtual world creation and AR/VR applications. Helping the Canadian VR industry to become a key player in the global VR market will result in a positive impact on the local economy, with capital inflow and the associated requirement for high tech workers in the booming VR/AR industry. The proposed LiDAR based reconstruction framework is expected to play a key role in digitizing existing cities around the globe which may be used as 3D content for VR based games or tours and 3D maps for navigation.

加拿大媒体基金会(CMF)发布的这份关键趋势报告指出，Transmedia(增强和虚拟现实(VR))即将在消费者中广泛发布。虽然“VR距离主流应用还有5到10年的时间，但内容企业如果希望参与竞争，现在就必须做好准备”[摘自CMF的报告]。拟议的研究包括两个与跨媒体应用有关的相关问题，即3D内容(虚拟世界)创建以及人与虚拟对象的交互，以获得完全身临其境的VR体验。虚拟世界，即城市的大规模多边形网格，越来越多地用于基于VR的应用，例如Google Earth VR。2018年，面积50平方米。芬兰赫尔辛基的KMS被Umbra技术数字化，目标是现实城市环境中的AR、VR游戏[www.forbes.com]。我们将我们的研究定位为支持图形和游戏行业正在进行的在跨媒体应用中利用数字化城市的努力。目前大规模城市数字化的实践包括使用激光雷达传感器获取物理世界的原始测量结果，以及从激光雷达数据中提取粗略模型。然而，这种粗糙的建筑模型缺乏几何细节，因此缺乏真实感，并不代表3D导航、虚拟现实和视频游戏创作的良好选择。因此，我们研究了一个全自动的计算框架，它可以大规模地创建高度详细和语义丰富的城市模型。这项工作是我从2015年开始进行的基于激光雷达的城市建模研究的后续工作。VR空间中交互的核心使能是实时全身跟踪，例如，完全沉浸式VR游戏将人与虚拟对象实时交互，这种交互需要可无缝部署的健壮且高精度的全身跟踪系统。我们的目标是在存在干扰几何体(例如操纵杆)的情况下使用多摄像头系统捕获高度健壮和低延迟的全身运动。将设计和建立一个多摄像头3D数据采集系统，这将为未来应用的研究开辟进一步的途径，如网上商店的虚拟布料试穿，以及运动和健身中的生物机械研究。我们计划收集带注释的人体运动数据，并将其公开供几何深度学习社区使用。这项研究的结果将对行业从业者以及学术界开发虚拟世界创建和AR/VR应用的更好实践和工具具有价值。帮助加拿大VR行业成为全球VR市场的关键参与者，将对当地经济产生积极影响，因为资本流入以及蓬勃发展的VR/AR行业对高科技工人的相关需求。拟议的基于LiDAR的重建框架预计将在全球现有城市数字化方面发挥关键作用，这些城市可能被用作基于VR的游戏或旅游的3D内容，以及用于导航的3D地图。