OAC Core: Small: Open-Source Robust 4D Reconstruction Framework for Real-Time Dynamic Human Capture

即将部署的5G技术使以极低的延迟与增强现实(AR)和混合现实(MR)所需的海量数据进行通信成为可能，这使得诸如3D远程沉浸式(或3D FaceTime)通信、使用捕获的4D人类内容的移动AR应用、逼真和个性化化身的AI助手、可以服务于人类或与人类合作的人类感知机器人、将物理治疗师与远离治疗设施的受伤患者连接的远程康复等实现了变革性的应用。所有这些AR和MR应用的例子都需要开发实时4D(空间和时间)捕获和重建涉及人体、面部、身体附属品(如衣服)及其周围环境。尽管已经有了实时4D重建的研究，但目前还没有一个开源的、健壮的重建系统来模拟动态场景的拓扑变化，并对运动曲面进行准确和健壮的跟踪。该项目旨在弥合这些差距，开发一个开源和强大的4D重建框架，使更广泛的科学界和行业联盟的研究人员和开发人员受益，包括5G医疗标准、AR和MR游戏引擎、栩栩如生的人工智能助手、人类感知机器人、远程康复等。该项目还通过夏令营为研究生、本科生和K-12学生提供课程开发和教育活动。该项目的研究围绕动态场景中拓扑变化的优雅建模和运动表面的健壮跟踪，最终目标是开发一个开源的健壮的4D重建框架，用于实时捕获动态人体场景。为了解决拓扑变化的挑战，将从根本上重新设计体积融合框架及其数据结构，将非流形体积网格引入截断符号距离场(TSDF)和嵌入变形图(EDG)表示中，允许体积单元复制自身和打破边。这种新颖的拓扑变化感知框架将允许重建的网格几何图形动态更新其连通性，以及在4D捕获过程中更新节点之间的连通性的灵活变形图。为了解决4D动态人体捕捉中的曲面跟踪问题，提出了一种结合参数人体模型和体积TSDF优点的参数化动画体积模型(PAVM)。TSDF体积网格建立在参数化的人体表面之上，因此可以用来表示人体的附加组件(如衣服)。我们的PAVM的规则体积结构使得它很容易集成到深度神经网络中，以实现稳健的曲面跟踪，以及提供语义建模功能。4D重建框架的技术可行性将通过开发移动3D Facetime试验台进行验证，该试验台将允许偏远地区的人们以自然AR方式相互交互。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The upcoming deployment of 5G technology makes it feasible to communicate with extremely low latency the vast amounts of data needed for Augmented Reality (AR) and Mixed Reality (MR), which enables transformative applications such as 3D tele-immersive (or 3D facetime) communication, mobile AR apps using captured 4D human contents, AI assistants of lifelike and personalized avatars, human-aware robots that can serve or work with humans, tele-rehabilitation to connect physical therapists with wounded patients far from treatment facilities, etc. All these examples of AR and MR applications require the development of real-time 4D (space and time) capture and reconstruction of dynamic scenes involving human bodies, faces, body add-ons (like clothes), and their surrounding environments. Despite prior research on real-time 4D reconstruction, there is still no open-source and robust reconstruction system that can model topological changes of the dynamic scenes and track the moving surfaces with accuracy and robustness. This project is designed to bridge such gaps, to develop an open-source and robust 4D reconstruction framework that can benefit researchers and developers in broader scientific communities as well as industry alliances, including 5G medical standards, AR and MR game engines, lifelike AI assistants, human-aware robotics, tele-rehabilitation, etc. This project also provides curriculum development and educational activities for graduate, undergraduate, and K-12 students through summer camps. The research proposed for this project centers around the elegant modeling of topological changes in dynamic scenes and the robust tracking of moving surfaces, towards the ultimate goal of developing an open-source robust 4D reconstruction framework for real-time capture of dynamic human scenes. To solve the challenges of topological changes, the volumetric fusion framework and its data structures will be fundamentally redesigned, by introducing Non-manifold Volumetric Grids into both Truncated Signed Distance Field (TSDF) and Embedded Deformation Graph (EDG) representations, allowing both the volumetric cells to replicate themselves and the edges to be broken. Such a novel topology-change-aware framework will allow the reconstructed mesh geometry to update its connectivity on-the-fly, along with a flexible deformation graph updating its connectivity between nodes throughout the 4D capture process. To solve the robust surface tracking problem in 4D dynamic human capture, a Parameterized Animatable Volumetric Model (PAVM) is proposed to combine the benefits of both the parametric human body model and the volumetric TSDF. The TSDF volumetric grids are built on top of the parameterized human body surfaces, so that they can be used to represent the add-ons (e.g. clothes) to the human body. The regular volumetric structure of our PAVM makes it easy to integrate into deep neural networks for robust surface tracking, as well as providing semantic modeling capability. The technical feasibility of the 4D reconstruction framework will be validated by the development of a Mobile 3D Facetime testbed, which will allow people in remote places to interact with each other in a natural AR fashion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.