OAC Core: Small: Open-Source Robust 4D Reconstruction Framework for Real-Time Dynamic Human Capture

OAC Core：小型：用于实时动态人体捕捉的开源稳健 4D 重建框架

• 批准号: 2007661
• 负责人: Xiaohu Guo
• 金额: $ 50万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2007661&HistoricalAwards=false
• 关键词: OAC; Core; Small; Open; Source

The upcoming deployment of 5G technology makes it feasible to communicate with extremely low latency the vast amounts of data needed for Augmented Reality (AR) and Mixed Reality (MR), which enables transformative applications such as 3D tele-immersive (or 3D facetime) communication, mobile AR apps using captured 4D human contents, AI assistants of lifelike and personalized avatars, human-aware robots that can serve or work with humans, tele-rehabilitation to connect physical therapists with wounded patients far from treatment facilities, etc. All these examples of AR and MR applications require the development of real-time 4D (space and time) capture and reconstruction of dynamic scenes involving human bodies, faces, body add-ons (like clothes), and their surrounding environments. Despite prior research on real-time 4D reconstruction, there is still no open-source and robust reconstruction system that can model topological changes of the dynamic scenes and track the moving surfaces with accuracy and robustness. This project is designed to bridge such gaps, to develop an open-source and robust 4D reconstruction framework that can benefit researchers and developers in broader scientific communities as well as industry alliances, including 5G medical standards, AR and MR game engines, lifelike AI assistants, human-aware robotics, tele-rehabilitation, etc. This project also provides curriculum development and educational activities for graduate, undergraduate, and K-12 students through summer camps. The research proposed for this project centers around the elegant modeling of topological changes in dynamic scenes and the robust tracking of moving surfaces, towards the ultimate goal of developing an open-source robust 4D reconstruction framework for real-time capture of dynamic human scenes. To solve the challenges of topological changes, the volumetric fusion framework and its data structures will be fundamentally redesigned, by introducing Non-manifold Volumetric Grids into both Truncated Signed Distance Field (TSDF) and Embedded Deformation Graph (EDG) representations, allowing both the volumetric cells to replicate themselves and the edges to be broken. Such a novel topology-change-aware framework will allow the reconstructed mesh geometry to update its connectivity on-the-fly, along with a flexible deformation graph updating its connectivity between nodes throughout the 4D capture process. To solve the robust surface tracking problem in 4D dynamic human capture, a Parameterized Animatable Volumetric Model (PAVM) is proposed to combine the benefits of both the parametric human body model and the volumetric TSDF. The TSDF volumetric grids are built on top of the parameterized human body surfaces, so that they can be used to represent the add-ons (e.g. clothes) to the human body. The regular volumetric structure of our PAVM makes it easy to integrate into deep neural networks for robust surface tracking, as well as providing semantic modeling capability. The technical feasibility of the 4D reconstruction framework will be validated by the development of a Mobile 3D Facetime testbed, which will allow people in remote places to interact with each other in a natural AR fashion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

即将部署的5G技术使得以极低的延迟传输增强现实（AR）和混合现实（MR）所需的大量数据成为可能，这使得3D远程沉浸式等变革性应用成为可能。（或3D facetime）通信，使用捕获的4D人类内容的移动的AR应用程序，逼真和个性化化身的AI助手，能够为人类服务或与人类一起工作的人类感知机器人，将物理治疗师与远离治疗设施的受伤患者联系起来的远程康复，等等。（空间和时间）捕获和重建涉及人体、面部、身体附加物（如衣服）及其周围环境的动态场景。尽管先前对实时4D重建进行了研究，但仍然没有开源且鲁棒的重建系统可以对动态场景的拓扑变化进行建模，并以准确性和鲁棒性跟踪移动表面。该项目旨在弥合这些差距，开发一个开源和强大的4D重建框架，可以使更广泛的科学界以及行业联盟的研究人员和开发人员受益，包括5G医疗标准，AR和MR游戏引擎，逼真的AI助手，人类感知机器人，远程康复等。和K-12学生通过夏令营。 该项目提出的研究围绕动态场景中拓扑变化的优雅建模和移动表面的鲁棒跟踪，最终目标是开发一个开源的鲁棒4D重建框架，用于实时捕获动态人体场景。为了解决拓扑变化的挑战，体积融合框架及其数据结构将从根本上重新设计，通过将非流形体积网格引入截断符号距离场（TSDF）和嵌入变形图（EDG）表示，允许体积单元复制自己和边缘被打破。这种新颖的拓扑变化感知框架将允许重构的网格几何结构在整个4D捕获过程中沿着更新其节点之间的连接的灵活变形图来动态地更新其连接。为了解决4D动态人体捕捉中的鲁棒表面跟踪问题，提出了一种参数化可动画体模型（PAVM），它联合收割机了参数化人体模型和体TSDF的优点。TSDF体积网格建立在参数化人体表面之上，因此它们可以用于表示人体的附加组件（例如衣服）。我们的PAVM的规则体积结构使其易于集成到深度神经网络中，以实现强大的表面跟踪，并提供语义建模功能。4D重建框架的技术可行性将通过移动的3D Facetime测试平台的开发得到验证，该测试平台将使偏远地区的人们能够以自然的AR方式相互交流。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

GPU-Based Supervoxel Generation With a Novel Anisotropic Metric

• DOI: 10.1109/tip.2021.3120878
• 发表时间: 2021-10
• 期刊: IEEE Transactions on Image Processing
• 影响因子: 10.6
• 作者: Xiaopan Dong;Zhonggui Chen;Yong-Jin Liu;Junfeng Yao;Xiaohu Guo

GPU-based supervoxel segmentation for 3D point clouds

• DOI: 10.1016/j.cagd.2022.102080
• 发表时间: 2022-02
• 期刊: Comput. Aided Geom. Des.
• 作者: Xiaopan Dong;Yanyang Xiao;Zhonggui Chen;Junfeng Yao;X. Guo

Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image

分层服装网络：从单个图像生成多个隐式服装层

• 发表时间: 2022
• 期刊: 2022 Asian Conference on Computer Vision (ACCV
• 作者: Aggarwal, Alakh;Wang, Jikai;Hogue, Steven;Ni, Saifeng;Budagavi, Madhukar;Guo, Xiaohu
• 通讯作者: Guo, Xiaohu

Neighbor Reweighted Local Centroid for Geometric Feature Identification

• DOI: 10.1109/tvcg.2021.3124911
• 发表时间: 2021-11
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Tong Liu;Zhenhua Yang;Shaojun Hu;Zhiyi Zhang;Chunxia Xiao;Xiaohu Guo;Long Yang

Study of Vocal Muscle Strain with Skin Deformation Tracking System

• DOI: 10.1109/cbms55023.2022.00026
• 发表时间: 2022-07
• 期刊: 2022 IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS)
• 作者: S. Hogue;Adrianna C. Shembel;X. Guo