面向数字虚拟人的手物交互动态三维重建方法研究

数字虚拟人可突破时间和空间的限制，满足人类“跨时越空”的社会交流需求。高逼真数字虚拟人的运动生成离不开手与物体的交互运动重建，是国际上本领域的前沿研究热点。本项目致力于面向数字虚拟人应用的手物交互动态三维重建难题，探索从视觉信息恢复手物交互动力学的技术框架，构建符合物理约束的高精度与通用手物交互动态三维信息采集系统与平台。具体研究内容包括：构建多对象、多姿态的高精度手势三维模型数据库，分析手势三维模型的空间结构，提出高精度手部通用三维表征模型；同时考虑手势的二维轮廓与姿态信息，提出基于深度学习的级联全卷积神经网络架构，实现单张彩色图像的三维手势姿态估计；结合智能体与强化学习思想，将时序卷积网络应用于物理动力学微分方程求解，突破传统“拟合—优化”的动力学物理量计算范式，实现灵活可靠的手物交互动态三维重建。

Digital virtual humans can break through the restrictions of time and space, and meet the social communication needs of human beings. The motion generation of high-fidelity digital virtual humans is inseparable from the interactive motion reconstruction of hands and objects, and is a frontier research hotspot in the field. This project is dedicated to the problem of hand-object interaction dynamic 3D reconstruction for digital virtual human applications, exploring the technical framework for restoring hand-object interaction dynamics from visual information, and building a high-precision and general hand-object interaction dynamic 3D information acquisition system and platform that meets physical constraints. The main contents of this project includes: Build a multi-object, multi-gesture high-precision gesture 3D model database, analyze the spatial structure of the gesture 3D model, and propose a high-precision hand general 3D representation model; Take into account the 2D contour and gesture information of the gesture, and propose deep learning based cascaded fully convolutional neural network architecture, to estimate the three-dimensional gesture pose estimation from a single color image; Combine the idea of agents and reinforcement learning, apply the time-series convolutional network to solve the differential equations of physical dynamics, and finally break the traditional "fitting - optimization" physical quantity calculation framework to achieve flexible and reliable hand-object dynamic 3D reconstruction result.