Statistical Approach to Real-time Estimation of Shape and Pose of Hand and Fingers in Image Sequence

我们提出了一种手部跟踪和手部和手指三维运动捕捉的新方法。这种新的动作捕捉系统不需要在手上安装标记。一只手有许多关节。为了降低计算复杂度，我们提出对数据手套采集的关节角度数据采用独立分量分析进行降维，使得手形可以只用5个参数表示。手部跟踪的另一个问题是，当手进行自由运动时，经常会发生自遮挡。为了解决这个问题，我们使用了多个校准相机。有了多个视点，一个摄像机看不到的部分仍然可以在其他摄像机上看到。对于每张图像，我们尝试估计代表手部形状的5个参数和手部位置和姿势的6个参数（3个用于旋转，3个用于平移）。这些参数的确定是为了最大化输入图像和反向投影的3D手部模型之间的重叠。采用粒子滤波方法搜索最优解，对局部最优解具有较强的鲁棒性。通过对实际图像序列进行手部自由旋转和变形的测试，证明了该方法的有效性。实验结果表明，该系统具有较强的抗遮挡能力。然而，粒子滤波是耗时的，系统还不能实时工作。

We propose a new approach to hand tracking and 3D motion capture of hand and fingers. This new motion capture system does not require markers be attached to hands.A hand has many joints. To reduce computational complexity, we propose to reduce dimensionality applying Independent component analysis to the joint angle data collected by a data glove, and as a result the hand shape can be represented by only 5 parameters.Another problem with hand tracking is that when a hand undergoes free motion, self-occlusion occurs frequently. To solve this problem, we use multiple calibrated cameras. With multiple viewpoints, parts that are not visible to one camera can still be visible to other cameras.For each image, we try to estimate the 5 parameters representing hand shape and the 6 parameters for hand position and pose (3 for rotation and 3 for translation). These parameters are determined to maximize overlapping between the input image and the back-projected 3D hand model. The optimum is searched using particle filtering, which is robust to local optimum.The effectiveness of this approach is proved by testing the program to real image sequences with the hand rotating and deforming freely. Experimental results show that the system is robust against occlusion.However, particle filtering is time-consuming and the system does not work in real-time yet.