Vision-based Sign Language Recognition in Complicated Background and Occlusion

本研究实现了在无背景条件下从图像序列中识别手语单词的技术。它包括以下三个子主题：在复杂背景下，利用手语词字典对手部区域进行检测和跟踪。利用基于手语场景示例的过渡网络，在混乱背景下，即使手部快速运动，也能稳定地检测和跟踪手部区域及其形状变化。该网络在手部快速运动时显著使用运动特征，在手部缓慢运动时显著使用形状特征来注册手部状态模型。提出了一种基于背景和手部图像对比度出现概率的模型匹配准则。基于时间和空间图像特征的手语单词识别，提取了约50万个手语单词的二维图像特征，包括手指状部位的数量、手区域的主轴、手相对于面部的位置和二维运动参数，并训练了隐马尔可夫模型（HMM）进行单词识别。由于手和脸之间的相互遮挡，当遮挡检测到时，使用前一帧中存储的纹理来跟踪每个区域。采用两阶段匹配方法，首先对隐马尔可夫模型进行位置和运动特征的粗略匹配，然后对隐马尔可夫模型进行所有特征的精确匹配，提高了识别成功率。基于三维形状模型和外观变形学习的手部形状估计基于轮廓匹配的单眼图像序列三维形状参数估计。预先生成多个可能的手形轮廓图像，并在模型数据库中对轮廓、关节角度和观察参数进行配准。由于人手的自由度很大，模型必须是稀疏采样的。为了弥补这一缺陷，提出了一种新的局部压缩流形结构，该结构以参数化的方式提取和存储每个模型周围可能的外观变化。构建了pc聚类估计系统，并对复杂的自遮挡手形进行了10fps估计。少

This research achieved techniques which enable to recognize sign language words from image sequences in background-free conditions. It includes tree sub-themes below :1.hand region detection and tracking in complex backgrounds using sign language word dictionaryHand region and its shape change was stably detected and tracked even when the hand moves quickly in cluttered backgrounds using a Transition network built from examples of sign language scenes. The network registers hand state models using motion feature significantly when hand moves quickly or using shape feature when the motion is slow. A novel model-matching criterion based on the probabilities of image contrast occurrance in both of hand and background was proposed.2.Sign language word recognition based on temporal and spacio image features2-D image features including the number of finger-like parts, principal axes of hand region, position of hand relative to the face and 2-D motion parameters were extracted from about 50 s … More ign language words and trained Hidden Markov Model (HMM) for word recognition. Because mutual occlusion between both hands and the face, when occlusion is detected, each region is tracked using textures stored in the preceding frames. Recognition success rate was improved by using two stage matching : first rough matching by the HMM using only position and motion features is done, and then precise matching by the HMM using all of features is done.3.Hand shape estimation based on 3-D shape model and learning of appearance deformation3-D shape parameters was estimated from monocular image sequence based on silouette matching. A number of possible hand shape silhouette images are generated in advance and the pairs of the silhouette and the joint angles and viewing parameters are registered in a model database. Because the degree of freedom of human hand is quite large, the models have to been sparsely sampled. In compensation for it, locally-compressed manifold architechture, which extracts and stored the possible appearance changes around each model in a parametric way, is novelly proposed. A PC-clustered estimation system was built and established 10 fps estimation even far the complicatedly self-occluded hand shape. Less