Vision-based Sign Language Recognition in Complicated Background and Occlusion

本研究实现了在无背景条件下从图像序列中识别手语单词的技术。它包括以下三个子主题：1.使用手语单词词典在复杂背景中的手部区域检测和跟踪即使当手部在杂乱的背景中快速移动时，使用从手语场景的示例建立的过渡网络，也可以稳定地检测和跟踪手部区域及其形状变化。该网络在手部快速运动时使用运动特征记录手部状态模型，在运动缓慢时使用形状特征记录手部状态模型。提出了一种新的基于手部和背景图像对比度出现概率的模型匹配准则。2.基于时间和空间图像特征的手语词识别从约50个S…中提取了手部的指状部分数目、手区域的主轴、手相对于人脸的位置和二维运动参数等二维图像特征更多的同名语言单词，并训练隐马尔可夫模型进行单词识别。由于双手和面部之间的相互遮挡，因此当检测到遮挡时，将使用存储在前面帧中的纹理跟踪每个区域。采用两阶段匹配的方法提高了识别成功率：首先利用隐马尔可夫模型仅利用位置和运动特征进行粗匹配，然后利用所有特征进行精确匹配。3.基于三维形状模型的头发形状估计和外观变形学习。预先生成多个可能的手形轮廓图像，并且将轮廓和关节角度以及观察参数对登记在模型数据库中。由于人手自由度较大，需要对模型进行稀疏采样。为了弥补这一缺陷，提出了局部压缩流形结构，它以一种参数化的方式提取和存储每个模型周围可能发生的外观变化。建立了一个PC集群评估系统，对复杂的自遮挡手形进行了10fps的评估。较少

This research achieved techniques which enable to recognize sign language words from image sequences in background-free conditions. It includes tree sub-themes below :1.hand region detection and tracking in complex backgrounds using sign language word dictionaryHand region and its shape change was stably detected and tracked even when the hand moves quickly in cluttered backgrounds using a Transition network built from examples of sign language scenes. The network registers hand state models using motion feature significantly when hand moves quickly or using shape feature when the motion is slow. A novel model-matching criterion based on the probabilities of image contrast occurrance in both of hand and background was proposed.2.Sign language word recognition based on temporal and spacio image features2-D image features including the number of finger-like parts, principal axes of hand region, position of hand relative to the face and 2-D motion parameters were extracted from about 50 s … More ign language words and trained Hidden Markov Model (HMM) for word recognition. Because mutual occlusion between both hands and the face, when occlusion is detected, each region is tracked using textures stored in the preceding frames. Recognition success rate was improved by using two stage matching : first rough matching by the HMM using only position and motion features is done, and then precise matching by the HMM using all of features is done.3.Hand shape estimation based on 3-D shape model and learning of appearance deformation3-D shape parameters was estimated from monocular image sequence based on silouette matching. A number of possible hand shape silhouette images are generated in advance and the pairs of the silhouette and the joint angles and viewing parameters are registered in a model database. Because the degree of freedom of human hand is quite large, the models have to been sparsely sampled. In compensation for it, locally-compressed manifold architechture, which extracts and stored the possible appearance changes around each model in a parametric way, is novelly proposed. A PC-clustered estimation system was built and established 10 fps estimation even far the complicatedly self-occluded hand shape. Less