Hand Tracking and Affine Shape-Appearance Handshape Sub-units in Continuous Sign Language Recognition
Hand Tracking and Affine Shape-Appearance Handshape Sub-units in Continuous Sign Language Recognition
复制标题
连续手语识别中的手部跟踪和仿射形状外观手形子单元
DOI：
--
复制
发表时间：
2010
期刊：
影响因子：
--
通讯作者：
P. Maragos
中科院分区：
文献类型：
--
作者：
A. Roussos;Stavros Theodorakis;Vassilis Pitsikalis;P. Maragos
我们提出并研究了一个框架，利用新的方面的概率和形态视觉处理的分割，跟踪和手形建模的手，这是作为前端的手语视频分析。我们的最终目标是探索手形子单位（HSU）的自动构建，并在自动手语识别（ASLR）的整体系统的开发。我们采用概率肤色检测，然后提出的形态学算法和相关的形状过滤快速和可靠的分割手和头。然后，这是我们的手跟踪系统，强调强大的处理遮挡的基础上向前向后预测和概率约束的结合。利用仿射不变建模的手形状外观图像的跟踪，提供了一个紧凑的和描述性的表示的手配置。我们进一步建议，通过该模型的拟合提取的手形特征被用来构建在一个无监督的方式基本HSU。我们首先提供直观的HSU符号映射的结果，并进一步定量评估集成系统和构造的HSU在子单元和符号水平上的ASLR实验。这些都进行了连续SL数据从BU400语料库，并调查所涉及的参数的效果。实验结果表明，整体方法的有效性，特别是手形建模时，纳入HSU为基础的框架显示出可喜的成果。
We propose and investigate a framework that utilizes novel aspects concerning probabilistic and morphological visual processing for the segmentation, tracking and handshape modeling of the hands, which is used as front-end for sign language video analysis. Our ultimate goal is to explore the automatic Handshape Sub-Unit (HSU) construction and moreover the exploitation of the overall system in automatic sign language recognition (ASLR). We employ probabilistic skin color detection followed by the proposed morphological algorithms and related shape filtering for fast and reliable segmentation of hands and head. This is then fed to our hand tracking system which emphasizes robust handling of occlusions based on forward-backward prediction and incorporation of probabilistic constraints. The tracking is exploited by an Affine-invariant Modeling of hand Shape-Appearance images, offering a compact and descriptive representation of the hand configurations. We further propose that the handshape features extracted via the fitting of this model are utilized to construct in an unsupervised way basic HSUs. We first provide intuitive results on the HSU to sign mapping and further quantitatively evaluate the integrated system and the constructed HSUs on ASLR experiments at the sub-unit and sign level. These are conducted on continuous SL data from the BU400 corpus and investigate the effect of the involved parameters. The experiments indicate the effectiveness of the overall approach and especially for the modeling of handshapes when incorporated in the HSU-based framework showing promising results.