Research on Practical Sign Language Interpretation by integration of image processing of hands and other information

1. 颜色用于手和脸区域的提取。由于这些区域的颜色可能因人而异，因此我们开发了一种在色彩空间中确定肤色区域的方法。在该方法中，首先使用先验的肤色区域提取近似区域。该区域是基于提取的手和脸区域的属性进行细化的。此外，我们还开发了一种方法，以避免在手快速移动时包含背景颜色。当手和脸重叠时，首先在皮肤区域确定面部区域，然后在皮肤区域的其余部分搜索手。由于人脸区域不能精确确定，因此利用人脸区域的确定性来匹配皮肤区域的手部模板。我们开发了一种从一组真实样本中合成手语样本的方法，以增加HMM学习的样本。虽然实验证明了识别率的提高，但提高的幅度很小。原因之一是由于特征提取不当，合成的样本特征不合适。改进了特征提取方法。对于手语单词的识别，我们使用无分支的HMM。因为同一个手语单词的特征因人而异，所以我们必须准备多个模型。我们开发了一种带有分支的HMM构建方法，以促进多种手语特征的模型。为了在手语识别中识别面部表情，我们开发了一种确定面部部位位置和形状，并检测面部表情有用特征的方法。在一个小规模的实验中，用这种方法识别出了诸如疑问、喜悦和愤怒等典型表情。

1. Color is used for extraction of hands and face regions. Because the color of these regions may be varied depending on persons, we developd a method of determining the skin color region in a color space. In this method, approximate regions are first extracted using an apriori skin color region. This region is refined based on the properties of extracted hands and face regions.In addition, we develop a method to avoid inclusion of the background colors while hands are moving quickly.2. When hands and face overlap, the face region is first determined in the skin region, and the rest of the skin region is searched for hands. Because the face region is not determned precisely, the certainty of the face region is used for matching hads templates in the skin region.3. We developed a method of synthesizing sign language samples from a set of real samples to increase samples for HMM learning. Although impovement of recognition rate is proved by experiments, the amount of improvement is small. One reason is that the features of synthesized samples are not suitable because of improper feature extraction. We imporved our feature extraction method.4. For recognition of sign language words, we used HMM without branching. Because the features of the same sign language word varies depending on persons, we had to prepare multiple models. We have developed a method of building HMM with branching to facilitate models of multiple sign language features.5. In order to recognize face expressions for sign language recognition, we developed a method of determining the position and shape of face parts, and detecting useful features of face face expressions. In a small scale experiment, typical expressions such ad question, joy, and anger are identified by the method.