手势语识别是模式识别和智能人机交互领域的研究热点，具有重要的理论意义和应用价值。基于视觉的手势语识别中，在复杂动态背景、视角变换和用户变换情况下其系统性能下降明显，已成为阻碍手势语识别研究发展的瓶颈问题；同时手势语的多模态特性也未能受到足够重视；本项目在以往相关研究的基础上对这些问题逐一进行研究。针对复杂动态背景下的手势语识别问题，研究基于多示例学习框架的手势语识别方法，以确保系统在手部区域定位和跟踪失准时仍能正常识别；针对手势语的多模态特性，充分利用手部信息以外的唇动、表情、头势等信息，研究基于多模态的多示例学习方法；针对视角变换和用户变换情况，采用半监督学习机制，利用大量无标注数据更新模型参数，来提高系统的可移植性。

Sign language recognition is a research focus in the pattern recognition and the intelligent human-computer interaction fields, and is of great theoretical significance and practical value. For vision based sign language recognition, the performance decreases obviousely in the case of cluttered dynamic backgrounds, view and signer variances, which limits the progress of sign language recognition research. At the same time, the multimodal characteristic of sign language is not mined sufficiently in the previous research. This project aims to solve these problems. To solve the problem of recognition in the clutteredd dynamic backgrounds, we introduce the multi-instance learning to sign language recognition, which can recognize gestures even when the hands are not well located and tracked. For multimodal sign lanuage recognition, we add the lip movement, facial expression and head pose information to the multimodal multi-instance framework. To recognize sign language in the cases of view variance and signer variance, we use large amount of unlabeled data to tailor the parameters of the models with the scheme of semi-supervised learning.