Multiple sensors system for sign language recognition

用于手语识别的多传感器系统

• 批准号: 227703-2011
• 负责人: Ahmed, Maher
• 金额: $ 1.02万
• 依托单位: Wilfrid Laurier University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2012
• 资助国家: 加拿大
• 起止时间: 2012-01-01 至 2013-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=507884
• 关键词: Multiple; sensors; system; sign; language

Sign language consists of symbolic gestures to establish communication by combining the shape, orientation and movement of hands and arms, as well as facial expressions to fluidly express a speaker's thoughts. American Sign Language is the primary means of communication for more than half million deaf people in the United States and Canada. My research is focused on communication with the hearing-impaired by developing a software tool that can interpret sign language. There are two methods of collecting information about hand gestures, data gloves and video cameras. Data glove is a glove equipped with sensors that sense the movements of the hand and interfaces those movements with a computer. The data obtained from data gloves are concise and requires little post-processing. However, the data gloves are limited by the number of sensors and they do not capture facial expressions which some signers use to communicate additional information. Some recent research has been shown that video cameras can capture hands positions as well as facial expressions. However, such cameras are prone to typical computer vision problems such as occlusion and sensitivity to lighting conditions. These vision-based methods typically use image processing techniques to segment the hand from the image background, and extract features such as the fingertips for use in gesture recognition. The proposed system depends on the inputs from the gloves and a single camera. It consists of four modules. First, the real time hand tracking and extraction algorithm is applied to trace the moving hand and extract the hand region. Secondly, Fourier or wavelet descriptors will be adopted to characterize the spatial features. A motion analysis will be conducted to characterize the temporal features. The new feature vector will be created with both spatial and temporal features. In the third module, the description of hand shape, orientation, and movements are used as inputs to the training module for the segmentation and recognition. The collected data will be transferred to the fourth module for classification and the word corresponding to the sign will be determined. This research will allow a hearing-impaired person to communicate with other people.

手语由象征性的手势组成，通过结合手和手臂的形状，方向和运动来建立沟通，以及面部表情来流畅地表达说话者的思想。美国手语是美国和加拿大50多万聋人的主要交流手段。我的研究重点是通过开发一种可以解释手语的软件工具来与听力受损者进行交流。有两种方法可以收集有关手势的信息，数据手套和摄像机。数据手套是一种配备有传感器的手套，这些传感器可以感知手的运动并将这些运动与计算机连接。从数据手套获得的数据是简洁的，需要很少的后处理。然而，数据手套受到传感器数量的限制，并且它们不能捕捉一些签名者用来传达额外信息的面部表情。最近的一些研究表明，摄像机可以捕捉手的位置以及面部表情。然而，这样的相机容易出现典型的计算机视觉问题，例如遮挡和对照明条件的敏感性。这些基于视觉的方法通常使用图像处理技术来从图像背景中分割手，并提取诸如指尖的特征以用于手势识别。所提出的系统依赖于来自手套和单个相机的输入。它由四个模块组成。首先，采用真实的时间手跟踪和提取算法跟踪运动的手，提取手的区域。其次，傅立叶或小波描述符将被用来描述空间特征。将进行运动分析以表征时间特征。新的特征向量将被创建为具有空间和时间特征。在第三个模块中，手的形状，方向和运动的描述被用作输入到训练模块的分割和识别。采集到的数据将被传送到第四模块进行分类，并确定与标志相对应的词。这项研究将使听力受损的人能够与其他人交流。