Multiple sensors system for sign language recognition

用于手语识别的多传感器系统

• 批准号: 227703-2011
• 负责人: Ahmed, Maher
• 金额: $ 1.02万
• 依托单位: Wilfrid Laurier University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2015
• 资助国家: 加拿大
• 起止时间: 2015-01-01 至 2016-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=570305
• 关键词: Multiple; sensors; system; sign; language

Sign language consists of symbolic gestures to establish communication by combining the shape, orientation and movement of hands and arms, as well as facial expressions to fluidly express a speaker's thoughts. American Sign Language is the primary means of communication for more than half million deaf people in the United States and Canada. My research is focused on communication with the hearing-impaired by developing a software tool that can interpret sign language. There are two methods of collecting information about hand gestures, data gloves and video cameras. Data glove is a glove equipped with sensors that sense the movements of the hand and interfaces those movements with a computer. The data obtained from data gloves are concise and requires little post-processing. However, the data gloves are limited by the number of sensors and they do not capture facial expressions which some signers use to communicate additional information. Some recent research has been shown that video cameras can capture hands positions as well as facial expressions. However, such cameras are prone to typical computer vision problems such as occlusion and sensitivity to lighting conditions. These vision-based methods typically use image processing techniques to segment the hand from the image background, and extract features such as the fingertips for use in gesture recognition. The proposed system depends on the inputs from the gloves and a single camera. It consists of four modules. First, the real time hand tracking and extraction algorithm is applied to trace the moving hand and extract the hand region. Secondly, Fourier or wavelet descriptors will be adopted to characterize the spatial features. A motion analysis will be conducted to characterize the temporal features. The new feature vector will be created with both spatial and temporal features. In the third module, the description of hand shape, orientation, and movements are used as inputs to the training module for the segmentation and recognition. The collected data will be transferred to the fourth module for classification and the word corresponding to the sign will be determined. This research will allow a hearing-impaired person to communicate with other people.

手语由象征性手势组成，通过结合手和手臂的形状、方向和运动来建立交流，以及流畅地表达说话者思想的面部表情。美国手语是美国和加拿大超过 50 万聋人的主要交流方式。我的研究重点是通过开发一种可以解释手语的软件工具来与听力障碍者进行交流。收集手势、数据手套和摄像机信息的方法有两种。数据手套是一种配备传感器的手套，可以感知手的运动并将这些运动与计算机连接。从数据手套获得的数据简洁，几乎不需要后处理。然而，数据手套受到传感器数量的限制，并且它们无法捕捉一些手​​语者用来传达附加信息的面部表情。最近的一些研究表明，摄像机可以捕捉手部位置以及面部表情。然而，此类相机容易出现典型的计算机视觉问题，例如遮挡和对照明条件的敏感性。这些基于视觉的方法通常使用图像处理技术将手从图像背景中分割出来，并提取指尖等特征以用于手势识别。所提出的系统取决于手套和单个摄像机的输入。它由四个模块组成。首先，应用实时手部跟踪和提取算法来跟踪移动的手部并提取手部区域。其次，将采用傅立叶或小波描述符来表征空间特征。将进行运动分析来表征时间特征。新的特征向量将使用空间和时间特征来创建。在第三个模块中，手部形状、方向和动作的描述被用作训练模块的输入，以进行分割和识别。收集到的数据将被传输到第四模块进行分类，并确定该符号对应的单词。这项研究将使听力障碍者能够与其他人交流。