Telesign: Towards a One-Way American Sign Language Translator

PI假设聋人和听力社区之间的交流可以通过使用移动单向手语翻译器来加强。他的这种翻译器的初步原型被称为TeleSign，它使用了一台穿戴在身体上的计算机，帽子边缘的扬声器，一个用于用户反馈和编辑的平视免提微型显示器，以及各种传感器(例如，嵌入用户帽子中的摄像头，戴在腕带上的无线加速计，以及Acceleglov加速计/电位计系统)。他在这个项目中的第一步将是比较聋人和听力(非手语)人之间的不同沟通方法--包括书写(目前的做法)、在移动设备上打字、手势驱动的单向手势翻译器和短语级别的单向翻译器--这两种方法都是为了衡量现有通信技术的表现，并为未来对TeleSign的评估建立基准。这些方法将通过速度(每秒传达的命题和完成任务的时间)、感知工作量(NASA任务负载指数)和英语生成的清晰度进行评估。然后，PI将显著扩展TeleSign原型，将语音识别技术应用于手语识别，方法是采用他的佐治亚理工学院手势工具包(GT2K)，这是一个公开可用的用于手势识别的隐马尔可夫模型(HMM)工具包，用于这些任务。PI还将研究传感技术，例如在移动环境中对手进行视觉跟踪，以便在更长的时间内为手语翻译测试创建合理的原型。最后，PI将定义可用于评估计算机增强的聋人通信辅助工具(现有技术和新的改进的TeleSign系统)的性能的研究级任务。广泛的影响：与听力社区的日常交流对数百万听力损失的人构成了重大挑战。大多数听力正常的人不懂手语，对耳聋也知之甚少。这个项目试图通过研究语言产生的辅助工具来降低聋人和听力人口之间的沟通障碍。然而，日常使用成功的交流辅助工具也可以帮助签名者更多地了解英语的书面和口语结构，就像日常互动帮助学生学习一门外语一样。这项研究将佐治亚理工学院的学生与亚特兰大地区聋人学校(一所K-12州立学校)的顾问和受试者配对，让美国一些顶尖的学生工程师和科学家接触到聋人社区，反之亦然。聋人学者和聋人社区的本土签名者将被积极招募为成员或研究团队，这也将在研究变得可行时帮助传播研究。

The PI hypothesizes that communication between the deaf and hearing communities can be augmented through the use of a mobile one-way sign language translator. His preliminary prototype of such a translator, dubbed Telesign, uses a body-worn computer, a speaker in the brim of the hat, a head-up, hands-free micro-display for user feedback and editing, and a variety of sensors (e.g., a camera embedded in the user's hat, wireless accelerometers worn in wristbands, and the Acceleglove accelerometer/potentiometer system). His first step in this project will be to compare different methods of communication between a deaf and hearing (non-signing) individual - including writing (current practice), typing on mobile devices, a gesture driven sign-by-sign one-way translator, and a phrase-level one-way translator - both to provide a measure for the performance obtained with existing communication technologies and to establish the baseline for future evaluation of Telesign. These methods will be evaluated by speed (propositions conveyed per second and time to task completion), perceived workload (NASA Task Load Index), and clarity of English generation. The PI will then significantly extend the Telesign prototype, by bringing speech recognition techniques to bear on sign language recognition by adapting his Georgia Tech Gesture Toolkit (GT2K), a publicly available hidden Markov model (HMM) toolkit for gesture recognition, for these tasks. The PI will also research sensing techniques, such as visual tracking of the hands in a mobile environment, in order to create reasonable prototypes for sign translation testing over an extended time. Finally, the PI will define research-grade tasks that can be used to evaluate the performance of computer enhanced communication aids for the deaf (both existing techniques and the new improved Telesign system).Broader Impacts: Everyday communication with the hearing community poses a major challenge to the millions of people with hearing loss. Most hearing people do not know sign language, and know very little about deafness in general. This project attempts to lower the communication barrier between the deaf and hearing populations, by researching aids for language production. However, everyday use of a successful communication aid may also help a signer learn more about the structure of written and spoken English, in much the same way that everyday interactions help students learn a foreign language. This research couples Georgia Tech students with consultants and subjects from the Atlanta Area School for the Deaf (a K-12 state school), exposing some of the nation's top student engineers and scientists to the deaf community, and vice versa. Deaf academics and native signers from the deaf community will be recruited actively as members or the research team, which will also help disseminate the research as it becomes viable.