CHS: Medium: Collaborative Research: Increasing Communication Rates through a TActile Phonemic Sleeve (TAPS)

CHS：媒介：协作研究：通过触觉音素套筒 (TAPS) 提高通信速率

• 批准号: 1954842
• 负责人: Hong Tan
• 金额: $ 25万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-06-15 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1954842&HistoricalAwards=false
• 关键词: CHS; Medium; Collaborative; Research; Increasing

How much information can be transmitted through the skin? Conventional wisdom would say not as much as vision or hearing. The researchers argue, however, that there is a need to better understand how to "talk" to the skin in its own language. The team has recently devised a sleeve with 24 vibrators that delivers speech sounds in the form of distinct vibration patterns. Testing results from more than 100 normal-hearing individuals (in addition to one participant with a severe hearing loss) show that the vibration-based representation of speech sounds can be easily learned and combined to "sound out" English words in isolation. The best learners have demonstrated an ability to learn one English word per minute, up to a vocabulary size of 500 English words. This is an exciting breakthrough that can lead to a communication device for people with severe hearing impairments and for those whose hearing is temporarily compromised due to noisy environments such as firefighters. The main objective of this project is to systematically explore ways that the communication rate through the sleeve can be increased. Our goal is to reach at least 60-80 words per minute, a communication rate that has been demonstrated by the Tadoma method of speech communication, in which persons who are both deaf and blind place a hand over the face and neck of the talker to understand speech. Such a communication rate will make the sleeve a practical solution for the community of people who need an alternate means for speech communication, such as the ones with both hearing and vision impairments. If successful, the project will facilitate a better quality of life (in terms of social interactions as well as employability) for this disadvantaged community.This project addresses the fundamental question of the information capacity of the sense of touch within the context of increasing communication rates through a haptic display of speech. The project leverages the team's research and development experience on: (i) natural speech communication methods used by the deaf and deaf-blind; (ii) wearable haptic technologies; (iii) an information-theoretical approach to assessing human performance; and (iv) psychophysical properties of the tactual sense. The project builds upon the researchers' recent success at demonstrating their ability to develop tactile codes for the 39 phonemes of English that were highly recognizable in isolation and in words formed from sequences of phonemes. The research is focused on increasing the rate at which information can be transmitted through touch by trained participants that include observers with normal hearing as well as those with sensorineural hearing impairments who represent a major category of potential end users of the device. The overall goal is to achieve transmission rates of tactile speech stimuli that are comparable to those achieved by deaf-blind users of the Tadoma method, and to employ effective methods of training users of the display for reception of connected speech signals with a high degree of accuracy. Experiments will be carried out in which users of the tactile system are trained and tested on their ability to recognize phonemes, words, short phrases, and finally conversational sentences. Durational properties of the stimuli will be explored in these experiments with the aim of establishing the lower limits on phoneme duration and the intervals between phonemes and words. Estimates of effective communication rates will take into account both the transmission rate and accuracy of reception, with the goal of demonstrating effective communication rates on the order of 50-100 words per minute. A lower-bound estimate for haptic communication rate in terms of bits per second can then be derived from the transmission rate achieved at the end of this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

皮肤能传递多少信息？传统的智慧会比视觉或听觉更能说明问题。然而，研究人员认为，有必要更好地理解如何用皮肤的母语与其“交谈”。该团队最近设计了一种带有24个振动器的套筒，可以以不同的振动模式传递语音。对100多名听力正常的人(包括一名严重听力损失的参与者)的测试结果表明，基于振动的语音表示可以很容易地学习并组合成孤立的英语单词。最好的学习者已经证明了每分钟学习一个英语单词的能力，词汇量高达500个英语单词。这是一项令人兴奋的突破，可以为严重听力障碍的人以及因消防员等嘈杂环境而暂时听力受损的人带来一种通信设备。该项目的主要目标是系统地探索通过套筒提高通信速率的方法。我们的目标是至少达到每分钟60-80个单词，这一交流速度已经被Tadoma语音交流方法所证明，在这种方法中，聋人和盲人都将一只手放在说话者的面部和颈部，以理解语音。这样的通信速率将使袖子成为需要替代语音通信手段的人群的实用解决方案，例如听力和视觉障碍的人群。如果成功，该项目将促进这一弱势群体更好的生活质量(在社会互动和就业能力方面)。该项目解决了在通过触觉展示语言提高通信率的背景下，触觉信息能力的基本问题。该项目利用该团队在以下方面的研究和开发经验：(I)聋人和聋盲人使用的自然语言交流方法；(Ii)可穿戴触觉技术；(Iii)评估人类表现的信息理论方法；以及(Iv)触觉的心理物理特性。该项目建立在研究人员最近成功展示了他们为39个英语音素开发触觉代码的能力的基础上，这些音素在孤立的情况下和在由音素序列组成的单词中高度可识别。这项研究的重点是提高训练有素的参与者通过触摸传递信息的速度，这些参与者包括听力正常的观察者以及代表该设备潜在最终用户的主要类别的感觉神经性听力障碍的观察者。总体目标是获得与Tadoma方法的聋盲用户所实现的速率相当的触觉语音刺激的传输速率，并采用有效的方法来训练显示器的用户以高精度地接收连接的语音信号。将进行实验，对触觉系统的用户进行培训和测试，以了解他们识别音素、单词、短句和最终对话句子的能力。在这些实验中，将探索刺激的持续时间属性，目的是确定音素持续时间的下限以及音素和单词之间的间隔。对有效通信速率的估计将同时考虑传输速率和接收的准确性，目标是展示每分钟50-100字的有效通信速率。以比特每秒为单位的触觉通信速率的下限估计可以从本项目结束时获得的传输速率得出。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Identification of Words and Phrases Through a Phonemic-Based Haptic Display: Effects of Inter-Phoneme and Inter-Word Interval Durations

通过基于音素的触觉显示识别单词和短语：音素间和单词间间隔持续时间的影响

• DOI: 10.1145/3458725
• 发表时间: 2021
• 期刊: ACM Transactions on Applied Perception
• 影响因子: 1.6
• 作者: Reed, Charlotte M.;Tan, Hong Z.;Jiao, Yang;Perez, Zachary D.;Wilson, E. Courtenay
• 通讯作者: Wilson, E. Courtenay

Methodology for Maximizing Information Transmission of Haptic Devices: A Survey

最大化触觉设备信息传输的方法：调查

• DOI: 10.1109/jproc.2020.2992561
• 发表时间: 2020
• 期刊: Proceedings of the IEEE
• 影响因子: 20.6
• 作者: Tan, Hong Z.;Choi, Seungmoon;Lau, Frances W.;Abnousi, Freddy
• 通讯作者: Abnousi, Freddy

Snake Effect: A Novel Haptic Illusion

蛇效应：一种新颖的触觉错觉

• DOI: 10.1109/toh.2021.3070277
• 发表时间: 2021
• 期刊: IEEE Transactions on Haptics
• 影响因子: 2.9
• 作者: Severgnini, Frederico M.;Martinez, Juan S.;Tan, Hong Z.;Reed, Charlotte M.
• 通讯作者: Reed, Charlotte M.

Improving Tactile Codes for Increased Speech Communication Rates in a Phonemic-Based Tactile Display

改进触觉代码以提高基于音素的触觉显示中的语音通信速率

• DOI: 10.1109/toh.2020.3008869
• 发表时间: 2021
• 期刊: IEEE Transactions on Haptics
• 影响因子: 2.9
• 作者: Martinez, Juan S.;Tan, Hong Z.;Reed, Charlotte M.
• 通讯作者: Reed, Charlotte M.

Incidental Categorization of Vibrotactile Stimuli

振动触觉刺激的附带分类

• DOI: 10.1109/toh.2020.2965446
• 发表时间: 2020
• 期刊: IEEE Transactions on Haptics
• 影响因子: 2.9
• 作者: Martinez, Juan S.;Holt, Lori L.;Reed, Charlotte M.;Tan, Hong Z.
• 通讯作者: Tan, Hong Z.