EAGER: Visual, Tactile, and Acoustic Signal Analysis and Perception for Tactile-Acoustic Display

多媒体技术在通信、商业、娱乐、艺术、教育和医学等领域日益占据主导地位。由于现代电子媒体有丰富的图形和图片信息，视障人士很难跟上。虽然这些信息中的一些也可以以语音或盲文文本的形式呈现，但以触觉形式直接呈现图形和图片信息的能力，与听觉信号相结合，将极大地增加这个庞大社区的成员可以获得的信息量，并将推动虚拟现实和医学等各种应用的界面的进步。尽管到目前为止，由于缺乏多功能设备，触觉受到的关注相对较少，但触觉显示技术的最新进展为新的研究提供了动力。虽然现有的触觉设备大多是静态的，但基于电活性聚合物、气动和MEMS等新兴技术的动态设备前景广阔。动态设备将使生成和显示任意触觉模式成为可能，但要估计不同设备配置的能力，理解设备特征并对其进行建模以及它们如何与人类感知相关是很重要的。结果表明，利用精确的静态物理模型可以模拟出相关的特性(材料、表面形状)。这为潜在的变革性研究奠定了基础，通过利用触觉显示设备的能力与人类触觉和听觉感知的能力相结合，使图形和图像信息能够以触觉-声学的形式呈现。为了实现这一目标，需要研究与现有设备或新设备设计相关的触觉知觉的基本问题，以及研究视觉、触觉和听觉知觉之间的基本关系。PI在这个探索性项目中的目标是沿着这些路线进行初步工作，以确定该方法的可行性。为此，他将开发触觉设备和感知的数学模型，并进行实验以验证它们。研究子任务将包括开发合成触觉纹理的算法，开发视觉、触觉和听觉纹理的结构相似性度量，以及视觉、触觉和听觉纹理感知维度的定量描述。对有视力(视障)和视障的人进行的测试将衡量我们在有和没有声音反馈的情况下区分触觉模式的能力，识别触觉纹理感知的维度(例如粗糙度、方向性)，建立模式标签可以在有和没有声音提示的情况下学习，并探索大脑将触觉信息整合到场景中的能力。广泛的影响：这项研究将有助于在感觉替代和使用触摸进行人机交互方面的根本性进展。它将解决视觉、触觉和听觉纹理分析和感知方面的基本问题，以及使用触摸进行图形和图像信息的交流。项目成果将有助于更深入地了解触觉及其与视觉的关系。除了最终使视障人士能够获取图片信息外，这项研究还将对许多其他领域产生影响，包括虚拟现实、触觉反馈界面、产品设计和医疗应用。

The world is increasingly dominated by multimedia technology for communication, commerce, entertainment, art, education, and medicine. Since modern electronic media are rich in graphical and pictorial information, it has been hard for the population of visually impaired people to keep up. While some of this information can also be presented as speech or Braille text, the ability to directly present graphical and pictorial information in tactile form, in combination with auditory signals, would dramatically increase the amount of information that can be made available to the members of this large community, and would also drive advances in interfaces for diverse applications such as virtual reality and medicine. Although the tactile sense has to date received relatively little attention due to the lack of versatile devices, recent advances in tactile display technology provide impetus for new research. While existing tactile devices are mostly static, there is great promise for dynamic devices based on emerging technologies such as electro-active polymers, pneumatics, and MEMs. Dynamic devices would make it possible to generate and display arbitrary tactile patterns, but to estimate the capabilities of different device configurations it is important to understand and model the device characteristics and how they relate to human perception. It has been shown that the relevant characteristics (material, surface shape) can be simulated using accurate static physical models. This sets the stage for potentially transformative research to enable the presentation of graphical and pictorial information in tactile-acoustic form, by exploiting the capabilities of tactile display devices in combination with the abilities of human tactile and auditory perception. To reach that goal will require the investigation of fundamental issues in tactile perception as it relates to existing devices or the design of new ones, and the study of fundamental relationships among visual, tactile, and auditory perception. The PI's objective in this exploratory project is to conduct preliminary work along these lines in order to establish the feasibility of the approach. To this end, he will develop mathematical models for tactile devices and perception, and conduct experiments to validate them. Research subtasks will include development of algorithms for synthesizing tactile textures, development of structural similarity metrics for visual, tactile, and acoustic textures, and quantitative description of perceptual dimensions of visual, tactile, and acoustic textures. Tests with sighted (visually blocked) and visually-impaired people will measure our ability to discriminate among tactile patterns with and without acoustic feedback, identify dimensions of tactile texture perception (e.g., roughness, directionality), establish that pattern labels can be learned with and without acoustic cues, and explore the brain's ability to integrate tactile information into a scene.Broader Impacts: This research will contribute to fundamental advances in sense substitution and the use of touch for human-computer interaction. It will address fundamental problems in visual, tactile, and acoustic texture analysis and perception, and the use of touch for communication of graphical and pictorial information. Project outcomes will contribute to a deeper understanding of the sense of touch and its relation to vision. In addition to ultimately enabling visually impaired people to access pictorial information, the research will have an impact on a number of other areas, including virtual reality, interfaces with tactile feedback, product design, and medical applications.