EAGER: Visual, Tactile, and Acoustic Signal Analysis and Perception for Tactile-Acoustic Display

多媒体技术在通信、商业、娱乐、艺术、教育和医学方面日益占据主导地位。 由于现代电子媒体的图形和图像信息丰富，视障人群很难跟上。 虽然这些信息中的一些也可以以语音或盲文文本的形式呈现，但以触觉形式直接呈现图形和图像信息的能力，结合听觉信号，将大大增加可以提供给这个大型社区的成员的信息量，并且还将推动虚拟现实和医学等各种应用程序的界面的进步。 尽管由于缺乏通用设备，触觉迄今为止受到的关注相对较少，但触觉显示技术的最新进展为新的研究提供了动力。 虽然现有的触觉设备大多是静态的，但基于新兴技术（如电活性聚合物，电化学和MEMS）的动态设备有很大的前景。 动态设备可以生成和显示任意的触觉模式，但要估计不同设备配置的能力，重要的是要理解和建模设备特性以及它们与人类感知的关系。 它已被证明，相关的特性（材料，表面形状）可以使用精确的静态物理模型进行模拟。 这为潜在的变革性研究奠定了基础，通过利用触觉显示设备的能力与人类触觉和听觉感知的能力相结合，使图形和图像信息以听觉形式呈现。 要达到这一目标，就需要研究触觉感知中的基本问题，因为它与现有设备或新设备的设计有关，并研究视觉，触觉和听觉感知之间的基本关系。 PI在该探索性项目中的目标是沿着这些路线进行沿着初步工作，以确定该方法的可行性。 为此，他将开发触觉设备和感知的数学模型，并进行实验来验证它们。 研究子任务将包括开发合成触觉纹理的算法，开发视觉，触觉和声学纹理的结构相似性度量，以及视觉，触觉和声学纹理的感知维度的定量描述。 对视力正常（视觉障碍）和视力受损的人进行的测试将测量我们在有和没有声音反馈的情况下区分触觉模式的能力，识别触觉纹理感知的维度（例如，粗糙度，方向性），建立了模式标签可以学习和没有声音线索，并探讨大脑的能力，整合触觉信息到一个场景。更广泛的影响：这项研究将有助于在感官替代和使用触摸人机交互的根本进步。 它将解决视觉，触觉，声学纹理分析和感知的基本问题，以及使用触摸来传达图形和图像信息。 项目成果将有助于更深入地了解触觉及其与视觉的关系。 除了最终使视障人士能够访问图像信息外，该研究还将对其他一些领域产生影响，包括虚拟现实，触觉反馈界面，产品设计和医疗应用。

The world is increasingly dominated by multimedia technology for communication, commerce, entertainment, art, education, and medicine. Since modern electronic media are rich in graphical and pictorial information, it has been hard for the population of visually impaired people to keep up. While some of this information can also be presented as speech or Braille text, the ability to directly present graphical and pictorial information in tactile form, in combination with auditory signals, would dramatically increase the amount of information that can be made available to the members of this large community, and would also drive advances in interfaces for diverse applications such as virtual reality and medicine. Although the tactile sense has to date received relatively little attention due to the lack of versatile devices, recent advances in tactile display technology provide impetus for new research. While existing tactile devices are mostly static, there is great promise for dynamic devices based on emerging technologies such as electro-active polymers, pneumatics, and MEMs. Dynamic devices would make it possible to generate and display arbitrary tactile patterns, but to estimate the capabilities of different device configurations it is important to understand and model the device characteristics and how they relate to human perception. It has been shown that the relevant characteristics (material, surface shape) can be simulated using accurate static physical models. This sets the stage for potentially transformative research to enable the presentation of graphical and pictorial information in tactile-acoustic form, by exploiting the capabilities of tactile display devices in combination with the abilities of human tactile and auditory perception. To reach that goal will require the investigation of fundamental issues in tactile perception as it relates to existing devices or the design of new ones, and the study of fundamental relationships among visual, tactile, and auditory perception. The PI's objective in this exploratory project is to conduct preliminary work along these lines in order to establish the feasibility of the approach. To this end, he will develop mathematical models for tactile devices and perception, and conduct experiments to validate them. Research subtasks will include development of algorithms for synthesizing tactile textures, development of structural similarity metrics for visual, tactile, and acoustic textures, and quantitative description of perceptual dimensions of visual, tactile, and acoustic textures. Tests with sighted (visually blocked) and visually-impaired people will measure our ability to discriminate among tactile patterns with and without acoustic feedback, identify dimensions of tactile texture perception (e.g., roughness, directionality), establish that pattern labels can be learned with and without acoustic cues, and explore the brain's ability to integrate tactile information into a scene.Broader Impacts: This research will contribute to fundamental advances in sense substitution and the use of touch for human-computer interaction. It will address fundamental problems in visual, tactile, and acoustic texture analysis and perception, and the use of touch for communication of graphical and pictorial information. Project outcomes will contribute to a deeper understanding of the sense of touch and its relation to vision. In addition to ultimately enabling visually impaired people to access pictorial information, the research will have an impact on a number of other areas, including virtual reality, interfaces with tactile feedback, product design, and medical applications.