CRII : HCC : Smart Acoustic Surfaces as Multimodal Interfaces

CRII：HCC：作为多模态接口的智能声学表面

• 批准号: 2104758
• 负责人: Michael Heilemann
• 金额: $ 17.48万
• 依托单位: University of Rochester
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2104758&HistoricalAwards=false
• 关键词: CRII; HCC; Smart; Acoustic; Surfaces

Over the course of the last decade, smart devices have become essential components of daily life. However, apart from conversational assistants, most smart devices and environments rely primarily on presenting information visually, a poor fit for many situations where users' visual attention is on something besides the device. One way to support these situations is to let people use sound or touch for interactions. However, adding separate microphones, loudspeakers, and vibration mechanisms using conventional means can impose extra costs and reduce the device’s durability and aesthetics. This project seeks to develop alternative technologies for recording and reproducing spatial audio through bending vibrations on flat surfaces ranging from smartphones to video walls. These vibrations can also allow the smart acoustic surface to serve as a touch interface. The advantage of this approach lies in the fact that the surface is already part of the device, allowing the device to maintain its durability and aesthetics while incorporating these new features. Adding spatial audio and haptic feedback to OLED displays, smartphones, and video walls will provide fundamental technology to improve people's ability to navigate complex data sets such as menus and maps, and enable a greater sense of immersion for remote applications such as video conference calls. The project develops a framework for designing surfaces such as the display screen of a device to serve as acoustic and vibrotactile interfaces. The research objectives will address the following three challenges by employing the vibro-acoustics of extended surfaces: (1) spatial audio reproduction, (2) spatial audio capture, and (3) haptic feedback and touch sensing. The vibrations of the extended surface induced by acoustic and touch interactions with the structure can be detected using arrays of sensors distributed on the surface. Similarly, arrays of force actuators distributed on the surface will induce bending vibrations that cause the structure to radiate sound. Loudspeaker and microphone array processing methods will be adapted for the vibration actuators and sensors with applications including noise reduction, sound-field control, and beamforming. Machine learning techniques will be used to identify source-location features stemming from vibration patterns induced by acoustic and touch interactions with the structure. Acoustic recordings and scanning laser vibrometer measurements will be used to analyze the effectiveness of the framework in real-world settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在过去的十年中，智能设备已经成为日常生活中必不可少的组成部分。然而，除了会话助手之外，大多数智能设备和环境主要依赖于视觉呈现信息，这对于许多用户视觉注意力不在设备上的情况不太适合。支持这些情况的一种方法是让人们使用声音或触摸进行互动。然而，使用传统方法增加单独的麦克风、扬声器和振动机制会增加额外的成本，并降低设备的耐用性和美观性。该项目旨在开发替代技术，通过在智能手机和视频墙等平面上弯曲振动来记录和再现空间音频。这些振动也可以让智能声学表面充当触摸界面。这种方法的优点在于，表面已经是设备的一部分，允许设备在结合这些新功能的同时保持其耐用性和美观性。将空间音频和触觉反馈添加到OLED显示器、智能手机和视频墙将提供基础技术，以提高人们浏览菜单和地图等复杂数据集的能力，并为视频电话会议等远程应用提供更大的沉浸感。该项目开发了一个框架，用于设计表面，如设备的显示屏，作为声学和振动触觉界面。研究目标将通过使用扩展表面的振动声学来解决以下三个挑战：(1)空间音频再现；(2)空间音频捕获；(3)触觉反馈和触摸传感。利用分布在表面上的传感器阵列，可以检测到由声学和触觉与结构相互作用引起的扩展表面的振动。同样，分布在表面上的力致动器阵列将引起弯曲振动，使结构辐射声音。扬声器和麦克风阵列处理方法将适用于振动致动器和传感器，其应用包括降噪、声场控制和波束形成。机器学习技术将用于识别由声学和触摸与结构相互作用引起的振动模式产生的源定位特征。声学记录和扫描激光测振仪测量将用于分析该框架在现实环境中的有效性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Audio Capture Using Structural Sensors on Vibrating Panel Surfaces

使用振动面板表面上的结构传感器进行音频捕获

• DOI: 10.17743/jaes.2022.0049
• 发表时间: 2022
• 期刊: Journal of the Audio Engineering Society
• 影响因子: 1.4
• 作者: Dipassio, Tre;Heilemann, Michael C.;Bocko, Mark F.
• 通讯作者: Bocko, Mark F.

Estimating the Direction of Arrival of a Spoken Wake Word Using a Single Sensor on an Elastic Panel

使用弹性面板上的单个传感器估计语音唤醒词的到达方向

• DOI: 10.1109/waspaa58266.2023.10248068
• 发表时间: 2023
• 期刊: IEEE Workshop on Applications of Signal Processing to Audio and Acoustics
• 作者: DiPassio, Tre;Heilemann, Michael C.;Thompson, Benjamin;Bocko, Mark F.
• 通讯作者: Bocko, Mark F.

Direction of arrival estimation of an acoustic wave using a single structural vibration sensor

使用单个结构振动传感器估计声波的到达方向

• DOI: 10.1016/j.jsv.2023.117671
• 发表时间: 2023
• 期刊: Journal of Sound and Vibration
• 影响因子: 4.7
• 作者: DiPassio, Tre;Heilemann, Michael C.;Bocko, Mark F.
• 通讯作者: Bocko, Mark F.

Audio Capture Using Piezoelectric Sensors on Vibrating Panel Surfaces

使用振动面板表面上的压电传感器进行音频捕获

• 发表时间: 2023
• 期刊: 154th Convention of the Audio Engineering Society
• 作者: DiPassio, Tre;Heilemann, Michael C.;Thompson, Benjamin;Bocko, Mark F.
• 通讯作者: Bocko, Mark F.

Estimating Acoustic Direction of Arrival Using a Single Structural Sensor on a Resonant Surface

使用谐振表面上的单个结构传感器估计声波到达方向

• DOI: 10.1109/icassp49357.2023.10095986
• 发表时间: 2023
• 期刊: Speech and Signal Processing (ICASSP
• 作者: DiPassio, Tre;Heilemann, Michael C.;Thompson, Benjamin;Bocko, Mark F.
• 通讯作者: Bocko, Mark F.