CRII : HCC : Smart Acoustic Surfaces as Multimodal Interfaces

在过去的十年中，智能设备已经成为日常生活的重要组成部分。然而，除了会话助理之外，大多数智能设备和环境主要依赖于视觉上呈现信息，这对于用户的视觉注意力集中在设备之外的东西上的许多情况来说是不合适的。支持这些情况的一种方法是让人们使用声音或触摸进行交互。然而，使用传统方式添加单独的麦克风、扬声器和振动机构可能会增加额外的成本，并降低设备的耐用性和美观性。该项目旨在开发替代技术，通过在从智能手机到视频墙的平面上弯曲振动来记录和再现空间音频。这些振动还可以允许智能声学表面用作触摸界面。这种方法的优点在于，表面已经是设备的一部分，允许设备在结合这些新特征的同时保持其耐用性和美观性。将空间音频和触觉反馈添加到OLED显示器、智能手机和视频墙将提供基础技术，以提高人们导航菜单和地图等复杂数据集的能力，并为视频会议电话等远程应用提供更大的沉浸感。该项目开发了一个框架，用于设计表面，如设备的显示屏，以作为声学和振动触觉界面。研究目标将通过采用扩展表面的振动声学来解决以下三个挑战：（1）空间音频再现，（2）空间音频捕获，以及（3）触觉反馈和触摸感测。可以使用分布在表面上的传感器阵列来检测由与结构的声学和触摸交互引起的延伸表面的振动。类似地，分布在表面上的力致动器阵列将引起弯曲振动，从而导致结构辐射声音。扬声器和麦克风阵列处理方法将适用于振动致动器和传感器，其应用包括降噪、声场控制和波束形成。机器学习技术将用于识别来源于与结构的声学和触摸相互作用引起的振动模式的源位置特征。声学记录和扫描激光测振仪测量将用于分析该框架在现实世界中的有效性。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估。

Over the course of the last decade, smart devices have become essential components of daily life. However, apart from conversational assistants, most smart devices and environments rely primarily on presenting information visually, a poor fit for many situations where users' visual attention is on something besides the device. One way to support these situations is to let people use sound or touch for interactions. However, adding separate microphones, loudspeakers, and vibration mechanisms using conventional means can impose extra costs and reduce the device’s durability and aesthetics. This project seeks to develop alternative technologies for recording and reproducing spatial audio through bending vibrations on flat surfaces ranging from smartphones to video walls. These vibrations can also allow the smart acoustic surface to serve as a touch interface. The advantage of this approach lies in the fact that the surface is already part of the device, allowing the device to maintain its durability and aesthetics while incorporating these new features. Adding spatial audio and haptic feedback to OLED displays, smartphones, and video walls will provide fundamental technology to improve people's ability to navigate complex data sets such as menus and maps, and enable a greater sense of immersion for remote applications such as video conference calls. The project develops a framework for designing surfaces such as the display screen of a device to serve as acoustic and vibrotactile interfaces. The research objectives will address the following three challenges by employing the vibro-acoustics of extended surfaces: (1) spatial audio reproduction, (2) spatial audio capture, and (3) haptic feedback and touch sensing. The vibrations of the extended surface induced by acoustic and touch interactions with the structure can be detected using arrays of sensors distributed on the surface. Similarly, arrays of force actuators distributed on the surface will induce bending vibrations that cause the structure to radiate sound. Loudspeaker and microphone array processing methods will be adapted for the vibration actuators and sensors with applications including noise reduction, sound-field control, and beamforming. Machine learning techniques will be used to identify source-location features stemming from vibration patterns induced by acoustic and touch interactions with the structure. Acoustic recordings and scanning laser vibrometer measurements will be used to analyze the effectiveness of the framework in real-world settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.