CAREER: Towards sensing and understanding fine-grained body postures in daily life using intelligent wearables with acoustic sensing

职业：利用具有声学传感功能的智能可穿戴设备来感知和理解日常生活中细粒度的身体姿势

• 批准号: 2239569
• 负责人: Cheng Zhang
• 金额: $ 66.74万
• 依托单位: Cornell University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-03-01 至 2028-02-29
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2239569&HistoricalAwards=false
• 关键词: CAREER; Towards; sensing; understanding; fine

Despite the one billion wearables in use every day, computers' ability to recognize human activities in daily settings is still limited. One key roadblock is the inability of wearables to sense behavior-relevant information such as limb posture (for both hands and feet) or fine-grained details such as facial expressions. This project will enable the next generation of wearables to continuously track and interpret a key set of fine-grained body postures (e.g., face, hands, limbs, eyes, tongue) in daily life using low-power, low-cost, and privacy-sensitive intelligent acoustic sensing technologies. The project outputs will help researchers and developers monitor and exploit a range of high-resolution data in everyday settings to significantly improve the performance of downstream applications in areas with positive societal impact, including accessibility, telemedicine, and activity recognition. As a demonstration, this project will use the new wearables to immediately improve the accessibility of computers for deaf and hard-of-hearing individuals as well as people with speech impairments, by advancing American Sign Language (ASL) and Silent Speech recognition. To achieve the desired goals, the research will employ an iterative and user-centered design process. First, a list of AI-powered wearables that use acoustic sensing to continuously track fine-grained postures will be developed. These wearables will be evaluated extensively and iteratively in both lab and real-world settings to ensure optimal user experience and performance, and to identify any remaining challenges. Next, the research will address the critical challenges of deploying these data-driven acoustic sensing technologies in everyday settings by using customized signal processing and AI algorithms (such as data simulation and synthesis, data augmentation, and edge computing) to improve the system's generalizability across users and its robustness in the presence of noise, and to minimize training efforts while protecting user privacy. Thirdly, the research will demonstrate how these new wearables can enhance computers' ability to understand complex human behaviors, which will naturally support users in two high-impact downstream applications: ASL and silent speech recognition. Throughout the design, development, and evaluation process, the work will be carried out in collaboration with experts in related fields (including wearable computing, AI, linguistics, and otorhinolaryngology), and with partners in the target community (including persons who are deaf and hard-of-hearing, and those with speech impairments).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管每天使用十亿可穿戴设备，但计算机在日常设置中识别人类活动的能力仍然有限。一个关键的障碍是无法可穿戴设备来感知与行为相关的信息，例如肢体姿势（无论是手脚还是脚）或细粒细节，例如面部表情。该项目将使下一代的可穿戴设备能够在日常生活中连续跟踪和解释一套关键的精细颗粒身体姿势（例如，面部，手，四肢，眼睛，舌头），使用低功率，低成本和对隐私敏感的智能声学传感技术。该项目的输出将帮助研究人员和开发人员在日常环境中监视和利用一系列高分辨率数据，以显着提高社会影响积极的地区的下游应用程序的性能，包括可访问性，远程医疗和活动认识。作为演示，该项目将使用新的可穿戴设备立即提高聋哑人和听力障碍的人以及具有言语障碍的人的可访问性，通过推进美国手语（ASL）和无声的语音识别。 为了实现所需的目标，研究将采用迭代和以用户为中心的设计过程。首先，将开发出使用声感应来连续跟踪细粒姿势的AI驱动的可穿戴设备列表。这些可穿戴设备将在实验室和现实世界中进行广泛和迭代的评估，以确保最佳的用户体验和性能，并确定剩余的挑战。接下来，这项研究将通过使用自定义的信号处理和AI算法（例如数据模拟和合成，数据增强和边缘计算）来解决日常设置中部署这些数据驱动的声学传感技术的关键挑战，以改善系统在噪音中的稳健性及其在噪音中的稳健性，并在噪音的范围内进行培训，并在降低噪音的情况下进行培训。第三，研究将证明这些新可穿戴设备如何增强计算机理解复杂的人类行为的能力，这自然会在两个高影响力的下游应用程序中为用户提供支持：ASL和无声的语音识别。在整个设计，开发和评估过程中，将与相关领域的专家合作（包括可穿戴计算，AI，语言学和耳鼻喉科学）以及目标社区中的伙伴（包括聋哑人（包括聋哑人）的人（包括障碍和演讲障碍的人）。优点和更广泛的影响审查标准。

项目成果

EchoNose: Sensing Mouth, Breathing and Tongue Gestures inside Oral Cavity using a Non-contact Nose Interface

EchoNose：使用非接触式鼻子接口感应口腔内的口腔、呼吸和舌头手势

• DOI: 10.1145/3594738.3611358
• 发表时间: 2023
• 期刊: The ACM International Symposium on Wearable Computing (ISWC
• 作者: Sun, Rujia;Zhou, Xiaohe;Steeper, Benjamin;Zhang, Ruidong;Yin, Sicheng;Li, Ke;Wu, Shengzhang;Tilsen, Sam;Guimbretiere, Francois;Zhang, Cheng
• 通讯作者: Zhang, Cheng

HPSpeech: Silent Speech Interface for Commodity Headphones

HPSpeech：商品耳机的无声语音接口

• DOI: 10.1145/3594738.3611365
• 发表时间: 2023
• 期刊: The ACM International Symposium on Wearable Computing (ISWC'23
• 作者: Zhang, Ruidong;Chen, Hao;Agarwal, Devansh;Jin, Richard;Li, Ke;Guimbretière, François;Zhang, Cheng
• 通讯作者: Zhang, Cheng

C-Auth: Exploring the Feasibility of Using Egocentric View of Face Contour for User Authentication on Glasses

• DOI: 10.1145/3594738.3611355
• 发表时间: 2023-10
• 期刊: Proceedings of the 2023 ACM International Symposium on Wearable Computers
• 作者: Hyunchul Lim;Guilin Hu;Richard Jin;Hao Chen;Ryan Mao;Ruidong Zhang;Cheng Zhang