CAREER: Towards sensing and understanding fine-grained body postures in daily life using intelligent wearables with acoustic sensing

尽管每天有10亿台可穿戴设备在使用，但计算机在日常环境中识别人类活动的能力仍然有限。一个关键的障碍是可穿戴设备无法感知与行为相关的信息，如肢体姿势（手和脚）或面部表情等细粒度细节。该项目将使下一代可穿戴设备能够持续跟踪和解释一组关键的细粒度身体姿势（例如，面部、手部、四肢、眼睛、舌头）的声学传感器。该项目的产出将帮助研究人员和开发人员监测和利用日常环境中的一系列高分辨率数据，以显着提高下游应用程序在具有积极社会影响的领域的性能，包括无障碍环境，远程医疗和活动识别。作为示范，该项目将使用新的可穿戴设备，通过推进美国手语（ASL）和无声语音识别，立即改善聋人和听力障碍者以及语言障碍者的计算机可访问性。 为了达到预期的目标，研究将采用迭代和以用户为中心的设计过程。首先，将开发一系列使用声学传感来持续跟踪细粒度姿势的AI驱动的可穿戴设备。这些可穿戴设备将在实验室和现实环境中进行广泛和迭代的评估，以确保最佳的用户体验和性能，并确定任何剩余的挑战。接下来，该研究将通过使用定制的信号处理和人工智能算法（如数据模拟和合成，数据增强和边缘计算）来解决在日常环境中部署这些数据驱动的声学传感技术的关键挑战，以提高系统在用户中的通用性及其在噪声存在下的鲁棒性，并在保护用户隐私的同时最大限度地减少培训工作。第三，研究将展示这些新的可穿戴设备如何增强计算机理解复杂人类行为的能力，这将自然地支持两个高影响力下游应用的用户：ASL和无声语音识别。在整个设计、开发、评估过程中，将与相关领域的专家合作开展工作（包括可穿戴计算、人工智能、语言学和耳鼻喉科），并与目标社区的合作伙伴（包括失聪和听力有困难的人，和有语言障碍的人）该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Despite the one billion wearables in use every day, computers' ability to recognize human activities in daily settings is still limited. One key roadblock is the inability of wearables to sense behavior-relevant information such as limb posture (for both hands and feet) or fine-grained details such as facial expressions. This project will enable the next generation of wearables to continuously track and interpret a key set of fine-grained body postures (e.g., face, hands, limbs, eyes, tongue) in daily life using low-power, low-cost, and privacy-sensitive intelligent acoustic sensing technologies. The project outputs will help researchers and developers monitor and exploit a range of high-resolution data in everyday settings to significantly improve the performance of downstream applications in areas with positive societal impact, including accessibility, telemedicine, and activity recognition. As a demonstration, this project will use the new wearables to immediately improve the accessibility of computers for deaf and hard-of-hearing individuals as well as people with speech impairments, by advancing American Sign Language (ASL) and Silent Speech recognition. To achieve the desired goals, the research will employ an iterative and user-centered design process. First, a list of AI-powered wearables that use acoustic sensing to continuously track fine-grained postures will be developed. These wearables will be evaluated extensively and iteratively in both lab and real-world settings to ensure optimal user experience and performance, and to identify any remaining challenges. Next, the research will address the critical challenges of deploying these data-driven acoustic sensing technologies in everyday settings by using customized signal processing and AI algorithms (such as data simulation and synthesis, data augmentation, and edge computing) to improve the system's generalizability across users and its robustness in the presence of noise, and to minimize training efforts while protecting user privacy. Thirdly, the research will demonstrate how these new wearables can enhance computers' ability to understand complex human behaviors, which will naturally support users in two high-impact downstream applications: ASL and silent speech recognition. Throughout the design, development, and evaluation process, the work will be carried out in collaboration with experts in related fields (including wearable computing, AI, linguistics, and otorhinolaryngology), and with partners in the target community (including persons who are deaf and hard-of-hearing, and those with speech impairments).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.