CAREER: Towards sensing and understanding fine-grained body postures in daily life using intelligent wearables with acoustic sensing

尽管每天有10亿台可穿戴设备在使用，但计算机在日常环境中识别人类活动的能力仍然有限。一个关键的障碍是，可穿戴设备无法感知与行为相关的信息，如肢体姿势（手和脚）或面部表情等细微细节。该项目将使下一代可穿戴设备能够使用低功耗、低成本、隐私敏感的智能声学传感技术，在日常生活中持续跟踪和解读一组关键的精细身体姿势（如面部、手部、四肢、眼睛、舌头）。该项目的产出将帮助研究人员和开发人员监测和利用日常环境中的一系列高分辨率数据，以显著改善具有积极社会影响的下游应用领域的性能，包括可访问性、远程医疗和活动识别。作为示范，该项目将使用新的可穿戴设备，通过推进美国手语（ASL）和无声语音识别，立即改善聋人、听力障碍者和语言障碍人士使用电脑的可及性。为了达到预期的目标，研究将采用迭代和以用户为中心的设计过程。首先，将开发一系列人工智能可穿戴设备，这些设备使用声学传感来持续跟踪精细的姿势。这些可穿戴设备将在实验室和现实环境中进行广泛和迭代的评估，以确保最佳的用户体验和性能，并确定任何遗留的挑战。接下来，该研究将通过使用定制的信号处理和人工智能算法（如数据模拟和合成、数据增强和边缘计算）来解决在日常环境中部署这些数据驱动的声学传感技术的关键挑战，以提高系统在用户中的通用性及其在噪声存在下的鲁棒性，并在保护用户隐私的同时最大限度地减少培训工作量。第三，该研究将展示这些新的可穿戴设备如何增强计算机理解复杂人类行为的能力，这将自然地支持用户在两个高影响力的下游应用：美国手语和无声语音识别。在整个设计、开发和评估过程中，这项工作将与相关领域的专家（包括可穿戴计算、人工智能、语言学和耳鼻喉科）以及目标社区的合作伙伴（包括耳聋和听力障碍人士以及有语言障碍的人）合作进行。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Despite the one billion wearables in use every day, computers' ability to recognize human activities in daily settings is still limited. One key roadblock is the inability of wearables to sense behavior-relevant information such as limb posture (for both hands and feet) or fine-grained details such as facial expressions. This project will enable the next generation of wearables to continuously track and interpret a key set of fine-grained body postures (e.g., face, hands, limbs, eyes, tongue) in daily life using low-power, low-cost, and privacy-sensitive intelligent acoustic sensing technologies. The project outputs will help researchers and developers monitor and exploit a range of high-resolution data in everyday settings to significantly improve the performance of downstream applications in areas with positive societal impact, including accessibility, telemedicine, and activity recognition. As a demonstration, this project will use the new wearables to immediately improve the accessibility of computers for deaf and hard-of-hearing individuals as well as people with speech impairments, by advancing American Sign Language (ASL) and Silent Speech recognition. To achieve the desired goals, the research will employ an iterative and user-centered design process. First, a list of AI-powered wearables that use acoustic sensing to continuously track fine-grained postures will be developed. These wearables will be evaluated extensively and iteratively in both lab and real-world settings to ensure optimal user experience and performance, and to identify any remaining challenges. Next, the research will address the critical challenges of deploying these data-driven acoustic sensing technologies in everyday settings by using customized signal processing and AI algorithms (such as data simulation and synthesis, data augmentation, and edge computing) to improve the system's generalizability across users and its robustness in the presence of noise, and to minimize training efforts while protecting user privacy. Thirdly, the research will demonstrate how these new wearables can enhance computers' ability to understand complex human behaviors, which will naturally support users in two high-impact downstream applications: ASL and silent speech recognition. Throughout the design, development, and evaluation process, the work will be carried out in collaboration with experts in related fields (including wearable computing, AI, linguistics, and otorhinolaryngology), and with partners in the target community (including persons who are deaf and hard-of-hearing, and those with speech impairments).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.