RI: Small: Enabling Sound-based Human Activity Monitoring for Home Service Robots

居家养老需求的不断增长带来了挑战，需要创新的解决方案。随着年龄的增长，越来越多的老年人喜欢住在自己的家里，独居可能会给那些患有与年龄有关的问题（如行动不便、痴呆或其他慢性疾病）的人带来严重风险。这些高危人群需要定期接受家庭保健服务，这反过来又给老年家庭保健行业带来了压力。家庭服务机器人通过促进就地智能老龄化，为这一社会问题提供了解决方案。该项目旨在解决服务机器人在复杂家庭环境中应用的关键基础研究问题：人类活动监测。通过在环境理解和人类行为理解之间架起一座桥梁，本项目提供了一种新的理论来实现现实家庭环境中居民行为的基于声音的监测。这种人类感知能力使家庭服务机器人可以自由地完成日常工作，同时能够更主动、更有效地照顾居民。基于声音的人类行为理解将极大地提高家庭服务机器人的能力和可用性，从而加速它们在人类日常生活中的应用。该项目还包括教育和外展活动，以激励未来和当前的大学生攻读科学和工程学位和职业，吸引代表性不足的少数民族学生参加这些研究活动，并向研究界和家庭医疗保健行业传播新的有用的数据集，以促进该领域的持续进步。本项目研究了一个新的家庭环境中人类活动监测的理论框架，该框架在考虑位置背景的同时利用深度学习，从而大大提高了人类行为理解的准确性。目标框架旨在更广泛地应用于类似的基于深度学习的机器感知问题。该项目旨在建立一种新的视觉-声学语义图（VASM）来连接环境理解和行为理解。VASM概念通过机器人语义地图和基于语音的人机交互构建，通过融入环境中丰富的声学信息，扩展了传统的视觉语义地图。当云连接并扩展到大量机器人时，这种方法有望为构建带有注释的家庭事件声音的大型数据集提供有效的分布式解决方案。然后，该数据集将用于训练用于声音事件识别的深度神经网络。该项目还开发了一种多传感器融合方法，将声音数据与分布式运动传感器数据相结合，以解决无需使用视觉传感器即可识别人类活动的问题。这种方法克服了与视觉传感器相关的缺点，为人类活动监测提供了一种根本不同的解决方案。最后，规划的理论框架将通过机器人集成智能家居的实验进行验证和评估。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The increasing demand for in-home elderly care involves challenges that call for innovative solutions. As more older adults prefer to live in their own homes as they age, living alone may pose serious risks to those who have age-related problems such as reduced mobility, dementia, or other chronic diseases. This at-risk population needs regular visits from in-home healthcare services, which in turn creates pressure on the geriatric home healthcare industry. Home service robots offer a solution to this societal problem by facilitating smart aging-in-place. This project aims to solve a fundamental research problem critical to the application of service robots in complex home environments: human activity monitoring. By creating a bridge between environmental understanding and human behavior understanding, this project offers a new theory to realize sound-based monitoring of resident behaviors in realistic home environments. Such a human-aware capability frees home service robots to do their daily routine work, while being able to care for the resident more proactively and effectively. Sound-based human behavior understanding will greatly improve the capability and usability of home service robots, therefore accelerating their adoption in human daily life. This project also incorporates education and outreach activities to stimulate prospective and current college students to pursue degrees and careers in science and engineering, attract underrepresented minority students to these research activities, and to disseminate new, useful datasets to the research community and home healthcare industry to promote continued advances in this area. This project investigates a new theoretical framework for human activity monitoring in home environments, which takes advantage of deep learning while considering the locational context, thereby greatly improving the accuracy of human behavior understanding. The target framework is intended for broader application to similar deep learning-based machine perception problems. The project aims to establish a novel visual-acoustic semantic map (VASM) to connect environmental understanding and behavior understanding. Constructed through robotic semantic mapping and voice-based human-robot interaction, the VASM concept extends traditional visual semantic maps by incorporating rich acoustic information in the environment. When cloud-connected and scaled up to a large number of robots, this approach is expected to provide an effective and distributed solution to constructing a large dataset with annotated home event sounds. That dataset will then be used to train deep neural networks for sound event recognition. The project also develops a multi-sensor fusion approach to combining sound data with distributed motion sensor data to solve the problem of human activity recognition without using visual sensors. Such an approach overcomes the shortcomings associated with vision sensors and offers a fundamentally different solution to human activity monitoring. Finally, the planned theoretical framework will be verified and evaluated through experiments in a robot-integrated smart home.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.