RI: Small: Enabling Sound-based Human Activity Monitoring for Home Service Robots

RI：小型：为家庭服务机器人提供基于声音的人体活动监控

• 批准号: 1910993
• 负责人: Weihua Sheng
• 金额: $ 48.87万
• 依托单位: Oklahoma State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910993&HistoricalAwards=false
• 关键词: RI; Small; Enabling; Sound; based

The increasing demand for in-home elderly care involves challenges that call for innovative solutions. As more older adults prefer to live in their own homes as they age, living alone may pose serious risks to those who have age-related problems such as reduced mobility, dementia, or other chronic diseases. This at-risk population needs regular visits from in-home healthcare services, which in turn creates pressure on the geriatric home healthcare industry. Home service robots offer a solution to this societal problem by facilitating smart aging-in-place. This project aims to solve a fundamental research problem critical to the application of service robots in complex home environments: human activity monitoring. By creating a bridge between environmental understanding and human behavior understanding, this project offers a new theory to realize sound-based monitoring of resident behaviors in realistic home environments. Such a human-aware capability frees home service robots to do their daily routine work, while being able to care for the resident more proactively and effectively. Sound-based human behavior understanding will greatly improve the capability and usability of home service robots, therefore accelerating their adoption in human daily life. This project also incorporates education and outreach activities to stimulate prospective and current college students to pursue degrees and careers in science and engineering, attract underrepresented minority students to these research activities, and to disseminate new, useful datasets to the research community and home healthcare industry to promote continued advances in this area. This project investigates a new theoretical framework for human activity monitoring in home environments, which takes advantage of deep learning while considering the locational context, thereby greatly improving the accuracy of human behavior understanding. The target framework is intended for broader application to similar deep learning-based machine perception problems. The project aims to establish a novel visual-acoustic semantic map (VASM) to connect environmental understanding and behavior understanding. Constructed through robotic semantic mapping and voice-based human-robot interaction, the VASM concept extends traditional visual semantic maps by incorporating rich acoustic information in the environment. When cloud-connected and scaled up to a large number of robots, this approach is expected to provide an effective and distributed solution to constructing a large dataset with annotated home event sounds. That dataset will then be used to train deep neural networks for sound event recognition. The project also develops a multi-sensor fusion approach to combining sound data with distributed motion sensor data to solve the problem of human activity recognition without using visual sensors. Such an approach overcomes the shortcomings associated with vision sensors and offers a fundamentally different solution to human activity monitoring. Finally, the planned theoretical framework will be verified and evaluated through experiments in a robot-integrated smart home.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

日益增长的居家养老需求带来了挑战，需要创新的解决方案。随着年龄的增长，越来越多的老年人喜欢住在自己的家里，独居可能会给那些有与年龄相关的问题的人带来严重的风险，如行动不便、痴呆症或其他慢性病。这些高危人群需要家庭医疗服务的定期访问，这反过来又给老年家庭医疗行业带来了压力。家庭服务机器人通过促进智能就地老龄化为这一社会问题提供了解决方案。该项目旨在解决对服务机器人在复杂家庭环境中的应用至关重要的一个基础研究问题：人类活动监测。通过在环境理解和人类行为理解之间架起一座桥梁，该项目为实现对现实家庭环境中居民行为的声音监测提供了新的理论。这种人类感知的能力使家庭服务机器人能够自由地进行日常工作，同时能够更主动、更有效地照顾居民。基于声音的人类行为理解将极大地提高家庭服务机器人的能力和可用性，从而加速其在人类日常生活中的应用。该项目还包括教育和推广活动，以激励未来和现有的大学生在科学和工程领域攻读学位和职业，吸引未被充分代表的少数族裔学生参与这些研究活动，并向研究界和家庭保健行业传播新的有用的数据集，以促进这一领域的持续进步。本项目研究了一种新的家庭环境中人类活动监测的理论框架，该框架利用深度学习的同时考虑位置上下文，从而极大地提高了人类行为理解的准确性。目标框架旨在更广泛地应用于类似的基于深度学习的机器感知问题。该项目旨在建立一种新的视听语义图(VASM)来连接环境理解和行为理解。VASM概念通过机器人语义映射和基于语音的人-机器人交互构建，通过在环境中整合丰富的声学信息来扩展传统的视觉语义地图。当云连接并扩展到大量机器人时，该方法有望为构建带有注释的家庭事件声音的大型数据集提供有效的分布式解决方案。然后，该数据集将被用于训练用于声音事件识别的深层神经网络。该项目还开发了一种将声音数据与分布式运动传感器数据相结合的多传感器融合方法，以解决不使用视觉传感器的人类活动识别问题。这种方法克服了视觉传感器的缺点，为人类活动监测提供了一种根本不同的解决方案。最后，计划中的理论框架将通过在机器人集成的智能家居中进行实验来验证和评估。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Addressing the Role of Smart Robotic Health Assistants Within the Human-Machine Frontier of Geriatric Healthcare

解决智能机器人健康助手在老年医疗保健人类机器边界内的作用

• DOI: 10.1093/geroni/igaa057.1319
• 发表时间: 2020-12-16
• 期刊: Innovation in Aging
• 影响因子: 7
• 作者: Firdausya N;Bishop A;Carlson B;Sheng W
• 通讯作者: Sheng W

Conversation-Based Medication Management System for Older Adults Using a Companion Robot and Cloud

使用伴侣机器人和云的老年人基于对话的药物管理系统

• DOI: 10.1109/lra.2021.3061996
• 发表时间: 2021
• 期刊: IEEE Robotics and Automation Letters
• 影响因子: 5.2
• 作者: Su, Zhidong;Liang, Fei;Do, Ha Manh;Bishop, Alex;Carlson, Barbara;Sheng, Weihua
• 通讯作者: Sheng, Weihua

SoHAM: A Sound-Based Human Activity Monitoring Framework for Home Service Robots

• DOI: 10.1109/tase.2021.3081406
• 发表时间: 2021-05-28
• 期刊: IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING
• 影响因子: 5.6
• 作者: Do, Ha Manh;Welch, Karla Conn;Sheng, Weihua
• 通讯作者: Sheng, Weihua

Energy Consumption in a Collaborative Activity Monitoring System using a Companion Robot and a Wearable Device

使用伴侣机器人和可穿戴设备的协作活动监控系统的能耗

• 发表时间: 2021
• 期刊: and Intelligent Systems
• 作者: Liang, Fei;Hernandez, Ricardo;Sheng, Weihua;Gu, Ye
• 通讯作者: Gu, Ye

Online Manhattan Keyframe-based Dense Reconstruction from Indoor Depth Sequences

基于室内深度序列的在线曼哈顿关键帧密集重建

• DOI: 10.1109/vcip47243.2019.8965937
• 发表时间: 2019
• 期刊: 2019 IEEE International Conference on Visual Communications and Image Processing (VCIP
• 作者: Yazdanpour, Mahdi;Fan, Guoliang;Sheng, Weihua
• 通讯作者: Sheng, Weihua