CAREER: Understanding Human Movement and Haptic Interaction toward Cognitive Augmentation Aids

职业：了解人体运动和触觉交互以实现认知增强辅助

• 批准号: 2142774
• 负责人: Troy McDaniel
• 金额: $ 56.12万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-07-01 至 2027-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2142774&HistoricalAwards=false
• 关键词: CAREER; Understanding; Human; Movement; Haptic

Humans interact with their immediate surroundings largely using their hands; from feeding oneself, to taking medication, to opening doors, humans manipulate many objects each day as part of the daily activities they wish to perform. Understanding what the hands are doing may provide insight into what activities are being performed and how they are being carried out. The most direct, and potentially clearest, view of the hands may be obtained through cameras that provide a hand-centric view, that is, wrist-worn cameras pointed toward the hands and fingers. To-date, such camera placements have seen little exploration, yet such a view has been demonstrated to capture the details of haptic interactions. Combined with advancements in computing, particularly machine learning and computer vision, such technologies may have the potential to become cognitive augmentation aids: assistive technologies to support attention, memory, and decision-making across a wide range of applications including assistive aids for seniors, which is the focus of this award. The outcomes of this project will impact principal disciplines of computer science and engineering as well as provide transformative innovations in haptics, machine learning, computer vision, wearable computing, and gerontechnology, particularly assistive technologies for seniors who desire to age in place safely and independently for longer. Investigations into the hardware, software, algorithms, and utility of intelligent, hand-centric wrist wearables will produce impactful results, not only for gerontechnology, but for a wide variety of fields, such as neurorehabilitation, virtual/mixed reality technologies, and smart industrial applications. The education activities undertaken as part of this project will include the development of a textbook on haptics and a project-oriented haptics course development.The goal of this project is to develop methods and technologies for understanding haptic interactions and human activities during senior activities of daily living (ADLs) using hand-centric wrist wearables, fused with sensors such as video cameras and inertial measurement units. The research hypothesis is that a wrist wearable using a hand-centric camera placement will simplify automatic analysis, recognition, recall, and prediction of hand-object interactions and human activities during senior ADLs. The activities of this project include determining the hardware design trade-offs for hand-centric wrist wearables and developing an annotated multi-camera hand-centric video dataset; determining algorithms and software for understanding haptic interactions and senior ADLs; and determining hardware and software integration toward the deployment of a prototype to investigate the impact and utility of intelligent, hand-centric wrist wearables for cognitive augmentation during senior ADLs. Hand-centric views for vision-based applications have seen little exploration, creating gaps in the knowledge related to the hardware, software, algorithms, and utility of this viewpoint to support cognitive tasks such as attention, memory, and decision-making. While this project focuses on gerontechnology for seniors, findings may propagate to many other fields including occupational and physical rehabilitation and compliance monitoring for smart industrial applications. Specific contributions include: (i) a better understanding of the hardware design tradeoffs that provide the most informative view of what the hands are doing; (ii) identification of the challenges of hand-centric views, and how to address them; (iii) development of a multi-view hand-centric video dataset; (iv) algorithms and software for robust and automated analysis, recognition, recall, and prediction of haptic interactions and senior ADLs. 3DCNNs (3-Dimensional Convoluted Neural Networks) will be developed to model and predict both hand-centric actions and objects being manipulated – i.e., the haptic interactions with hand; and (v) a better understanding of the impact and utility of intelligent, hand-centric wrist wearables on ADLs of seniors living independently, and more broadly, on their quality of life.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人类主要通过双手与周围环境进行互动。从吃饭、吃药到开门，人类每天都会操纵许多物体，作为他们希望进行的日常活动的一部分。了解双手在做什么可以让我们深入了解正在执行哪些活动以及如何进行这些活动。最直接且可能最清晰的手部视图可以通过提供以手部为中心的视图的摄像机来获得，即，腕戴式摄像机指向手和手指。迄今为止，这种相机放置方式还没有什么探索，但这种视图已被证明可以捕捉触觉交互的细节。结合计算技术的进步，特别是机器学习和计算机视觉，此类技术可能有潜力成为认知增强辅助工具：在广泛的应用中支持注意力、记忆和决策的辅助技术，包括针对老年人的辅助工具，这是该奖项的重点。该项目的成果将影响计算机科学和工程的主要学科，并在触觉、机器学习、计算机视觉、可穿戴计算和老年技术方面提供变革性创新，特别是为那些希望安全、独立地生活更长时间的老年人提供辅助技术。对智能、以手为中心的腕部可穿戴设备的硬件、软件、算法和实用性的研究将产生有影响力的结果，不仅对老年技术，而且对神经康复、虚拟/混合现实技术和智能工业应用等广泛领域也适用。作为该项目的一部分进行的教育活动将包括编写触觉教科书和开发面向项目的触觉课程。该项目的目标是开发方法和技术，以使用以手为中心的腕部可穿戴设备（与摄像机和惯性测量单元等传感器融合）来理解高级日常生活活动（ADL）期间的触觉交互和人类活动。研究假设是，使用以手为中心的摄像头放置的腕部可穿戴设备将简化高级 ADL 期间手部物体交互和人类活动的自动分析、识别、回忆和预测。该项目的活动包括确定以手为中心的腕部可穿戴设备的硬件设计权衡，以及开发带注释的多摄像头以手为中心的视频数据集；确定用于理解触觉交互和高级 ADL 的算法和软件；确定硬件和软件集成以部署原型，以研究智能、以手为中心的腕部可穿戴设备对高级 ADL 期间认知增强的影响和效用。基于视觉的应用程序以手为中心的视图几乎没有被探索过，这在与硬件、软件、算法以及支持注意力、记忆和决策等认知任务的实用性相关的知识方面造成了空白。虽然该项目侧重于老年人的老年技术，但研究结果可能会传播到许多其他领域，包括职业和身体康复以及智能工业应用的合规性监控。具体贡献包括：（i）更好地理解硬件设计权衡，为双手正在做什么提供最丰富的信息； (ii) 确定以手为中心的观点的挑战以及如何解决这些挑战； (iii) 开发以手部为中心的多视图视频数据集； (iv) 用于触觉交互和高级 ADL 的稳健和自动化分析、识别、回忆和预测的算法和软件。将开发 3DCNN（3 维卷积神经网络）来建模和预测以手为中心的动作和被操纵的物体，即与手的触觉交互； (v) 更好地了解智能、以手为中心的手腕可穿戴设备对独立生活的老年人的日常生活能力以及更广泛的生活质量的影响和效用。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。