CAREER: Understanding Human Movement and Haptic Interaction toward Cognitive Augmentation Aids

人类在很大程度上是用手与周围环境互动的；从自己吃饭，到服药，再到开门，人类每天都会操纵许多物体，这是他们希望进行的日常活动的一部分。了解手在做什么可以提供对正在执行什么活动以及如何执行这些活动的洞察。通过提供以手为中心的视角的摄像头，即戴在手腕上的摄像头指向手和手指，可以获得最直接、也可能是最清晰的手的图像。到目前为止，这样的摄像头放置很少被探索，但这样的视角已经被证明捕捉到了触觉交互的细节。结合计算的进步，特别是机器学习和计算机视觉，这些技术可能有潜力成为认知增强辅助：支持广泛应用的注意力、记忆和决策的辅助技术，包括为老年人提供的辅助辅助，这是本奖项的重点。该项目的成果将影响计算机科学和工程的主要学科，并在触觉、机器学习、计算机视觉、可穿戴计算和老年技术方面提供变革性创新，特别是为希望安全和独立生活更长时间的老年人提供辅助技术。对以手为中心的智能手腕可穿戴设备的硬件、软件、算法和实用性的研究将产生影响的结果，不仅对老年技术，而且对广泛的领域，如神经康复、虚拟/混合现实技术和智能工业应用。作为该项目的一部分，开展的教育活动将包括开发一本触觉教科书和一门面向项目的触觉课程开发。该项目的目标是开发方法和技术，利用手腕可穿戴设备，与摄像机和惯性测量单元等传感器融合，在日常生活的高级活动(ADL)中了解触觉交互和人类活动。研究假设是，使用以手为中心的摄像头放置的手腕可穿戴设备将简化高级ADL期间手-物体交互和人类活动的自动分析、识别、回忆和预测。该项目的活动包括确定以手为中心的手腕可穿戴设备的硬件设计权衡，并开发带注释的多摄像头手部为中心的视频数据集；确定用于理解触觉交互和高级ADL的算法和软件；以及确定用于部署原型的硬件和软件集成，以调查智能以手为中心的手腕可穿戴设备在高级ADL期间对认知增强的影响和用途。基于视觉的应用程序的以手为中心的视图几乎没有进行过探索，在与硬件、软件、算法和该视点用于支持注意力、记忆和决策等认知任务的实用程序相关的知识中造成了空白。虽然该项目的重点是老年人的老年技术，但研究成果可能会传播到许多其他领域，包括职业和身体康复以及智能工业应用的遵从性监测。具体贡献包括：(I)更好地理解硬件设计的权衡取舍，提供关于手在做什么的最丰富信息；(Ii)确定以手为中心的视图面临的挑战，以及如何解决这些挑战；(Iii)开发多视点以手为中心的视频数据集；(Iv)用于对触觉交互和高级ADL进行健壮、自动化分析、识别、回忆和预测的算法和软件。3DCNN(三维卷积神经网络)将被开发来建模和预测以手为中心的动作和被操纵的物体--即与手的触觉交互；以及(V)更好地了解智能、以手为中心的手腕可穿戴设备对独立生活的老年人的ADL的影响和效用，以及更广泛地对他们的生活质量的影响。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Humans interact with their immediate surroundings largely using their hands; from feeding oneself, to taking medication, to opening doors, humans manipulate many objects each day as part of the daily activities they wish to perform. Understanding what the hands are doing may provide insight into what activities are being performed and how they are being carried out. The most direct, and potentially clearest, view of the hands may be obtained through cameras that provide a hand-centric view, that is, wrist-worn cameras pointed toward the hands and fingers. To-date, such camera placements have seen little exploration, yet such a view has been demonstrated to capture the details of haptic interactions. Combined with advancements in computing, particularly machine learning and computer vision, such technologies may have the potential to become cognitive augmentation aids: assistive technologies to support attention, memory, and decision-making across a wide range of applications including assistive aids for seniors, which is the focus of this award. The outcomes of this project will impact principal disciplines of computer science and engineering as well as provide transformative innovations in haptics, machine learning, computer vision, wearable computing, and gerontechnology, particularly assistive technologies for seniors who desire to age in place safely and independently for longer. Investigations into the hardware, software, algorithms, and utility of intelligent, hand-centric wrist wearables will produce impactful results, not only for gerontechnology, but for a wide variety of fields, such as neurorehabilitation, virtual/mixed reality technologies, and smart industrial applications. The education activities undertaken as part of this project will include the development of a textbook on haptics and a project-oriented haptics course development.The goal of this project is to develop methods and technologies for understanding haptic interactions and human activities during senior activities of daily living (ADLs) using hand-centric wrist wearables, fused with sensors such as video cameras and inertial measurement units. The research hypothesis is that a wrist wearable using a hand-centric camera placement will simplify automatic analysis, recognition, recall, and prediction of hand-object interactions and human activities during senior ADLs. The activities of this project include determining the hardware design trade-offs for hand-centric wrist wearables and developing an annotated multi-camera hand-centric video dataset; determining algorithms and software for understanding haptic interactions and senior ADLs; and determining hardware and software integration toward the deployment of a prototype to investigate the impact and utility of intelligent, hand-centric wrist wearables for cognitive augmentation during senior ADLs. Hand-centric views for vision-based applications have seen little exploration, creating gaps in the knowledge related to the hardware, software, algorithms, and utility of this viewpoint to support cognitive tasks such as attention, memory, and decision-making. While this project focuses on gerontechnology for seniors, findings may propagate to many other fields including occupational and physical rehabilitation and compliance monitoring for smart industrial applications. Specific contributions include: (i) a better understanding of the hardware design tradeoffs that provide the most informative view of what the hands are doing; (ii) identification of the challenges of hand-centric views, and how to address them; (iii) development of a multi-view hand-centric video dataset; (iv) algorithms and software for robust and automated analysis, recognition, recall, and prediction of haptic interactions and senior ADLs. 3DCNNs (3-Dimensional Convoluted Neural Networks) will be developed to model and predict both hand-centric actions and objects being manipulated – i.e., the haptic interactions with hand; and (v) a better understanding of the impact and utility of intelligent, hand-centric wrist wearables on ADLs of seniors living independently, and more broadly, on their quality of life.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.