CRII: HCC: 3D Hand & Full-Body Pose Estimation in Telehealth for Children with Autism

该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。该项目的总体目标是在不使用头盔显示器的虚拟现实系统中提供高效的全身交互。该项目旨在创建准确和实时的3D手和身体姿势估计，在自闭症儿童的非常重要的应用领域。一种新的合成手部数据生成框架将生成3D手部姿势，其在与相机的手距离、手的大小、相机的视点、遮挡、背景和肤色方面具有更多的多样性。结果将是一个新的3D合成手部数据集，包括真实的和运动学上准确的具有关节姿势的手部模型，这将推动当前和未来在3D手部姿势估计研究方面的研究努力。该项目将推进最先进的3D人体姿势估计，让人们在房间尺度的距离内远离摄像机。该合成数据集、算法和编程库将公开供广泛采用，从而推动姿势估计研究。这项研究将产生广泛的社会影响，因为它将改善以人类为中心的远程医疗应用的可用性和互动性，初步有助于自闭症儿童的应用行为分析。现有的使用头盔显示器或可穿戴传感器来跟踪用户的手和身体运动的系统不适合自闭症儿童，并且在许多其他应用领域存在劣势。因此，通过实现3D手部和全身姿势估计，该项目将推动大量3D沉浸式应用，如教育、虚拟STEM实验室、远程康复、远程操作、军事训练、娱乐和通信。考虑到最近的混乱导致虚拟活动的增加，对实时、远程和交互式人体运动感知的需求比以往任何时候都更加迫切。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). The overall objective of this project is to provide efficient full-body interaction in virtual reality systems that do not use head-mounted displays. This project aims to create accurate and real-time 3D hand and body pose estimation, in the highly significant application area of children with autism. A novel synthetic hand data generation framework will generate 3D hand poses with increased diversity in terms of hand distance from camera, hand size, camera viewpoint, occlusion, background, and skin color. The outcome will be a novel 3D synthetic hand dataset consisting of realistic and kinematically accurate hand models with articulated poses that will advance current and future research endeavors in 3D hand pose estimation research. The project will advance the state-of-the-art in 3D body pose estimation for humans present further away from the camera at room-scale distances. The synthetic dataset, algorithms, and programming libraries will be made publicly available for wide-spread adoption, thereby advancing pose estimation research.This research will have broad societal impact because it will improve the usability and interaction in human centered telehealth applications, initially helping with the applied behavior analysis for children with autism. Existing systems that employ head-mounted displays or wearable sensors for tracking the user's hand and body movements are not suitable for children with autism, and have disadvantages in many other application areas. Therefore, by enabling 3D hand and full-body pose estimation, this project will advance a plethora of 3D immersive applications such as education, virtual STEM laboratories, tele-rehabilitation, tele-operation, military training, entertainment, and communication. The need for real-time, remote and interactive human motion sensing exists now more than ever, considering the increase in virtual activities because of the recent pandemic.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.