Neuromorphic High-Frequency 3D Dancing Pose Estimation in Dynamic Environment
Neuromorphic High-Frequency 3D Dancing Pose Estimation in Dynamic Environment
复制标题
DOI：
10.1016/j.neucom.2023.126388
复制
发表时间：
2023-01
期刊：
影响因子：
6
通讯作者：
Zhongyang Zhang;Kaidong Chai;Haowen Yu;Ramzi M Majaj;Francesca Walsh;Edward Wang;U. Mahbub;H. Siegelmann;Donghyun Kim-;Tauhidur Rahman
中科院分区：
文献类型：
--
作者：
Zhongyang Zhang;Kaidong Chai;Haowen Yu;Ramzi M Majaj;Francesca Walsh;Edward Wang;U. Mahbub;H. Siegelmann;Donghyun Kim-;Tauhidur Rahman
以技术为媒介的舞蹈体验，作为一种娱乐媒介，是传统和虚拟现实游戏平台的关键元素。这些平台主要依赖于不引人注目的和连续的人体姿势估计作为捕获输入的手段。目前的解决方案主要采用RGB或RGB深度相机用于舞蹈游戏应用;然而，前者由于运动模糊和灵敏度降低而受到低光条件的阻碍，而后者表现出过度的功耗，降低的帧速率和有限的操作距离。神经形态相机具有超低延迟、能效和宽动态范围，为克服这些限制提供了可行的解决方案。在这里，我们介绍YeLan，一个神经形态相机驱动的，三维，高频人体姿态估计（HPE）系统，能够承受低光环境和动态背景。我们编制了有史以来第一个神经形态相机舞蹈HPE数据集，并设计了一个完全自适应的运动到事件、物理意识模拟器。YeLan在艰苦的条件下超越了基线模型，并对不同的服装类型，背景运动，视角，遮挡和照明波动表现出弹性。
Technology-mediated dance experiences, as a medium of entertainment, are a key element in both traditional and virtual reality-based gaming platforms. These platforms predominantly depend on unobtrusive and continuous human pose estimation as a means of capturing input. Current solutions primarily employ RGB or RGB-Depth cameras for dance gaming applications; however, the former is hindered by low-light conditions due to motion blur and reduced sensitivity, while the latter exhibits excessive power consumption, diminished frame rates, and restricted operational distance. Boasting ultra-low latency, energy efficiency, and a wide dynamic range, neuromorphic cameras present a viable solution to surmount these limitations. Here, we introduceYeLan, a neuromorphic camera-driven, three-dimensional, high-frequency human pose estimation (HPE) system capable of withstanding low-light environments and dynamic backgrounds. We have compiled the first-ever neuromorphic camera dance HPE dataset and devised a fully adaptable motion-to-event, physics-conscious simulator. YeLan surpasses baseline models under strenuous conditions and exhibits resilience against varying clothing types, background motion, viewing angles, occlusions, and lighting fluctuations.