Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio
Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio
复制标题
DOI：
10.1038/s41467-020-18441-5
复制
发表时间：
2020-09-11
影响因子：
16.6
通讯作者：
Zimmermann, Jan
中科院分区：
文献类型：
--
作者：
Bala, Praneet C.;Eisenreich, Benjamin R.;Zimmermann, Jan
恒河猴是神经科学、心理学、行为学和医学等多个学科的重要模式动物。猕猴模型的实用性将大大提高的能力，精确测量行为在自由移动的条件下。现有的方法不能提供足够的跟踪。在这里，我们描述了OpenMonkeyStudio，这是一种基于深度学习的无标记运动捕捉系统，用于估计大型无约束环境中自由移动的猕猴的3D姿态。我们的系统使用62台机器视觉摄像机，环绕一个开放的2.45 m x 2.45 m x 2.75 m的外壳。由此产生的多视图图像流允许通过注释图像的3D重建来进行数据增强，以训练鲁棒的视图不变深度神经网络。这种视图不变性代表了对以前的无标记2D跟踪方法的重要进步，并允许对不受约束的自然运动进行全自动姿态推断。我们表明，OpenMonkeyStudio可以用来准确地识别动作和跟踪社交互动。
The rhesus macaque is an important model species in several branches of science, including neuroscience, psychology, ethology, and medicine. The utility of the macaque model would be greatly enhanced by the ability to precisely measure behavior in freely moving conditions. Existing approaches do not provide sufficient tracking. Here, we describe OpenMonkeyStudio, a deep learning-based markerless motion capture system for estimating 3D pose in freely moving macaques in large unconstrained environments. Our system makes use of 62 machine vision cameras that encircle an open 2.45 m x 2.45 m x 2.75 m enclosure. The resulting multiview image streams allow for data augmentation via 3D-reconstruction of annotated images to train a robust view-invariant deep neural network. This view invariance represents an important advance over previous markerless 2D tracking approaches, and allows fully automatic pose inference on unconstrained natural motion. We show that OpenMonkeyStudio can be used to accurately recognize actions and track social interactions.