CHS: Small: High Resolution Motion Capture

该项目研究了一种新型的全身动作捕捉技术，该技术是由深度学习和高分辨率数码相机的最新进展实现的。经典的动作捕捉系统依赖于带有小球体作为标记的西装，而这个项目引入了一种新型的西装，使用特殊的印刷图案代替任何附件。这种模式将包含一种具有两个明显优势的新型标记：(1)能够自动检测哪个标记是哪个标记，以及(2)比以前的系统更密集的标记集。所提出的方法是完全被动的，因此易于使用，只依赖于(a)室内照明或自然光，(b)由带有特殊印刷图案的弹性织物制成的西装，但没有任何电线或电池。唯一需要的电子设备是标准的数码相机，从几个相机到大型多相机系统。这种灵活性将使我们能够支持各种规模的应用程序，从个人研究人员或制造商到大型机构或生产工作室。新的全身捕捉技术，具有更高的精度，同时易于使用，有可能影响人体运动的研究和临床研究，例如骨科，运动医学，康复，物理治疗和人体工程学。高质量的人体运动数据也可以促进更好的虚拟或增强现实系统和应用程序。通过将计算机科学与人体运动相结合，动作捕捉系统提供了独特的教育和推广机会，包括在年轻人中流行的活动，如体育和体操。使用使用人工神经网络识别的基于纹理的标记的新型运动捕捉服的想法之前尚未被探索过，并开辟了许多有趣的研究问题，例如“尽管皮肤运动引起了很大的弹性扭曲，但哪种类型的标记和西装纹理将导致最佳的检测和标记结果？”所提出的计算方法需要对神经网络进行训练和验证，并有助于以下主题的研究：(1)合成数据生成，(2)自动数据增强，(3)神经网络的置信度校准，特别是教神经网络量化自己预测中的错误风险。为了进一步提高鲁棒性和准确性，神经网络的概率输出将与先验相结合，例如三维可变形形状模型；这可以通过贝叶斯推理有原则地完成，这就导致了连续优化和离散优化相结合的研究问题。最后，所设想的系统产生的高分辨率数据激发了对提高可变形形状模型的解剖真实感的研究，特别是关节运动学和肌肉激活的数据驱动建模。设想中的全身捕获系统将被设计成易于在各种机构、诊所或工作室复制和部署。该项目将通过公共开源代码库共享研究成果，促进协作和数据共享。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project studies a new type of full-body motion capture technology which has been enabled by recent advances in deep learning and high-resolution digital cameras. Unlike classical motion capture systems which rely on suits with small attached spheres that serve as markers, this project introduces a new type of suit using a special printed pattern instead of any attachments. This pattern will contain a new type of markers with two distinct advantages: (1) the ability to automatically detect which marker is which, and (2) a significantly more dense set of markers than previous systems. The proposed approach is fully passive and therefore easy to use, relying only (a) indoor lighting or natural daylight and (b) a suit made of elastic fabric with a special printed pattern, but without any wires or batteries. The only required electronics are standard digital cameras, ranging from just a few cameras to massive multi-camera systems. This flexibility will allow us to support applications on various scales, from individual researchers or makers to large institutions or production studios. New technology for full-body capture, featuring higher accuracy while being easy to use, has the potential to impact research and clinical studies of human motion, e.g., in orthopedics, sports medicine, rehabilitation, physical therapy and ergonomics. High-quality human motion data can also facilitate better virtual or augmented reality systems and applications. By combining computer science and human motion, motion capture systems enables unique educational and outreach opportunities involving activities popular among young people, such as sports and gymnastics. The idea of using a new type of motion capture suit with texture-based markers recognized using artificial neural networks has not been explored before and opens up many interesting research questions such as "Which types of markers and suit textures will lead to the best detection and labeling results, despite large elastic distortions induced by the motion of the skin?" The proposed computing methodology requires training and validation of neural networks and contributes to research on the following topics: (1) synthetic data generation, (2) automated data augmentation, (3) confidence calibration of neural networks, in particular teaching neural networks to quantify the risk of errors in their own predictions. To further improve robustness and accuracy, the probabilistic output of neural networks will be combined with priors, such as a 3D deformable shape model; this can be done in a principled way via Bayesian inference, which leads to research problems combining continuous and discrete optimization. Finally, the high-resolution data produced by the envisioned system motivates research on improving the anatomical realism of deformable shape models, in particular data-driven modeling of joint kinematics and muscle activations. The envisioned full-body capture system will be designed to be easy to replicate and deploy at various institutions, clinics or studios. This project will share research results through common open source codebase, facilitating collaboration and data sharing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.