AuMoCap: Augmented, Portable, Real-Time, Markerless Motion Capture for Digital Humans

成像仪集团需要无标记运动捕捉(MOCAP)系统来支持计算机视觉(CV)、计算机图形学(CG)和人机交互(HCI)的基础研究。对于CV，它将用于三个目的：作为多视角重建的捕获方式，用于验证新算法的参考系统，以及用于记录深度学习算法的训练数据集的来源。对于CG来说，它将促进对更自然、更身临其境的网真和数字人类化身的研究。在人机界面中，它将提供一个全身输入传感器系统，用于原型新的交互技术。此外，它将使计算机科学之外的应用成为可能，例如像基于标记的解决方案所做的那样，实现定量生命科学研究的自动化，但不会阻碍或影响受试者的自然行为。*所需的设备包括以下硬件和软件：1)将8个高清视频传输到中央处理器；2)自动校准；3)将用户的运动重建为虚拟骨骼的3D位置，包括凝视方向的细节；和4)增强现实，在受控条件下模拟真实环境。*它将成为Imager集团一个新的、专门的增强现实工作室的一部分，并为运行多模式、具体化和沉浸式远程呈现研究提供技术基础设施。然而，它的设计是便携的，因此可以在户外环境中使用MOCAP，并在具有固定设备的专用实验室中设置MOCAP，例如重型测力板。已经与共同申请者和成像成员在人机界面以及不列颠哥伦比亚大学的心理学和物理治疗实验室建立了具体的合作。*该系统具有以下独特的特性，可以补充现有的基于标记的解决方案：*1)它是非侵入性的，无论是身体上还是心理上。这对于运动来说很重要，因为在体育运动中，汗水本身就对记分器构成了挑战，对于运行心理测量学研究以消除外部影响来说也是很重要的。*2)在全阳光和不受控制的运动背景下进行户外拍摄。*3)拍摄对象穿着宽松的衣服，如裙子，因为不必须附加记分器。*4)快速设置时间，因为不需要仪器。*5)受控的照明和增强，用于训练和测试计算机视觉算法。*6)记录完整的外观，不同于稀疏的基于记分器的解决方案。*同时，它以1厘米的精度提供多人的实时运动轨迹，接近基于标记的解决方案的精度。*我的学生将在他们研究的所有阶段定期使用该系统：开发、培训和测试时间。此外，HCI的5名Imager学生在他们的用户学习中半常规地需要这个运动跟踪系统，偶尔，但大规模的心理学和物理治疗研究将得到支持。所有这些都将有助于确保UBC和加拿大在以人为中心的人工智能领域保持领先地位。

The Imager Group needs a markerless motion capture (MoCap) system to support fundamental research in computer vision (CV), computer graphics (CG), and human-computer interaction (HCI). For CV, it will serve three purposes: as a capture modality for multi-view reconstruction, a reference system for validating new algorithms, and a source for recording training datasets for deep learning algorithms. For CG, it will facilitate research on more natural and immersive telepresence and digital human avatars. In HCI, it will provide a full-body input sensor system for prototyping new interaction techniques. Moreover, it will enable applications outside Computer Science, such as automating quantitative life science studies, as marker-based solutions do, but without hampering or influencing the subjects' natural behavior.******The equipment required includes the hardware and software to 1) stream eight HD videos to a central processing unit; 2) automated calibration; 3) reconstruction of the user's motion as the 3D position of a virtual skeleton over time, including the details of gaze direction; and 4) augmented reality to simulate real environments in controlled conditions.******It will be part of a new, dedicated augmented reality studio of the Imager group and provide the technical infrastructure for running multi-modal, embodied, and immersive telepresence research. Nevertheless, it is designed to be portable, thereby allowing MoCap in outdoor environments and setups in dedicated labs with immobile equipment, such as heavy force plates. Concrete collaborations have been set up with the co-applicant and Imager members in HCI, and the Psychology and Physiotherapy labs at UBC.******The system has the following unique properties which complement existing marker-based solutions:***1) It is nonintrusive, whether physically or psychologically. This is important for sports, where sweat alone poses a challenge to markers, and for running psychometrics studies for eliminating external influence.***2) Outdoor capture at full sunlight and against an uncontrolled moving background.***3) Capturing subjects with loose apparel, such as skirts, as no markers must be attached.***4) Quick setup time, as no instrumentation is necessary.***5) Controlled illumination and augmentation for training and testing computer vision algorithms.***6) Recording the complete appearance, unlike sparse marker-based solutions.***At the same time, it provides real-time motion trajectories of multiple people at an accuracy of 1cm, which is close to the accuracy of marker-based solutions.******My students will use the system regularly, at all stages of their research: at development, training, and test time. Additionally, 5 Imager students in HCI have semi-regular need of this motion tracking system as part of their user studies, and occasional, yet, large-scale studies in Psychology and Physiotherapy will be supported. All this will help ensure that UBC and Canada remain leaders in the human-centered AI field.