AuMoCap: Augmented, Portable, Real-Time, Markerless Motion Capture for Digital Humans

Imager Group需要一个无标记运动捕捉（MoCap）系统来支持计算机视觉（CV）、计算机图形（CG）和人机交互（HCI）的基础研究。对于CV，它将用于三个目的：作为多视图重建的捕获模式，验证新算法的参考系统，以及记录深度学习算法的训练数据集的来源。对于CG，它将促进对更自然和身临其境的远程呈现和数字人类化身的研究。在HCI中，它将提供一个全身输入传感器系统，用于新交互技术的原型设计。此外，它将使计算机科学之外的应用成为可能，例如自动化定量生命科学研究，就像基于标记的解决方案一样，但不会妨碍或影响受试者的自然行为。所需的设备包括硬件和软件，以1）将八个高清视频流传输到中央处理器; 2）自动校准; 3）随着时间的推移将用户的运动重建为虚拟骨架的3D位置，包括注视方向的细节;以及4）增强现实，以在受控条件下模拟真实的环境。它将成为Imager集团新的专用增强现实工作室的一部分，并为运行多模式，体现和沉浸式远程呈现研究提供技术基础设施。尽管如此，它被设计为便携式的，从而允许在户外环境中进行MoCap，并在专用实验室中使用固定设备（如重型力板）进行设置。已经与HCI的共同申请人和Imager成员以及UBC的心理学和物理治疗实验室建立了具体的合作关系。该系统具有以下独特的性质，补充了现有的基于标记的解决方案：*1）它是非侵入性的，无论是在身体上还是心理上。这对运动很重要，因为汗水本身就对标记者构成了挑战，也对进行心理测量学研究以消除外部影响很重要。2)在阳光充足的室外拍摄，并对不受控制的移动背景。* 3)拍摄穿着宽松衣服的拍摄对象，如裙子，因为不需要附加标记。* 4)快速设置时间，因为不需要仪器。* 5)用于训练和测试计算机视觉算法的受控照明和增强。* 6)记录完整的外观，与基于稀疏标记的解决方案不同。*同时，它以1cm的精度提供多人的实时运动轨迹，接近基于标记的解决方案的精度。我的学生将在他们研究的各个阶段定期使用该系统：在开发、培训和测试时。此外，HCI中的5名成像仪学生半定期需要这种运动跟踪系统作为其用户研究的一部分，偶尔也会支持心理学和物理治疗的大规模研究。所有这些都将有助于确保UBC和加拿大在以人为本的人工智能领域保持领先地位。

The Imager Group needs a markerless motion capture (MoCap) system to support fundamental research in computer vision (CV), computer graphics (CG), and human-computer interaction (HCI). For CV, it will serve three purposes: as a capture modality for multi-view reconstruction, a reference system for validating new algorithms, and a source for recording training datasets for deep learning algorithms. For CG, it will facilitate research on more natural and immersive telepresence and digital human avatars. In HCI, it will provide a full-body input sensor system for prototyping new interaction techniques. Moreover, it will enable applications outside Computer Science, such as automating quantitative life science studies, as marker-based solutions do, but without hampering or influencing the subjects' natural behavior.******The equipment required includes the hardware and software to 1) stream eight HD videos to a central processing unit; 2) automated calibration; 3) reconstruction of the user's motion as the 3D position of a virtual skeleton over time, including the details of gaze direction; and 4) augmented reality to simulate real environments in controlled conditions.******It will be part of a new, dedicated augmented reality studio of the Imager group and provide the technical infrastructure for running multi-modal, embodied, and immersive telepresence research. Nevertheless, it is designed to be portable, thereby allowing MoCap in outdoor environments and setups in dedicated labs with immobile equipment, such as heavy force plates. Concrete collaborations have been set up with the co-applicant and Imager members in HCI, and the Psychology and Physiotherapy labs at UBC.******The system has the following unique properties which complement existing marker-based solutions:***1) It is nonintrusive, whether physically or psychologically. This is important for sports, where sweat alone poses a challenge to markers, and for running psychometrics studies for eliminating external influence.***2) Outdoor capture at full sunlight and against an uncontrolled moving background.***3) Capturing subjects with loose apparel, such as skirts, as no markers must be attached.***4) Quick setup time, as no instrumentation is necessary.***5) Controlled illumination and augmentation for training and testing computer vision algorithms.***6) Recording the complete appearance, unlike sparse marker-based solutions.***At the same time, it provides real-time motion trajectories of multiple people at an accuracy of 1cm, which is close to the accuracy of marker-based solutions.******My students will use the system regularly, at all stages of their research: at development, training, and test time. Additionally, 5 Imager students in HCI have semi-regular need of this motion tracking system as part of their user studies, and occasional, yet, large-scale studies in Psychology and Physiotherapy will be supported. All this will help ensure that UBC and Canada remain leaders in the human-centered AI field.