Mobile Motion Capture: From Photorealistic Avatars to Privacy Protection

这项研究计划的目标是通过在人、移动的计算机及其物理环境之间创建直观的界面来改善日常生活。到目前为止，智能设备缺乏用户的详细图片和周围环境。例如，由于不知道用户的情况，不适当的推送通知会扰乱我们的注意力。 此外，尽管增强现实的显示能力已经成熟，但逼真的全息远程呈现仍然是科幻小说。只有在配备了数十台高端摄像机的专用工作室中，才能实现用户的真实化身图形表示。一些机器学习（ML）方法在控制较少的条件下取得了成功;但是，对于中等复杂的服装，如裙子，高分辨率或极端运动，由于监督ML需要大量的标记示例，因此失败。 我建议用世界表征学习（WRL）的概念来克服这些局限性，WRL使用强大的自我监督形式分层地捕获我们物理世界的几何结构。WRL没有在专有工作室中扫描并使用人工月的注释，而是通过利用几何、时间和物理约束作为监督信号，从纯粹的消费级视频中学习。 最大的挑战之一将是使WRL参数化，并由艺术家和休闲用户访问，与ML模型的可解释性密切相关。如果成功，WRL将取代广泛使用的手工制作的网格表示，这些表示在表示不规则形状和拓扑变化方面存在问题。此外，从丰富多样的未标记消费者视频中学习将减轻偏见，例如对白人肤色、身体特征和服装风格的偏见; WRL将实现个性化化身。 我们的目标是开发消费级设备的移动的动作捕捉，以恢复细粒度和微妙的动作，揭示无处不在的人机交互（HCI）的情感和意图，以及上述的推送通知定时。它与我在神经科学方面的补充研究线索有关，研究神经回路如何协调肢体行为，以及自动捕获有助于挖掘新发现的其他生命科学。 用智能设备吸引人（参见谷歌眼镜）在未经同意的情况下仔细检查旁观者。我提倡一种保护隐私的摄像机：一种声学摄像机，它可以感知超声波回波而不是可见光，并使用ML算法来重建人体运动。声音回声携带的信息要少得多，这掩盖了主体身份，有利于隐私，但足以定位人;甚至可能超出视线。 我认为WRL，移动的捕获和匿名跟踪解决方案非常适合智能环境和可穿戴设备，并预测在解决道德问题的同时对我们的日常效用产生巨大的积极影响。

The objective of this research program is to enhance everyday life by creating intuitive interfaces between persons, mobile computers, and their physical environment. As of now, smart devices lack a detailed picture of the user and the context of the surrounding. For instance, by not knowing the situation of the user, inapt push notifications disrupt our attention. Moreover, even though display capabilities matured for augmented reality, life-like, holographic telepresence is still Sci-Fi. Realistic avatarsgraphical representation of a userare only realized in dedicated studios equipped with dozens of high-end cameras. Some machine learning (ML) methods succeed in less-controlled conditions; however, fail for moderately complex apparel, such as a skirt, high resolution, or extreme motions because supervised ML demands huge amounts of labeled examples. I propose to overcome these limitations with the concept of world representation learning (WRL), which captures the geometric structure of our physical world hierarchically, using strong forms of self-supervision. Instead of scanning in proprietary studios and using person-months of annotation, WRL learns from mere consumer-level videos by utilizing geometric, temporal, and physical constraints as supervision signals. One of the biggest challenges will be to make WRL parametric and accessible by artists and casual usersclosely related to the interpretability of ML models. If successful, WRL will supersede the widely used hand-crafted mesh representations that have problems representing irregular shapes and topological changes. Furthermore, learning from unlabeled consumer videos that are plentiful and diverse will alleviate biases, such as towards Caucasian skin tone, body characteristics, and clothing style; WRL will enable personalized avatars. The objective is developing mobile motion capture for consumer-level devices for recovering the fine-grained and subtle motions that reveal emotion and intent for ubiquitous human-computer interaction (HCI) and the mentioned push-notification timing. It is linked to my complementary research threads on neuroscience, on studying how neural circuits orchestrate limbed behaviors, and to other life sciences where automated capture helps to unroot new discoveries. Capturing people with smart devices (cf. Google Glass) scrutinizes bystanders without consent. I advocate a privacy-preserving camera: an acoustic camera which senses ultrasonic echoes instead of visible light and uses ML algorithms to reconstruct human motion. Sound echoes carry far less information, which masks subject identity in favor of privacy, yet, could suffice to localize persons; perhaps even beyond the line of sight. I see a great fit of the WRL, mobile capture, and anonymous tracking solutions for smart environments and wearables, and predict a huge positive impact on our everyday utility while resolving ethical concerns.