Mobile Motion Capture: From Photorealistic Avatars to Privacy Protection

这项研究计划的目标是通过在人、移动计算机和他们的物理环境之间创建直观的界面来改善日常生活。到目前为止，智能设备缺乏用户和周围环境的详细图片。例如，由于不了解用户的情况，不恰当的推送通知会干扰我们的注意力。此外，尽管增强现实的显示能力已经成熟，但逼真的全息远程呈现仍然是科幻小说。逼真的虚拟形象——用户的图形化表现——只有在配备了数十台高端摄像机的专门工作室才能实现。一些机器学习（ML）方法在控制较少的条件下取得成功；然而，对于中等复杂的服装，如裙子、高分辨率或极端运动，它就失败了，因为有监督的机器学习需要大量的标记示例。我建议用世界表征学习（world representation learning， WRL）的概念来克服这些限制，WRL使用强大的自我监督形式，分层地捕捉我们物理世界的几何结构。WRL没有在专有工作室进行扫描，也没有使用人-月的注释，而是通过利用几何、时间和物理约束作为监督信号，从纯粹的消费者级视频中学习。最大的挑战之一将是使WRL参数化，并由艺术家和普通用户访问，这与ML模型的可解释性密切相关。如果成功，WRL将取代广泛使用的手工制作的网格表示，这些表示在表示不规则形状和拓扑变化方面存在问题。此外，从大量且多样化的未标记消费者视频中学习将减轻偏见，例如对高加索人肤色、身体特征和服装风格的偏见；WRL将启用个性化头像。目标是为消费者级设备开发移动动作捕捉，以恢复显示无处不在的人机交互（HCI）和上述推送通知时间的情感和意图的细粒度和微妙动作。它与我在神经科学方面的补充研究线索有关，研究神经回路如何协调肢体行为，以及其他生命科学领域的自动捕获有助于根除新发现。用智能设备（如b谷歌Glass）捕捉人，在未经同意的情况下仔细观察旁观者。我提倡一种隐私保护相机：一种声学相机，它感知超声波回声而不是可见光，并使用ML算法重建人体运动。声音回声携带的信息少得多，这掩盖了主体的身份，有利于隐私，然而，足以定位人；甚至可能在视线之外。我认为WRL、移动捕捉和匿名跟踪解决方案非常适合智能环境和可穿戴设备，并预测在解决道德问题的同时，对我们的日常效用会产生巨大的积极影响。

The objective of this research program is to enhance everyday life by creating intuitive interfaces between persons, mobile computers, and their physical environment. As of now, smart devices lack a detailed picture of the user and the context of the surrounding. For instance, by not knowing the situation of the user, inapt push notifications disrupt our attention. Moreover, even though display capabilities matured for augmented reality, life-like, holographic telepresence is still Sci-Fi. Realistic avatars-graphical representation of a user-are only realized in dedicated studios equipped with dozens of high-end cameras. Some machine learning (ML) methods succeed in less-controlled conditions; however, fail for moderately complex apparel, such as a skirt, high resolution, or extreme motions because supervised ML demands huge amounts of labeled examples. I propose to overcome these limitations with the concept of world representation learning (WRL), which captures the geometric structure of our physical world hierarchically, using strong forms of self-supervision. Instead of scanning in proprietary studios and using person-months of annotation, WRL learns from mere consumer-level videos by utilizing geometric, temporal, and physical constraints as supervision signals. One of the biggest challenges will be to make WRL parametric and accessible by artists and casual users-closely related to the interpretability of ML models. If successful, WRL will supersede the widely used hand-crafted mesh representations that have problems representing irregular shapes and topological changes. Furthermore, learning from unlabeled consumer videos that are plentiful and diverse will alleviate biases, such as towards Caucasian skin tone, body characteristics, and clothing style; WRL will enable personalized avatars. The objective is developing mobile motion capture for consumer-level devices for recovering the fine-grained and subtle motions that reveal emotion and intent for ubiquitous human-computer interaction (HCI) and the mentioned push-notification timing. It is linked to my complementary research threads on neuroscience, on studying how neural circuits orchestrate limbed behaviors, and to other life sciences where automated capture helps to unroot new discoveries. Capturing people with smart devices (cf. Google Glass) scrutinizes bystanders without consent. I advocate a privacy-preserving camera: an acoustic camera which senses ultrasonic echoes instead of visible light and uses ML algorithms to reconstruct human motion. Sound echoes carry far less information, which masks subject identity in favor of privacy, yet, could suffice to localize persons; perhaps even beyond the line of sight. I see a great fit of the WRL, mobile capture, and anonymous tracking solutions for smart environments and wearables, and predict a huge positive impact on our everyday utility while resolving ethical concerns.