Mobile Motion Capture: From Photorealistic Avatars to Privacy Protection

这项研究计划的目标是通过在人、移动计算机和他们的物理环境之间创建直观的界面来改善日常生活。到目前为止，智能设备缺乏对用户和周围环境的详细了解。例如，由于不知道用户的情况，不恰当的推送通知扰乱了我们的注意力。此外，尽管增强现实的显示能力已经成熟，但栩栩如生的全息远程呈现仍然是科幻。逼真的化身--用户的图形表示--只有在配备了数十个高端摄像头的专用工作室才能实现。一些机器学习(ML)方法在较不受控制的条件下成功；然而，对于中等复杂的服装，如裙子、高分辨率或极端运动，则失败，因为有监督的ML需要大量的标记样本。我建议用世界表征学习(WRL)的概念来克服这些限制，它利用强大的自我监督形式，分层次地捕捉我们物理世界的几何结构。WRL不是在专有工作室中扫描并使用人月的注释，而是通过利用几何、时间和物理约束作为监督信号，从纯粹的消费者级别的视频中学习。最大的挑战之一将是使WRL参数化，并可供艺术家和临时用户访问-这与ML模型的可解释性密切相关。如果成功，WRL将取代广泛使用的手工制作的网格表示法，这些表示法在表示不规则形状和拓扑变化方面存在问题。此外，学习丰富多样的无标签消费者视频将减轻偏见，如对高加索肤色、身体特征和服装风格的偏见；WRL将使个性化头像成为可能。目标是开发用于消费级设备的移动动作捕获，以恢复显示无处不在的人机交互(HCI)和上述推送通知时间的情感和意图的细粒度和细微动作。它与我在神经科学、研究神经回路如何协调肢体行为方面的补充研究线索有关，也与其他生命科学有关，在这些领域，自动捕获有助于挖掘新发现的根源。使用智能设备捕获人员(请参阅谷歌眼镜(Google Glass)在未经同意的情况下仔细观察旁观者。我提倡一种保护隐私的相机：一种声学相机，它能感知超声波回波，而不是可见光，并使用ML算法重建人体运动。声音回声携带的信息要少得多，这掩盖了主体身份，有利于隐私，但足以定位人；甚至可能超出视线。我认为WRL、移动捕获和匿名跟踪解决方案非常适合智能环境和可穿戴设备，并预测将对我们的日常效用产生巨大的积极影响，同时解决伦理问题。

The objective of this research program is to enhance everyday life by creating intuitive interfaces between persons, mobile computers, and their physical environment. As of now, smart devices lack a detailed picture of the user and the context of the surrounding. For instance, by not knowing the situation of the user, inapt push notifications disrupt our attention. Moreover, even though display capabilities matured for augmented reality, life-like, holographic telepresence is still Sci-Fi. Realistic avatars-graphical representation of a user-are only realized in dedicated studios equipped with dozens of high-end cameras. Some machine learning (ML) methods succeed in less-controlled conditions; however, fail for moderately complex apparel, such as a skirt, high resolution, or extreme motions because supervised ML demands huge amounts of labeled examples. I propose to overcome these limitations with the concept of world representation learning (WRL), which captures the geometric structure of our physical world hierarchically, using strong forms of self-supervision. Instead of scanning in proprietary studios and using person-months of annotation, WRL learns from mere consumer-level videos by utilizing geometric, temporal, and physical constraints as supervision signals. One of the biggest challenges will be to make WRL parametric and accessible by artists and casual users-closely related to the interpretability of ML models. If successful, WRL will supersede the widely used hand-crafted mesh representations that have problems representing irregular shapes and topological changes. Furthermore, learning from unlabeled consumer videos that are plentiful and diverse will alleviate biases, such as towards Caucasian skin tone, body characteristics, and clothing style; WRL will enable personalized avatars. The objective is developing mobile motion capture for consumer-level devices for recovering the fine-grained and subtle motions that reveal emotion and intent for ubiquitous human-computer interaction (HCI) and the mentioned push-notification timing. It is linked to my complementary research threads on neuroscience, on studying how neural circuits orchestrate limbed behaviors, and to other life sciences where automated capture helps to unroot new discoveries. Capturing people with smart devices (cf. Google Glass) scrutinizes bystanders without consent. I advocate a privacy-preserving camera: an acoustic camera which senses ultrasonic echoes instead of visible light and uses ML algorithms to reconstruct human motion. Sound echoes carry far less information, which masks subject identity in favor of privacy, yet, could suffice to localize persons; perhaps even beyond the line of sight. I see a great fit of the WRL, mobile capture, and anonymous tracking solutions for smart environments and wearables, and predict a huge positive impact on our everyday utility while resolving ethical concerns.