Mobile Motion Capture: From Photorealistic Avatars to Privacy Protection

点击翻译按钮获取中文摘要

The objective of this research program is to enhance everyday life by creating intuitive interfaces between persons, mobile computers, and their physical environment. As of now, smart devices lack a detailed picture of the user and the context of the surrounding. For instance, by not knowing the situation of the user, inapt push notifications disrupt our attention. Moreover, even though display capabilities matured for augmented reality, life-like, holographic telepresence is still Sci-Fi. Realistic avatars-graphical representation of a user-are only realized in dedicated studios equipped with dozens of high-end cameras. Some machine learning (ML) methods succeed in less-controlled conditions; however, fail for moderately complex apparel, such as a skirt, high resolution, or extreme motions because supervised ML demands huge amounts of labeled examples. I propose to overcome these limitations with the concept of world representation learning (WRL), which captures the geometric structure of our physical world hierarchically, using strong forms of self-supervision. Instead of scanning in proprietary studios and using person-months of annotation, WRL learns from mere consumer-level videos by utilizing geometric, temporal, and physical constraints as supervision signals. One of the biggest challenges will be to make WRL parametric and accessible by artists and casual users-closely related to the interpretability of ML models. If successful, WRL will supersede the widely used hand-crafted mesh representations that have problems representing irregular shapes and topological changes. Furthermore, learning from unlabeled consumer videos that are plentiful and diverse will alleviate biases, such as towards Caucasian skin tone, body characteristics, and clothing style; WRL will enable personalized avatars. The objective is developing mobile motion capture for consumer-level devices for recovering the fine-grained and subtle motions that reveal emotion and intent for ubiquitous human-computer interaction (HCI) and the mentioned push-notification timing. It is linked to my complementary research threads on neuroscience, on studying how neural circuits orchestrate limbed behaviors, and to other life sciences where automated capture helps to unroot new discoveries. Capturing people with smart devices (cf. Google Glass) scrutinizes bystanders without consent. I advocate a privacy-preserving camera: an acoustic camera which senses ultrasonic echoes instead of visible light and uses ML algorithms to reconstruct human motion. Sound echoes carry far less information, which masks subject identity in favor of privacy, yet, could suffice to localize persons; perhaps even beyond the line of sight. I see a great fit of the WRL, mobile capture, and anonymous tracking solutions for smart environments and wearables, and predict a huge positive impact on our everyday utility while resolving ethical concerns.