Teaching machines to see in 4D

增强现实（AR）和混合现实（MR）是最近引起市场极大兴趣的两项应用，它们是iPhoneX和微软Hololens等高科技产品旗舰功能背后的核心技术。在这些新技术的背后，是机器模拟和跟踪其观察到的东西的能力。不幸的是，尽管深度学习取得了惊人的进步，但这些建模和跟踪方法仍然严重依赖于十年前的算法和人为设计的想法。最近深度学习的成功，以AlphaGo超越人类为例，表明我们可以做得更好。在这个提议中，我们将通过开发基于学习的实时四维视觉跟踪方法，教会机器在四维（空间和时间）中观看。我们将让机器提供更精确的建模和跟踪结果，最终实现现实世界和虚拟世界的无缝集成，基于更精确和详细的手势识别增强人机交互，让机器智能地与世界互动。所有这些都是实时的。该项目的成功将推动加拿大向前发展，使加拿大在AR/MR产品和自主机器人等新兴产业中处于领先地位。

Augmented reality (AR) and mixed reality (MR) are two very recent applications that are drawing much market interest, being the core technology behind flagship features of high-tech products, for example the iPhoneX and Microsoft Hololens. Under the hood of these new technology is the ability of machines to model and track what it is observing. Unfortunately, despite the phenomenal advancements from deep learning, these modeling and tracking methods still rely heavily on decade-old algorithms and human- designed ideas. The recent success of deep learning, exemplified by AlphaGo that surpassed humans, suggests that we can do much better.In this proposal we will teach machines to see in 4D (space and time), by developing learning-based approaches for real-time 4D visual tracking. We will let machines provide much more accurate modeling and tracking results, ultimately resulting in seamless integration of the reality and virtual world, enhanced human machine interaction based on more accurate and detailed gesture recognitions, and allow machines to intelligently interact with the world. All in real-time.The success of this project will push Canada forward and allow Canada to lead the newly emerging industry of AR/MR products and autonomous robotics.