Teaching machines to see in 4D

增强现实(AR)和混合现实(MR)是最近引起市场兴趣的两个应用，它们是iPhoneX和Microsoft Hololens等高科技产品旗舰功能背后的核心技术。在这些新技术的背后，是机器对其观察到的东西进行建模和跟踪的能力。不幸的是，尽管深度学习取得了惊人的进步，但这些建模和跟踪方法仍然严重依赖于十年前的算法和人类设计的想法。最近深度学习的成功，以AlphaGo超越人类为例，表明我们可以做得更好。在这个建议中，我们将通过开发基于学习的方法来实时4D视觉跟踪，来教机器在4D(空间和时间)中看东西。我们将让机器提供更准确的建模和跟踪结果，最终实现现实和虚拟世界的无缝融合，基于更准确和详细的手势识别增强人机交互，并允许机器与世界智能交互。这个项目的成功将推动加拿大的发展，并使加拿大在AR/MR产品和自主机器人这一新兴行业处于领先地位。

Augmented reality (AR) and mixed reality (MR) are two very recent applications that are drawing much market interest, being the core technology behind flagship features of high-tech products, for example the iPhoneX and Microsoft Hololens. Under the hood of these new technology is the ability of machines to model and track what it is observing. Unfortunately, despite the phenomenal advancements from deep learning, these modeling and tracking methods still rely heavily on decade-old algorithms and human- designed ideas. The recent success of deep learning, exemplified by AlphaGo that surpassed humans, suggests that we can do much better.In this proposal we will teach machines to see in 4D (space and time), by developing learning-based approaches for real-time 4D visual tracking. We will let machines provide much more accurate modeling and tracking results, ultimately resulting in seamless integration of the reality and virtual world, enhanced human machine interaction based on more accurate and detailed gesture recognitions, and allow machines to intelligently interact with the world. All in real-time.The success of this project will push Canada forward and allow Canada to lead the newly emerging industry of AR/MR products and autonomous robotics.