Hand Segmentation and Detection Applied to Mobile Devices Using a Noisy Input Depth Image Stream****

随着手势识别工具的兴起，人机交互正在迅速发展。有几种最先进的算法可以解决这个问题，但是没有令人信服的结果可以跟踪运动、定位手和解释手势。Leap Motion专注于手部跟踪，但不能产生合适的手势检测和解释。像iPhone这样的移动设备配备了深度摄像头，但该行业需要创新的、改进的算法来生成移动手的精确3D表示。因此，准确的实时人手跟踪代表了先进研究和技术发展的肥沃领域，迄今为止还没有强大的解决方案。**总部位于多伦多的Xesto开发了一个基于云的机器学习平台，可以识别和记录手势，用于使用Leap Motion的深度图和带有深度摄像头的移动设备的基于手势的应用程序。Xesto已经建立了一个原型，目前需要一个卓越的手部3D表示，同时利用Xesto的设备不可知平台上强大的手部检测算法。**该项目是Deepa Kundur教授和Xesto之间的合作，旨在提高移动设备上手部跟踪的实时性能。具体来说，该项目利用了Xesto的RGB-D手部姿势检测和手势识别管道，适用于任何深度传感器。本研究旨在设计一种新颖的卷积神经网络架构，利用时间信息生成准确的实时手部特征热图。该问题在构建具有大自由度、自相似部件和自遮挡的可关节物体的能量函数估计和优化方面具有很高的技术深度。**拟议的研究预计将对手部跟踪领域产生影响，为Xesto提供技术优势，以开发健康，零售和汽车行业的卓越应用。此外，这项工作将为每个移动设备用户提供一种仅通过手势与他们的机器交互的新方法。

Human computer interaction is rapidly evolving with an emerging focus on gesture recognition tools. There are several state-of-the-art algorithms that tackle aspects of this problem, but no convincing result has been presented that tracks motion, localizes hands, and interprets gestures. Leap Motion specializes in hand tracking but does not produce suitable gesture detection and interpretation. Mobile devices such as iPhone are equipped with depth cameras, but the industry needs innovative, improved algorithms to produce an accurate 3D representation of a moving hand. Hence, accurate real-time human hand tracking represents a fertile area of advanced research and technology development, with no robust solution to date.** Toronto-based Xesto has developed a cloud-based machine learning platform to recognize and record gestures for use in gesture-based applications using depth map from Leap Motion and mobile devices with depth cameras. Xesto has built a prototype that currently requires a superior 3D representation of the hand, while leveraging robust hand detection algorithms on Xesto's device agnostic platform.** This project represents a collaboration between Professor Deepa Kundur and Xesto to improve the real time performance of hand tracking on mobile devices. Specifically, the project makes use of Xesto's RGB-D hand-pose detection and gesture recognition pipeline that is applicable to any depth sensor. The research aims to design a novel convolutional neural network architecture that uses temporal information to produce an accurate real-time heat map of hand-features. This problem has a high level of technical depth in estimating and optimizing the energy function that constructs articulable objects with large degrees of freedom, self-similar parts and is subject to self-occlusion. ** The proposed research is expected to have impact in the field of hand tracking providing Xesto a technological advantage to develop superior applications for health, retail and automotive industry. Further, this work will grant each mobile device user a new way to interact with their machines only through gestures.