Hand Segmentation and Detection Applied to Mobile Devices Using a Noisy Input Depth Image Stream****

随着手势识别工具的出现，人机交互正在迅速发展。有几种最先进的算法可以解决这个问题的各个方面，但还没有提出令人信服的结果来跟踪运动、定位手势和解释手势。Leap Motion专门从事手部跟踪，但不能产生合适的手势检测和解释。IPhone等移动设备配备了深度摄像头，但该行业需要创新的、改进的算法来生成运动手的准确3D表示。因此，准确的实时手部跟踪是先进研究和技术开发的一个肥沃领域，到目前为止还没有强大的解决方案。**总部位于多伦多的Xesto开发了一个基于云的机器学习平台，可以识别和记录手势，用于基于手势的应用程序，使用Leap Motion的深度图和配备深度摄像头的移动设备。Xesto已经构建了一个原型，目前需要高级的手3D表示，同时在Xesto的设备不可知平台上利用强大的手检测算法。**该项目代表Deepa ukdur教授和Xesto教授合作，以提高移动设备上手跟踪的实时性能。具体地说，该项目利用了Xesto的RGB-D手势检测和手势识别流水线，该流水线适用于任何深度传感器。本研究旨在设计一种新颖的卷积神经网络结构，利用时间信息来生成准确的实时手部特征热图。这个问题在估计和优化能量函数方面具有很高的技术深度，该能量函数构造具有大自由度、自相似部件和受自遮挡影响的可关节对象。**拟议的研究预计将在手跟踪领域产生影响，为Xesto提供技术优势，为健康、零售和汽车行业开发卓越的应用程序。此外，这项工作将为每个移动设备用户提供一种仅通过手势与机器交互的新方式。

Human computer interaction is rapidly evolving with an emerging focus on gesture recognition tools. There are several state-of-the-art algorithms that tackle aspects of this problem, but no convincing result has been presented that tracks motion, localizes hands, and interprets gestures. Leap Motion specializes in hand tracking but does not produce suitable gesture detection and interpretation. Mobile devices such as iPhone are equipped with depth cameras, but the industry needs innovative, improved algorithms to produce an accurate 3D representation of a moving hand. Hence, accurate real-time human hand tracking represents a fertile area of advanced research and technology development, with no robust solution to date.** Toronto-based Xesto has developed a cloud-based machine learning platform to recognize and record gestures for use in gesture-based applications using depth map from Leap Motion and mobile devices with depth cameras. Xesto has built a prototype that currently requires a superior 3D representation of the hand, while leveraging robust hand detection algorithms on Xesto's device agnostic platform.** This project represents a collaboration between Professor Deepa Kundur and Xesto to improve the real time performance of hand tracking on mobile devices. Specifically, the project makes use of Xesto's RGB-D hand-pose detection and gesture recognition pipeline that is applicable to any depth sensor. The research aims to design a novel convolutional neural network architecture that uses temporal information to produce an accurate real-time heat map of hand-features. This problem has a high level of technical depth in estimating and optimizing the energy function that constructs articulable objects with large degrees of freedom, self-similar parts and is subject to self-occlusion. ** The proposed research is expected to have impact in the field of hand tracking providing Xesto a technological advantage to develop superior applications for health, retail and automotive industry. Further, this work will grant each mobile device user a new way to interact with their machines only through gestures.