Analysing visual behaviours of articulated objects from images and videos

每天，世界各地的摄像头都会产生大量的图像和视频剪辑，包括我们的手机摄像头。图像和视频中关注的焦点往往是关于人类个人及其行为的。正如人们可能预料的那样，解释人类视觉行为的能力已经成为计算机视觉的一个中心主题，它确实有广泛的应用范围。其根源是一系列基本的计算机视觉问题，包括姿势估计和从图像和视频中检测个人的动作等。*由于相机硬件和机器学习技术的最新进步，现在有几个专用系统在解释人类全身行为方面取得了令人印象深刻的结果，我们开始见证它们在日常生活中的影响，从游戏中的微软Kinect到没有收银员的Amazon Go杂货店。我的研究小组是第一批开发算法和演示系统的人之一，他们从深度图像分析3D人类手势(2013)，以及从深度相机分析3D实验室动物的行为，例如开阔场地的实验室老鼠和鱼缸里的斑马鱼(2017)。与此同时，仍有许多障碍需要克服。*拟议的研究计划是为了推进视觉行为分析的最先进水平，长期目标是使机器能够正确地看到人类居住的环境并与其互动。特别是，它旨在通过调查以下两个方向来解决一些主要缺陷。在第一个方向，我们专注于设计遵循关节对象的几何约束的学习算法，以及策划具有丰富注释的全面基准以促进整体方法的发展。此外，我们不再详细讨论传统的基于帧的摄像机，这些摄像机具有延迟、内存和计算负担等遗传缺陷，而是计划检查我们基于几何和学习的方法与最新的动态视觉摄像机之间的协同效应。在第二个方向，我们研究可能超越人体全身的重要应用。人们对观察人类和动物的手的行为非常感兴趣。现在也是审查手语应用的时候了，这些应用将帮助弱势群体并产生重大的社会影响。*这项建议的潜在影响包括(1)解决(自我)遮挡以及不一致和不完整的注释的缺点，(2)引入新的视觉行为分析基准，以及(3)加快传播相关实际应用的新进展，如视觉动物行为分析和视觉手语翻译。通过这一研究计划培训的HQP将获得计算机视觉和机器学习方面的研究和实践经验，这些经验可能对他们未来的学术或行业职业生涯有价值。

Every day, an enormous amount of images and video clips are produced by cameras around the world, including our mobile phone cameras. The focus of attention in the images and videos is often about human individuals and their behaviours. As one might expect, the ability to interpret human visual behaviours has become a central theme in computer vision, which has indeed a wide spectrum of applications. At its root is a set of fundamental computer vision problems, including pose estimation and action detection of individuals from images and videos, among others. ******Thanks to the recent advances in camera hardware and machine learning technologies, now there are several dedicated systems with impressive results in interpreting human full-body behaviours, and we start to witness their influence in everyday life, ranging from Microsoft Kinect in gaming to the cashier-less Amazon Go grocery stores. My research group has been among the first in developing algorithms and demonstration systems in analysing 3D human hand poses from depth images (2013), as well as 3D lab animal behaviours from a depth camera, such as lab mouse in the open-field arena and zebrafish in the fish tank (2017). Meanwhile, there are still many hurdles to overcome. ******The proposed research program is to advance the state-of-the-art in visual behaviour analysis, with the long term goal of enabling machines to properly see and interact with a human inhabited environment. In particular, it aims to address some of the main shortcomings by investigating the following two directions. In the first direction, we focus on devising learning algorithms that follow the geometric constraints of articulated objects, as well as curating comprehensive benchmarks with rich annotations to facilitate the development of holistic approaches. Moreover, instead of elaborating around the traditional frame-based cameras that have the inherit drawbacks including latency, memory, and computational burdens, we plan to examine the synergism between our geometry and learning based approach and the very recent dynamic vision cameras. In the second direction, we investigate important applications that may go beyond human full-bodies. It is of strong interest to look at the behaviours of human hands, as well as animals. It is also timely to review sign language applications that would help disadvantaged people and have significant societal impacts. ******Potential impact of this proposal includes (1) addressing the shortcomings of (self-)occlusion, and inconsistent & incomplete annotations, (2) introducing new benchmark for visual behaviour analysis, and (3) expediting the dissemination of new advances in related practical applications, such as visual animal behaviour analysis and visual sign language translation. The HQP trained through this research program will gain research as well as hands-on experiences in computer vision and machine learning that could be valuable in their future academic or industry careers.