Analysing visual behaviours of articulated objects from images and videos

每天，世界各地的摄像机都会产生大量的图像和视频剪辑，包括我们的移动的手机摄像机。图像和视频中关注的焦点通常是人类个体及其行为。正如人们所预料的那样，解释人类视觉行为的能力已经成为计算机视觉的一个中心主题，它确实有广泛的应用。其根源是一系列基本的计算机视觉问题，包括从图像和视频中对个体进行姿势估计和动作检测等。** 由于相机硬件和机器学习技术的最新进展，现在有几个专用系统在解释人类全身行为方面取得了令人印象深刻的结果，我们开始见证它们在日常生活中的影响，从游戏中的Microsoft Kinect到无收银员的Amazon Go杂货店。我的研究小组是最早开发算法和演示系统的团队之一，可以从深度图像分析3D人类手部姿势（2013年），以及从深度相机分析3D实验室动物行为，例如开放领域竞技场中的实验室老鼠和鱼缸中的斑马鱼（2017年）。与此同时，仍有许多障碍需要克服。** 拟议的研究计划是推进视觉行为分析的最新技术，长期目标是使机器能够正确地看到人类居住的环境并与之互动。特别是，它旨在通过调查以下两个方向来解决一些主要缺点。在第一个方向上，我们专注于设计学习算法，遵循铰接对象的几何约束，以及策划具有丰富注释的综合基准，以促进整体方法的开发。此外，我们计划研究我们的几何和基于学习的方法与最近的动态视觉相机之间的协同作用，而不是围绕传统的基于帧的相机进行详细说明，这些相机具有固有的缺点，包括延迟，内存和计算负担。在第二个方向，我们研究可能超越人类全身的重要应用。观察人类和动物的手的行为是非常有趣的。审查手语应用也是及时的，这将有助于弱势群体并产生重大的社会影响。** 该提案的潜在影响包括：（1）解决（自）遮挡以及不一致和不完整注释的缺点;（2）引入视觉行为分析的新基准;以及（3）加速传播相关实际应用的新进展，例如视觉动物行为分析和视觉手语翻译。通过该研究计划培训的HQP将获得计算机视觉和机器学习方面的研究和实践经验，这对他们未来的学术或行业职业生涯可能是有价值的。

Every day, an enormous amount of images and video clips are produced by cameras around the world, including our mobile phone cameras. The focus of attention in the images and videos is often about human individuals and their behaviours. As one might expect, the ability to interpret human visual behaviours has become a central theme in computer vision, which has indeed a wide spectrum of applications. At its root is a set of fundamental computer vision problems, including pose estimation and action detection of individuals from images and videos, among others. ******Thanks to the recent advances in camera hardware and machine learning technologies, now there are several dedicated systems with impressive results in interpreting human full-body behaviours, and we start to witness their influence in everyday life, ranging from Microsoft Kinect in gaming to the cashier-less Amazon Go grocery stores. My research group has been among the first in developing algorithms and demonstration systems in analysing 3D human hand poses from depth images (2013), as well as 3D lab animal behaviours from a depth camera, such as lab mouse in the open-field arena and zebrafish in the fish tank (2017). Meanwhile, there are still many hurdles to overcome. ******The proposed research program is to advance the state-of-the-art in visual behaviour analysis, with the long term goal of enabling machines to properly see and interact with a human inhabited environment. In particular, it aims to address some of the main shortcomings by investigating the following two directions. In the first direction, we focus on devising learning algorithms that follow the geometric constraints of articulated objects, as well as curating comprehensive benchmarks with rich annotations to facilitate the development of holistic approaches. Moreover, instead of elaborating around the traditional frame-based cameras that have the inherit drawbacks including latency, memory, and computational burdens, we plan to examine the synergism between our geometry and learning based approach and the very recent dynamic vision cameras. In the second direction, we investigate important applications that may go beyond human full-bodies. It is of strong interest to look at the behaviours of human hands, as well as animals. It is also timely to review sign language applications that would help disadvantaged people and have significant societal impacts. ******Potential impact of this proposal includes (1) addressing the shortcomings of (self-)occlusion, and inconsistent & incomplete annotations, (2) introducing new benchmark for visual behaviour analysis, and (3) expediting the dissemination of new advances in related practical applications, such as visual animal behaviour analysis and visual sign language translation. The HQP trained through this research program will gain research as well as hands-on experiences in computer vision and machine learning that could be valuable in their future academic or industry careers.