Analysing visual behaviours of articulated objects from images and videos

每天，世界各地的相机都会产生大量的图像和视频片段，包括我们的手机摄像头。图像和视频的关注焦点通常是关于人类个体及其行为的。正如人们所期望的那样，解释人类视觉行为的能力已经成为计算机视觉的中心主题，它确实有广泛的应用。其根源是一系列基本的计算机视觉问题，包括从图像和视频中对个体进行姿态估计和动作检测等。由于最近相机硬件和机器学习技术的进步，现在有几个专用系统在解释人类全身行为方面取得了令人印象深刻的成果，我们开始见证它们在日常生活中的影响，从游戏中的微软Kinect到没有收银员的亚马逊Go杂货店。我的研究小组是最早开发算法和演示系统的团队之一，用于分析来自深度图像的3D人手姿势（2013年），以及来自深度相机的3D实验动物行为，例如开放场地舞台上的实验室老鼠和鱼缸里的斑马鱼（2017年）。与此同时，仍有许多障碍需要克服。拟议的研究计划是推进视觉行为分析的最新技术，其长期目标是使机器能够正确地看到人类居住的环境并与之互动。特别是，它旨在通过调查以下两个方向来解决一些主要缺点。在第一个方向上，我们专注于设计遵循铰接对象几何约束的学习算法，以及策划具有丰富注释的综合基准，以促进整体方法的开发。此外，我们没有详细阐述传统的基于帧的相机的缺点，包括延迟、内存和计算负担，我们计划研究基于几何和学习的方法与最新的动态视觉相机之间的协同作用。在第二个方向上，我们研究可能超越人体全身的重要应用。观察人的手和动物的手的行为是很有趣的。审查手语的应用也是及时的，它将帮助弱势群体并产生重大的社会影响。该提案的潜在影响包括：(1)解决了（自我）遮挡和不一致和不完整注释的缺点，(2)引入了视觉行为分析的新基准，(3)加速了相关实际应用的新进展的传播，例如视觉动物行为分析和视觉手语翻译。通过该研究项目培训的HQP将获得计算机视觉和机器学习方面的研究和实践经验，这对他们未来的学术或工业生涯可能很有价值。

Every day, an enormous amount of images and video clips are produced by cameras around the world, including our mobile phone cameras. The focus of attention in the images and videos is often about human individuals and their behaviours. As one might expect, the ability to interpret human visual behaviours has become a central theme in computer vision, which has indeed a wide spectrum of applications. At its root is a set of fundamental computer vision problems, including pose estimation and action detection of individuals from images and videos, among others. Thanks to the recent advances in camera hardware and machine learning technologies, now there are several dedicated systems with impressive results in interpreting human full-body behaviours, and we start to witness their influence in everyday life, ranging from Microsoft Kinect in gaming to the cashier-less Amazon Go grocery stores. My research group has been among the first in developing algorithms and demonstration systems in analysing 3D human hand poses from depth images (2013), as well as 3D lab animal behaviours from a depth camera, such as lab mouse in the open-field arena and zebrafish in the fish tank (2017). Meanwhile, there are still many hurdles to overcome. The proposed research program is to advance the state-of-the-art in visual behaviour analysis, with the long term goal of enabling machines to properly see and interact with a human inhabited environment. In particular, it aims to address some of the main shortcomings by investigating the following two directions. In the first direction, we focus on devising learning algorithms that follow the geometric constraints of articulated objects, as well as curating comprehensive benchmarks with rich annotations to facilitate the development of holistic approaches. Moreover, instead of elaborating around the traditional frame-based cameras that have the inherit drawbacks including latency, memory, and computational burdens, we plan to examine the synergism between our geometry and learning based approach and the very recent dynamic vision cameras. In the second direction, we investigate important applications that may go beyond human full-bodies. It is of strong interest to look at the behaviours of human hands, as well as animals. It is also timely to review sign language applications that would help disadvantaged people and have significant societal impacts. Potential impact of this proposal includes (1) addressing the shortcomings of (self-)occlusion, and inconsistent & incomplete annotations, (2) introducing new benchmark for visual behaviour analysis, and (3) expediting the dissemination of new advances in related practical applications, such as visual animal behaviour analysis and visual sign language translation. The HQP trained through this research program will gain research as well as hands-on experiences in computer vision and machine learning that could be valuable in their future academic or industry careers.