Analysing visual behaviours of articulated objects from images and videos

点击翻译按钮获取中文摘要

Every day, an enormous amount of images and video clips are produced by cameras around the world, including our mobile phone cameras. The focus of attention in the images and videos is often about human individuals and their behaviours. As one might expect, the ability to interpret human visual behaviours has become a central theme in computer vision, which has indeed a wide spectrum of applications. At its root is a set of fundamental computer vision problems, including pose estimation and action detection of individuals from images and videos, among others. Thanks to the recent advances in camera hardware and machine learning technologies, now there are several dedicated systems with impressive results in interpreting human full-body behaviours, and we start to witness their influence in everyday life, ranging from Microsoft Kinect in gaming to the cashier-less Amazon Go grocery stores. My research group has been among the first in developing algorithms and demonstration systems in analysing 3D human hand poses from depth images (2013), as well as 3D lab animal behaviours from a depth camera, such as lab mouse in the open-field arena and zebrafish in the fish tank (2017). Meanwhile, there are still many hurdles to overcome. The proposed research program is to advance the state-of-the-art in visual behaviour analysis, with the long term goal of enabling machines to properly see and interact with a human inhabited environment. In particular, it aims to address some of the main shortcomings by investigating the following two directions. In the first direction, we focus on devising learning algorithms that follow the geometric constraints of articulated objects, as well as curating comprehensive benchmarks with rich annotations to facilitate the development of holistic approaches. Moreover, instead of elaborating around the traditional frame-based cameras that have the inherit drawbacks including latency, memory, and computational burdens, we plan to examine the synergism between our geometry and learning based approach and the very recent dynamic vision cameras. In the second direction, we investigate important applications that may go beyond human full-bodies. It is of strong interest to look at the behaviours of human hands, as well as animals. It is also timely to review sign language applications that would help disadvantaged people and have significant societal impacts. Potential impact of this proposal includes (1) addressing the shortcomings of (self-)occlusion, and inconsistent & incomplete annotations, (2) introducing new benchmark for visual behaviour analysis, and (3) expediting the dissemination of new advances in related practical applications, such as visual animal behaviour analysis and visual sign language translation. The HQP trained through this research program will gain research as well as hands-on experiences in computer vision and machine learning that could be valuable in their future academic or industry careers.