Event-based Vision for Robotic Scene Understanding

点击翻译按钮获取中文摘要

Robotic and autonomous systems that are capable of operating in diverse and unstructured environments hold the promise to enable a big variety of new products and impact broad areas of society. To fulfill that promise robots need to provide a broad scope of capabilities for better environment understanding. While recent years have seen remarkable results in that space, robust energy-efficient high-rate scene understanding on a broad variety of platforms in diverse application settings remains a major challenge for robotic perception. Addressing this challenge requires significant fundamental research on novel algorithms and sensing modalities. For practical applications, high computational cost and low-rate visual sensing can make operation in highly dynamic settings infeasible. Effects such as motion-blur or an insufficient update rate can prohibit the desired levels of robustness. Simply opting for a more elaborate computation and sensing setup often comes with a higher financial cost and higher energy consumption. Both limit the scope of applications reducing the potential for low-cost deployment and long-term autonomous operation. The proposed work aims to address the challenge by investigating fundamental algorithms for using event-based cameras in robotic applications. Event-based cameras are a visual sensing modality in which the sensor provides instantaneous information about brightness changes in pixels rather than full image frames at a fixed rate. This allows for much higher processing rates while at the same time reducing required energy consumption. To overcome the limitations imposed by the absence of full-frame data, this research program will investigate both, data-fusion with classical RGB cameras and operation within interactive perception pipelines. Compared to several decades of research in computer vision, event-based perception is still a young research topic with a plethora of open problems. In this work we want to address (i) fundamentals of event-based representation learning for perception and interaction, (ii) novel algorithms for event-based scene understanding such as for segmentation, object detection, and object pose-estimation in highly dynamic settings, and (iii) integration of these algorithms in robot learning and decision-making pipelines. This work program will create new theoretical foundations, algorithms, and benchmarks for using event-based cameras in robotics. Our goal is to contribute to the foundations for expanding robotic perception systems to a wider variety of platforms and devices. Additionally, to the merit of basic academic research, we will ensure transfer of technology and ideas through exchange with sensor manufacturers and industrial application partners. We will leverage Canada's strong competence in developing robotic technology while providing the innovation and personnel needed for bringing this technology to the market.