Event-based Vision for Robotic Scene Understanding

能够在多样化和非结构化环境中运行的机器人和自主系统有望实现各种各样的新产品，并影响广泛的社会领域。为了实现这一承诺，机器人需要提供更广泛的能力来更好地理解环境。虽然近年来在该领域取得了显著的成果，但在各种应用环境下的各种平台上实现高效节能的高速率场景理解仍然是机器人感知的主要挑战。解决这一挑战需要对新算法和传感模式进行重要的基础研究。在实际应用中，高计算成本和低速率的视觉感知使得在高动态环境下的操作变得不可行。诸如运动模糊或更新速率不足等影响可能会妨碍理想的鲁棒性水平。简单地选择更复杂的计算和传感设置通常会带来更高的财务成本和更高的能耗。两者都限制了应用范围，降低了低成本部署和长期自主操作的潜力。提出的工作旨在通过研究在机器人应用中使用基于事件的相机的基本算法来解决这一挑战。基于事件的相机是一种视觉感知方式，其中传感器以像素为单位提供亮度变化的瞬时信息，而不是以固定速率提供完整的图像帧。这允许更高的处理率，同时减少所需的能源消耗。为了克服缺乏全帧数据所带来的限制，本研究计划将研究与经典RGB相机的数据融合和交互式感知管道中的操作。与计算机视觉几十年的研究相比，基于事件的感知仍然是一个年轻的研究课题，有大量的开放性问题。在这项工作中，我们希望解决(i)基于事件的感知和交互表示学习的基础知识，（ii）基于事件的场景理解的新算法，如高度动态设置中的分割，对象检测和对象姿态估计，以及（iii）将这些算法集成到机器人学习和决策管道中。这项工作计划将为在机器人技术中使用基于事件的相机创造新的理论基础、算法和基准。我们的目标是为将机器人感知系统扩展到更广泛的平台和设备奠定基础。此外，为了基础学术研究的价值，我们将通过与传感器制造商和工业应用合作伙伴的交流，确保技术和思想的转移。我们将利用加拿大在开发机器人技术方面的强大能力，同时提供将这项技术推向市场所需的创新和人才。

Robotic and autonomous systems that are capable of operating in diverse and unstructured environments hold the promise to enable a big variety of new products and impact broad areas of society. To fulfill that promise robots need to provide a broad scope of capabilities for better environment understanding. While recent years have seen remarkable results in that space, robust energy-efficient high-rate scene understanding on a broad variety of platforms in diverse application settings remains a major challenge for robotic perception. Addressing this challenge requires significant fundamental research on novel algorithms and sensing modalities. For practical applications, high computational cost and low-rate visual sensing can make operation in highly dynamic settings infeasible. Effects such as motion-blur or an insufficient update rate can prohibit the desired levels of robustness. Simply opting for a more elaborate computation and sensing setup often comes with a higher financial cost and higher energy consumption. Both limit the scope of applications reducing the potential for low-cost deployment and long-term autonomous operation. The proposed work aims to address the challenge by investigating fundamental algorithms for using event-based cameras in robotic applications. Event-based cameras are a visual sensing modality in which the sensor provides instantaneous information about brightness changes in pixels rather than full image frames at a fixed rate. This allows for much higher processing rates while at the same time reducing required energy consumption. To overcome the limitations imposed by the absence of full-frame data, this research program will investigate both, data-fusion with classical RGB cameras and operation within interactive perception pipelines. Compared to several decades of research in computer vision, event-based perception is still a young research topic with a plethora of open problems. In this work we want to address (i) fundamentals of event-based representation learning for perception and interaction, (ii) novel algorithms for event-based scene understanding such as for segmentation, object detection, and object pose-estimation in highly dynamic settings, and (iii) integration of these algorithms in robot learning and decision-making pipelines. This work program will create new theoretical foundations, algorithms, and benchmarks for using event-based cameras in robotics. Our goal is to contribute to the foundations for expanding robotic perception systems to a wider variety of platforms and devices. Additionally, to the merit of basic academic research, we will ensure transfer of technology and ideas through exchange with sensor manufacturers and industrial application partners. We will leverage Canada's strong competence in developing robotic technology while providing the innovation and personnel needed for bringing this technology to the market.