Exploiting multi-task learning for endoscopic vision in robotic surgery

博士项目的目的：学习适当的手术视觉模型，多任务执行比独立学习每个任务更好学习适当的手术视觉模型，多任务和有限的监督，以执行接近学习多任务和充分的监督，推进最先进的结合深度和光流估计，手术器械检测和解剖识别，以及手术动作识别。评估和验证基于内窥镜视觉的学习范式项目描述：多任务学习在深度学习中很常见：对于类似的任务，如检测和分割，或检测和计数，这已经实现了一个对另一个的监督。有明确的证据表明，增加一个副任务将有助于改善主要任务，但目前还不清楚这两个任务可以在这些组合中获得多少好处，特别是如果它们没有很强的相关性。出于这个原因，多个任务通常以当前方式独立处理。另一个原因在于在网络优化和实际实现方面一起学习多个任务的可扩展性。为了解决这一问题，需要仔细设计多个任务的结合，也期望学习范式的新方法。我们的目标是开发一个机器学习模型，具有机器人手术中的一般视觉智能能力，包括深度和光流估计，手术器械检测和解剖识别，以及手术动作识别。深度和光流估计以及解剖识别是开发认识手术场景的自主机器人控制方案的关键要求。考虑到与手术机器人运动链相关的不确定性以及机器人可能存在不直接操纵的工具，腹腔镜手术视频中手术器械的自动检测和跟踪对于为临床团队提供先进的手术辅助进一步发挥着重要作用。能够知道有多少和在哪里的仪器找到它的应用，如：在屏幕上放置信息覆盖;执行增强现实，而不闭塞的仪器;视觉伺服;手术任务自动化;等手术动作识别也是至关重要的，以推进自主机器人援助过程中的程序和自动审计的目的。

Aim of the PhD Project:Learning appropriate surgical vision models with multi-task to perform better than learning each task independentlyLearning appropriate surgical vision models with multi-task and limited supervision to perform close to learning with multi-task and adequate supervisionAdvancing the state of the art in combining depth and optical flow estimation, surgical instrument detection and anatomy recognition, as well as surgical action recognition.Evaluating and validating endoscopic vision-based learning paradigmsProject description:Multi-task learning is common in deep learning: For similar tasks like detection and segmentation, or detection and counting, this has already been achieved given the supervision of one for the other. There exists clear evidence that adding one side task would help the improvement of the main task, yet it is unclear how much benefits both tasks can get in these combinations, especially if they are not strongly correlated. For this reason, multiple tasks are normally processed independently in the current fashion. Another reason lies in the scalability of learning multiple tasks together in terms of both network optimization and practical implementation. To tackle this, careful designs of the conjunction of multiple tasks are needed; novel methodologies of learning paradigms are also expected.This project is placed in the endoscopic image processing domain. We aim to develop a machine learning model with general visual intelligence capacity in robotic surgery, which includes depth and optical flow estimation, surgical instrument detection and anatomy recognition, as well as surgical action recognition. Depth and optical flow estimation as well as anatomy recognition are key requirements to develop autonomous robotic control schemes that are cognizant of the surgical scene. Automatic detection and tracking of surgical instruments from laparoscopic surgery videos further plays an important role for providing advanced surgical assistance to the clinical team, given the uncertainties associated with surgical robots kinematic chains and the potential presence of tools not directly manipulated by the robot. Being able to know how many and where are the instruments finds its applications such as: placing informative overlays on the screen; performing augmented reality without occluding instruments; visual servoing; surgical task automation; etc. Surgical action recognition is also critical to advance autonomous robotic assistance during the procedure and for automated auditing purposes.