CAREER: Digital Twins of Surgical Environments for Situational Awareness and Immersive Simulation

开发机器人辅助手术的新工具和程序既困难又昂贵。这是因为甚至在临床测试开始之前，就需要特殊用途的工具和环境来构建和测试原型手术机器人。该项目旨在通过开发图像引导手术的“数字双胞胎”来解决这些问题。高层次的想法是开发虚拟现实仿真和计算模型，密切模仿真实的病人，手术工具和操作环境。这些模拟将帮助设计师尽早纠正机器人和程序设计中可能出现的问题；这反过来将使更快，更便宜的设计周期，将使机器人辅助手术设计更实用和广泛使用。模拟还可以用于支持实际操作；房间大小的手术环境模型将支持算法来监控机器人系统，以实现更安全的手术，并为外科医生和公众提供新的机会，让他们了解如何利用技术来推进手术。该项目的技术目标分为两个重点：在第一个重点中，团队将开发创新的方法，使创建虚拟沉浸式和交互式模拟环境能够精确地再现现实世界的场景。为了构建这些所谓的数字双胞胎，该团队将创建计算机视觉和机器学习技术，以从多模态测量中创建、更新和利用手术环境的数字双胞胎。第二个重点将使用非结构化、人在环环境的数字孪生模型，探索如何使用这些沉浸式仿真模型来加速智能手术系统的设计，并提供补充的态势感知，以增强决策。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Developing new tools and procedures for robot-assisted surgery is difficult and expensive. This is because of the need for special-purpose tools and environments for building and testing prototype surgical robots, even before clinical testing can begin. This project seeks to address these problems through developing “digital twins” of image-guided surgery. The high level idea is to develop virtual reality simulations and computational models that closely mimic real patients, surgical tools, and operating environments. These simulations will help designers correct possible problems with robot and procedure designs early on; this in turn will allow faster, cheaper design cycles that will make robot-assisted surgery design more practical and widely available. The simulations can also be used to support real operations; room-scale models of the operating environment will support algorithms to monitor the robotic systems for safer surgeries, as well as provide new opportunities for educating both surgeons and the general public about ways technology is used to advance surgery.The technical aims of this project are divided into two thrusts: In the first thrust, the team will develop innovative methods that enable the creation of virtual immersive and interactive simulation environments that precisely recreate scenarios of the real world. To build these so-called digital twins, the team will create computer vision and machine learning techniques to create, update, and leverage digital twins of the surgical environment from multi-modal measurements. The second thrust will use digital twin models of unstructured, human-in-the-loop environments to explore how these immersive simulation models may be used to accelerate the design of intelligent surgical systems and provide complementary situational awareness to enhance decision making.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.