Novel human-computer interaction paradigms for surgical augmented reality

增强现实（AR）技术能够将视觉信息叠加到我们环境中的真实物体上，以促进某些任务的执行。通过这样做，可以从根本上改进几个工作环境中的工作流。尽管最近取得了一些进展，但在精度、准确性和安全性至关重要的领域，AR仍然在很大程度上缺席。拟议的研究旨在消除采用AR的技术障碍，以用于计算机辅助手术的具体应用。每个AR应用程序都可以通过其三个关键处理步骤来描述其特征：3D虚拟模型的渲染，虚拟模型和真实图像的对齐以及渲染模型与现实的混合。为了激发AR在特定应用领域的应用，必须分析每个处理步骤的感知和物理准确性，并经常进行改进。在本研究计划中，我们将制定新的人机交互范例，利用用户的专业知识来解决外科AR中每个处理步骤的准确性，这是一个应用领域，几乎所有不准确的来源都被举例说明。更具体地说，我们将1)使用廉价的虚拟现实头戴式显示器和手部跟踪设备作为基础，设计和实施新的交互式方法，促进3D医学图像的探索（渲染组件），2)提出交互式半自动技术，以纠正AR图像的真实和虚拟组件（对齐组件）的错位，3)设计和实现渲染方法，使AR图像的真实和虚拟组件（混合组件）能够在感知上融合。在研究计划结束时，我们将获得一系列解决方案，以提高创建AR图像所涉及的每个处理步骤的物理和感知准确性，以指导外科手术。更重要的是，我们将定义一个框架，在解决AR的新应用领域时，分析和纠正最重要的可用性问题。

The technology of augmented reality (AR) enables the superimposition of visual information onto real objects in our environment in order to facilitate the execution of certain tasks. By doing so, it is possible to radically improve the workflow in several work environments. Despite recent progress, AR is still largely absent from fields where precision, accuracy and safety are critical. The proposed research aims to remove the technological barriers to the adoption of AR for the specific application of computer-assisted surgery. Every AR application can be characterized by its approach to three key processing steps: the rendering of 3D virtual models, the alignment of virtual models and real images and the blending of rendered models with reality. To spark adoption of AR for a specific application domain, the perceptual and physical accuracy of each processing step must be analysed and often, improved. In this research program, we will formulate novel human-computer interaction paradigms that take advantage of the user's expertise to address the accuracy of every processing step in surgical AR, an application domain where nearly all sources of inaccuracies are exemplified. More specifically, we will 1) use inexpensive virtual reality head-mounted displays and hand tracking devices as a basis to design and implement new interactive methods that facilitate the exploration of 3D medical images (rendering component), 2) propose interactive semi-automatic techniques to correct for the misalignment of real and virtual components of an AR image (aligning component) and, 3) design and implement rendering methods that enable the perceptually sound fusion of real and virtual components of an AR image (blending component). At the end of the research program we will obtain a series of solutions to improve the physical and perceptual accuracy of each of the processing steps involved in the creation of AR images to guide surgical operations. More importantly, we will have defined a framework to analyse and correct the most important usability issues when addressing a new application domain for AR.