Novel human-computer interaction paradigms for surgical augmented reality

增强现实(AR)技术能够将视觉信息叠加到我们所处环境中的真实对象上，以便于执行某些任务。通过这样做，可以从根本上改进多个工作环境中的工作流程。尽管最近取得了进展，但在精确度、准确性和安全性至关重要的领域，AR仍在很大程度上缺席。这项拟议的研究旨在消除在计算机辅助手术的具体应用中采用AR的技术障碍。每个AR应用程序都可以通过三个关键处理步骤来表征：3D虚拟模型的渲染，虚拟模型和真实图像的对齐，以及渲染模型与现实的混合。为了促进AR在特定应用领域的采用，必须分析并经常改进每个处理步骤的感知和物理准确性。在这个研究计划中，我们将制定新的人机交互范例，利用用户的专业知识来解决外科AR中每个处理步骤的准确性，这是一个几乎所有不准确来源都得到例证的应用领域。更具体地说，我们将1)使用廉价的虚拟现实头盔显示器和手部跟踪设备作为基础来设计和实现促进3D医学图像探索的新的交互方法(渲染组件)，2)提出交互式半自动技术来校正AR图像的实部和虚部的错位(对准组件)，以及3)设计和实现能够实现AR图像的实部和虚部的感知声音融合的渲染方法(混合部)。在研究计划结束时，我们将获得一系列解决方案，以提高AR图像创建过程中每个处理步骤的物理和感知精度，以指导外科手术。更重要的是，我们将定义一个框架来分析和纠正在解决AR的新应用领域时最重要的可用性问题。

The technology of augmented reality (AR) enables the superimposition of visual information onto real objects in our environment in order to facilitate the execution of certain tasks. By doing so, it is possible to radically improve the workflow in several work environments. Despite recent progress, AR is still largely absent from fields where precision, accuracy and safety are critical. The proposed research aims to remove the technological barriers to the adoption of AR for the specific application of computer-assisted surgery. Every AR application can be characterized by its approach to three key processing steps: the rendering of 3D virtual models, the alignment of virtual models and real images and the blending of rendered models with reality. To spark adoption of AR for a specific application domain, the perceptual and physical accuracy of each processing step must be analysed and often, improved. In this research program, we will formulate novel human-computer interaction paradigms that take advantage of the user's expertise to address the accuracy of every processing step in surgical AR, an application domain where nearly all sources of inaccuracies are exemplified. More specifically, we will 1) use inexpensive virtual reality head-mounted displays and hand tracking devices as a basis to design and implement new interactive methods that facilitate the exploration of 3D medical images (rendering component), 2) propose interactive semi-automatic techniques to correct for the misalignment of real and virtual components of an AR image (aligning component) and, 3) design and implement rendering methods that enable the perceptually sound fusion of real and virtual components of an AR image (blending component). At the end of the research program we will obtain a series of solutions to improve the physical and perceptual accuracy of each of the processing steps involved in the creation of AR images to guide surgical operations. More importantly, we will have defined a framework to analyse and correct the most important usability issues when addressing a new application domain for AR.