CHS: Small: Direct physical grasping, manipulation, and tooling of simulated objects

本项目将发展并评估一种物理抓取、操纵和应用工具来模拟物体的方法，以及一种探索三维信息的方法。该项目建立在工具使用的概念上，其中手持工具对象用于修改目标对象的属性或外观。该项目整合了人机交互和可视化方面的广泛发现，从手动和有形用户界面到增强现实。这项研究的核心是一个桌面增强现实系统，该系统具有立体或单视显示器、触觉指向设备和一个聚焦于用户手部的摄像头。用户一只手拿着包含虚拟对象的物理线框立方体，另一只手拿着指向设备，其尖端在视觉上增强以显示其功能，这将与几种可能的工具之一相关，包括：(a)指向、选择和移动对象的探针，(b)用于过滤、重新编码和详细说明信息的放大或语义透镜，以及(c)显示切片或投影视图的切割平面。在显示器上，用户可以看到他们使用工具对模拟对象的操作产生的即时、直接的影响。该系统将以流畅自然的交互技术支持可视化，在一定程度上提高用户探索和理解3D物体的能力，就像他们把物体握在手中一样。该项目将产生跨多个学科的理论和实践进步，从而带来社会效益，比如对心理学理论和计算机算法的更深入理解，这些理论和算法需要给人一种无缝的印象，即他或她正在操纵一个物体，这个物体看起来就在他或她面前，但实际上只是虚拟的。将这项工作的实际成果转化为低成本的硬件组件（如头戴式3D显示器）应该是相对简单的，这样该项目最终可以为日常用户提供一种直接创建和修改他们心目中的3D打印对象的方法。大学生将帮助开发和评估该系统，从而使这些学生接触到一种新的、潜在的变革性方法，即使用物理工具增强模拟空间，从而为更多的学生提供令人兴奋的动手学习计算机科学的机会。骨科医生已被招募来协助探索该系统在术前手术计划中的使用，例如允许探索复杂的3D骨结构。在这个实际操作的增强现实系统中开发和评估新的交互技术将导致更好地理解如何通过增强工具的使用来提高数据探索的性能。研究将探讨：用户行为与系统效果的空间搭配；手、物体和环境之间的物理限制；本体感觉的作用更大。实验结果将深入了解人机交互和可视化研究中相对不同领域之间的问题，特别是在双手任务中本体感觉的潜在好处，机械约束和稳定在多大程度上可以提高精确交互任务的性能，以及这些因素如何补偿与在立体或单镜显示器上呈现3D信息相关的视觉误差。这项研究将从实验室扩展到一个真实而重要的医学领域，这将有助于在实践中验证我们的工作。

This project will develop and evaluate an approach to physical grasping, manipulating, and applying tools to simulated objects, and a means of exploring of three-dimensional information. The project is founded on the concept of tool use, in which handheld tool objects are used to modify the properties or appearance of target objects. The project integrates a wide range of findings in human-computer interaction and visualization, from bimanual and tangible user interfaces to augmented reality. At the core of this research endeavor is a desktop augmented reality system with a stereoscopic or monoscopic display, a haptic pointing device, and a camera focused on the user's hands. In one hand the user holds a physical wireframe cube that contains virtual objects, and in the other hand the pointing device, its tip visually augmented to show its function, which will relate to one of several possible tools, including: (a) a probe for pointing at, selecting, and moving objects, (b) a magnifying or semantic lens for filtering, recoding, and elaborating information, and (c) a cutting plane that shows slices or projection views. On the display, users watch the immediate, direct effects of their actions with the tools on the simulated object. The system will support visualization with fluid and natural interaction techniques, improving the ability of users to explore and understand 3D objects to some extent as if they were holding the objects in their hands. The project will provide societal benefits by generating theoretical and practical advances across multiple disciplines, such as a deeper understanding of the psychological theory and computer algorithms that are needed to give a person a seamless impression that he or she is manipulating an object that appears to be physically in front of the person, but which is only there virtually. It should be relatively straightforward to transition the practical outcome of this work to low-cost hardware components such as head-mounted 3D displays, such that the project could ultimately provide everyday users with a means of directly creating and modifying objects that they have in mind for 3D printing. University students will help to develop and evaluate the system, thus exposing these students to a new and potentially transformative approach to augmenting a simulated space with physical tools, and thus providing a larger population of students with opportunities for exciting hands-on computer science learning. Orthopedic surgeons have been recruited to assist in exploring the use of the system for preoperative surgical planning such as by permitting the exploration of complex 3D bone structures.Development and evaluation of the novel interaction techniques in this hands-on augmented reality system will lead to a better understanding of how performance may be improved in data exploration with augmented tool use. The research will investigate: the spatial collocation of user actions and system effects; physical constraints between the hands, objects, and the environment; and a greater role for proprioception. Experimental results will give insight into questions that cross the boundaries between relatively disparate areas of research in HCI and visualization, specifically the potential benefits of proprioception in bimanual tasks, the extent to which mechanical constraints and stabilization can improve performance in precise interaction tasks, and how these factors may compensate for the visual errors associated with presenting 3D information on a stereo or monoscopic display. The research will extend beyond the laboratory to a real and important medical domain, which will help validate our work in practice.