HCC: Small: Super Reality -- Immersive Visual Interfaces with No Line-of-Sight Restriction

该项目开发了超级现实，这是一种新型的人机视觉界面，旨在高效和有效地探索3D数据集。超现实（SR）界面建立在传统的虚拟现实界面，保留其沉浸感，直观性和深度提示的好处，同时从根本上增加了显示给用户的图像的信息带宽。 SR界面通过消除传统虚拟现实界面的视线限制，让用户从当前位置看到更多的数据集。用户不仅可以看到从当前视点可见的数据集区域，还可以看到隐藏在遮挡物后面的区域。用户可以从当前位置预览被遮挡的数据集区域，而不必在遮挡物周围导航，从而避免当被遮挡区域被证明是不感兴趣的时浪费的导航。用户可以在同一图像中直接比较远处的数据集区域，而不必依赖于先前看到的数据集区域的记忆。使用SR界面，用户可以从多个视角并行探索数据集，但没有基于传统图像矩阵的复杂可视化界面的负担。通过3D数据集探索效率和有效性的飞跃式增长，该项目将导致基础层面的创新，影响数据可视化的使用领域。这种SR界面有可能最大限度地减少用户定向障碍和晕电病，这是由传统虚拟现实界面中不匹配的用户运动和用户视觉感知触发的。该项目将由不同的本科生参与研究，目标是设计和开发超级现实（SR）界面，通过将3D数据集投影到2D图像平面上的计算相机的泛化来消除视线限制。传统摄像机的线性射线被弯曲射线所取代，弯曲射线绕着遮挡器路由，以将用户连接到感兴趣的数据集区域。超级现实界面本质上是派遣一个虚拟下级用户团队，每个用户从自己的视角对数据集进行成像，并向用户呈现从多个视角捕获的数据的集成。构建这些集成图像以满足多个设计要求：（a）图像连续性，通过限制相邻射线的轨迹之间的差异;（B）图像非冗余性，通过避免射线交叉;（c）自适应图像分辨率，通过将更多射线路由到更高复杂性或重要性的数据集区域;（d）通过投影在没有遮挡器的情况下用户将看到的被遮挡区域，就地去除遮挡;快速渲染，通过设计具有快速投影操作的通用相机。SR界面将被设计为使用传统的第一人称视图将这些集成图像的一部分呈现给用户。这样，帧的对应于附近几何形状的部分响应于用户的视图改变，如用户期望的那样，锚定用户以避免迷失方向和晕电脑。此外，SR接口限制了用户需要操纵的自由度的数量，以便展开和缩回所需的解除咬合效果。超级现实界面在基于客观和主观指标的受控用户研究中进行了反复改进和验证，这些指标量化了设计要求的满足程度，以及3D数据集探索的效率和有效性。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

The project develops Super Reality, a new class of human-computer visual interfaces designed for efficient and effective 3D dataset exploration. Super Reality (SR) interfaces build upon conventional virtual reality interfaces, preserving their benefits of immersion, intuitiveness, and depth cues, while radically increasing the information bandwidth of the images shown to the user. SR interfaces let the user see more of the dataset from the current location by removing the line-of-sight restriction of conventional virtual reality interfaces. The user sees not only the regions of the dataset that are visible from the current viewpoint, but also regions that are hidden behind occluders. The user can preview an occluded dataset region from the current position, without having to navigate around occluders, thereby avoiding the wasted navigation when the occluded region proves to be of no interest. The user can compare distant dataset regions directly, in the same image, without having to rely on the memory of dataset regions seen earlier. With a SR interface, the user explores the dataset in parallel, from multiple viewpoints, but without the burden of a complex visualization interface based on a matrix of conventional images. Through the leap-forward increase of 3D dataset exploration efficiency and effectiveness, this project will lead to a foundational level innovation influencing the domains where data visualization is used. This SR interface has the potential to minimize user disorientation and cybersickness, which are triggered by mismatched user motion and user visual perception in the traditional virtual reality interfaces. The project will be carried out by involving a diverse group of undergraduate students in research.The goal of the project is the design and development of Super Reality (SR) interfaces that will remove the line-of-sight restriction through a generalization of the computational camera that projects the 3D dataset onto the 2D image plane. The linear rays of conventional cameras are replaced with curved rays that are routed around occluders to connect the user to dataset regions of interest. Super Reality interfaces essentially dispatch a team of virtual subordinate users that each image the dataset from their own viewpoint and present the user with an integration of the data captured from the multiple viewpoints. These integrated images are constructed to meet multiple design requirements: (a) image continuity, by bounding the difference between trajectories of neighboring rays; (b) image non-redundancy, by avoiding ray intersections; (c) adaptive image resolution, by routing more rays towards dataset regions of higher complexity or importance; (d) in-place disocclusion, by projecting the disoccluded region where the user would see it in the absence of the occluder; (e) minimal image distortion, by limiting the curvature of the non-linear rays; fast rendering, by designing the generalized cameras with a fast projection operation. The SR interfaces will be designed to render the part of these integrated images close to the user with the conventional, first-person view. This way, the part of the frame that corresponds to nearby geometry responds to the user’s view changes as the user expects, anchoring the user to avoid disorientation and cybersickness. Also, SR interface limits the number of degrees of freedom the user needs to manipulate in order to deploy and retract the desired disocclusion effects. The Super Reality interfaces are iteratively refined and validated in controlled user studies based on objective and subjective metrics that quantify the degree to which the design requirements are met, as well as the efficiency and effectiveness of 3D dataset exploration.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.