HCC: Small: Super Reality -- Immersive Visual Interfaces with No Line-of-Sight Restriction

点击翻译按钮获取中文摘要

The project develops Super Reality, a new class of human-computer visual interfaces designed for efficient and effective 3D dataset exploration. Super Reality (SR) interfaces build upon conventional virtual reality interfaces, preserving their benefits of immersion, intuitiveness, and depth cues, while radically increasing the information bandwidth of the images shown to the user. SR interfaces let the user see more of the dataset from the current location by removing the line-of-sight restriction of conventional virtual reality interfaces. The user sees not only the regions of the dataset that are visible from the current viewpoint, but also regions that are hidden behind occluders. The user can preview an occluded dataset region from the current position, without having to navigate around occluders, thereby avoiding the wasted navigation when the occluded region proves to be of no interest. The user can compare distant dataset regions directly, in the same image, without having to rely on the memory of dataset regions seen earlier. With a SR interface, the user explores the dataset in parallel, from multiple viewpoints, but without the burden of a complex visualization interface based on a matrix of conventional images. Through the leap-forward increase of 3D dataset exploration efficiency and effectiveness, this project will lead to a foundational level innovation influencing the domains where data visualization is used. This SR interface has the potential to minimize user disorientation and cybersickness, which are triggered by mismatched user motion and user visual perception in the traditional virtual reality interfaces. The project will be carried out by involving a diverse group of undergraduate students in research.The goal of the project is the design and development of Super Reality (SR) interfaces that will remove the line-of-sight restriction through a generalization of the computational camera that projects the 3D dataset onto the 2D image plane. The linear rays of conventional cameras are replaced with curved rays that are routed around occluders to connect the user to dataset regions of interest. Super Reality interfaces essentially dispatch a team of virtual subordinate users that each image the dataset from their own viewpoint and present the user with an integration of the data captured from the multiple viewpoints. These integrated images are constructed to meet multiple design requirements: (a) image continuity, by bounding the difference between trajectories of neighboring rays; (b) image non-redundancy, by avoiding ray intersections; (c) adaptive image resolution, by routing more rays towards dataset regions of higher complexity or importance; (d) in-place disocclusion, by projecting the disoccluded region where the user would see it in the absence of the occluder; (e) minimal image distortion, by limiting the curvature of the non-linear rays; fast rendering, by designing the generalized cameras with a fast projection operation. The SR interfaces will be designed to render the part of these integrated images close to the user with the conventional, first-person view. This way, the part of the frame that corresponds to nearby geometry responds to the user’s view changes as the user expects, anchoring the user to avoid disorientation and cybersickness. Also, SR interface limits the number of degrees of freedom the user needs to manipulate in order to deploy and retract the desired disocclusion effects. The Super Reality interfaces are iteratively refined and validated in controlled user studies based on objective and subjective metrics that quantify the degree to which the design requirements are met, as well as the efficiency and effectiveness of 3D dataset exploration.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.