CHS: Medium: Nanophotonics Phased Array for Virtual and Augmented Reality Multifocal Displays

CHS：中：用于虚拟和增强现实多焦点显示器的纳米光子相控阵

• 批准号: 1564212
• 负责人: Amitabh Varshney
• 金额: $ 119.97万
• 依托单位: University of Maryland, College Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-07-15 至 2020-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1564212&HistoricalAwards=false
• 关键词: CHS; Medium; Nanophotonics; Phased; Array

The current explosion of virtual and augmented reality (VR and AR) technologies is the subject of much academic, industrial and popular enthusiasm. Unfortunately, these technologies have also been shown to induce a host of serious user complaints, including headaches, nausea and blurred vision. This is particularly problematic for 3D displays and content, and the National Academy of Engineering has identified "enhancing virtual reality interfaces" as a grand challenge for the 21st century. Studies have suggested that the vergence-accommodation conflict (or the decoupling of the intended "virtual focus" from the actual focal plane of a display) is the cause of many of these psychophysical problems. The PI's goal in this project is to address this mismatch by utilizing advances in silicon nanophotonics to develop a prototype natural-to-the-senses VR and AR multifocal display along with the requisite visual computing algorithms for multifocal and automultiscopic displays. The PI will also design and carry out user studies to validate the new technology (display and algorithms). Project outcomes will not only directly impact the fields of computer graphics and visualization, but an array of other fields as well. Multifocal accommodative-accurate stereo visualization of large-scale fluid flow simulations would represent a breakthrough in science and engineering generally, in areas ranging from the design of artificial arterial valves to aerial vehicles, the rational drug design process through protein docking, and in surgical advances that could take advantage of real-time MRI technologies and other high-dimensional medical imaging data. No less important are the potential educational benefits of the new technology, which would unleash the power of VR and AR to immerse young students in worlds to which they would normally not have access, providing excitement, engagement and a breadth and depth of context that is nearly impossible to achieve in a traditional classroom setting.The heart of this project lies in the design and implementation of a novel nanophotonic phased-array chip that will enable the generation of arbitrary radiation patterns with large-scale phased arrays, which would extend the functionality of phased arrays beyond conventional beam focusing and steering, communication and radar, and would open up new opportunities in image processing, 3D interactive holography, and virtual reality. This project brings together collaborators with significant expertise in 3D computer graphics and scientific visualization, in nanophotonics and plasmonics for the sub-wavelength confinement and steering of light, and in microelectronics, integrated circuits, and microstructure science and technology. The research will involve three thrusts: (a) slow light to significantly reduce the size and power requirements for the nanophotonics chip; (b) separate electronic and optical components of the chip to better optimize each; and (c) develop efficient algorithms to render and validate, through user studies, multifocal 3D graphics in the Fourier domain. The design of the multifocal display is the highest-risk and highest-gain component of the work; the team has contingency plans in place to proceed if necessary with developing multifocal and automultiscopic display algorithms, and for conducting user studies, employing existing multiple-focal-plane displays.

当前虚拟和增强现实（VR和AR）技术的爆炸式增长是学术界、工业界和大众热情的主题。 不幸的是，这些技术也被证明会引起大量严重的用户投诉，包括头痛、恶心和视力模糊。 这对于3D显示和内容来说尤其成问题，美国国家工程院已经将“增强虚拟现实界面”确定为21世纪世纪的重大挑战。 研究表明，聚散-调节冲突（或预期的“虚拟焦点”与显示器的实际焦平面的解耦）是许多这些心理物理问题的原因。 PI在该项目中的目标是通过利用硅纳米光子学的进步来解决这种不匹配问题，以开发一种原型自然的VR和AR多焦点显示器沿着必要的多焦点和自动多焦显示器的视觉计算算法。 PI还将设计和开展用户研究，以验证新技术（显示和算法）。 项目成果不仅会直接影响计算机图形和可视化领域，还会影响其他领域。 大规模流体流动模拟的多焦点精确立体可视化将代表科学和工程领域的突破，从人工动脉瓣膜的设计到飞行器，通过蛋白质对接的合理药物设计过程，以及可以利用实时MRI技术和其他高维医学成像数据的外科进步。 同样重要的是新技术的潜在教育效益，它将释放VR和AR的力量，让年轻学生沉浸在他们通常无法进入的世界中，提供兴奋，参与和上下文的广度和深度，这是几乎不可能实现在传统的课堂设置。这个项目的核心在于设计和实施一种新颖的纳米光子相-该阵列芯片将能够用大规模相控阵产生任意辐射模式，这将扩展相控阵的功能，超越传统的波束聚焦和转向、通信和雷达，并将在图像处理、3D交互式全息术和虚拟现实领域开辟新的机会。 该项目汇集了在3D计算机图形和科学可视化，纳米光子学和等离子体子波长限制和光转向，以及微电子学，集成电路和微结构科学和技术方面具有重要专业知识的合作者。 该研究将涉及三个方面：（a）减缓光，以显著降低纳米光子芯片的尺寸和功率要求;（B）分离芯片的电子和光学组件，以更好地优化每个组件;（c）开发有效的算法，通过用户研究，在傅立叶域中渲染和验证多焦点3D图形。 多焦点显示器的设计是这项工作中风险最高、收益最高的部分;该团队已经制定了应急计划，以便在必要时继续开发多焦点和自动多焦显示算法，并利用现有的多焦平面显示器进行用户研究。

项目成果

A Log-Rectilinear Transformation for Foveated 360-degree Video Streaming

• DOI: 10.1109/tvcg.2021.3067762
• 发表时间: 2021-03
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: David Li;Ruofei Du;Adharsh Babu;C. Brumar;Amitabh Varshney

Deep Depth Estimation on 360° Images with a Double Quaternion Loss

双四元数损失的 360° 图像深度估计

• DOI: 10.1109/3dv50981.2020.00062
• 发表时间: 2020
• 期刊: 2020 International Conference on 3D Vision (3DV
• 作者: Feng, Brandon Yushan;Yao, Wangjue;Liu, Zheyuan;Varshney, Amitabh
• 通讯作者: Varshney, Amitabh

Tracking-Tolerant Visual Cryptography

• DOI: 10.1109/vr.2019.8797924
• 发表时间: 2019-03
• 期刊: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)
• 作者: Ruofei Du;Eric Lee;A. Varshney

Proximity Effect on Nanophotonic Phased Arrays

纳米光子相控阵的邻近效应

• DOI: 10.1364/fio.2019.jw3a.101
• 发表时间: 2019
• 期刊: Frontiers in Optics
• 作者: Sun, Xuetong;Zhang, Yang;Huang, Po-Chun;Dagenais, Mario;Peckerar, Martin;Varshney, Amitabh
• 通讯作者: Varshney, Amitabh

Kernel Foveated Rendering

• DOI: 10.1145/3203199
• 发表时间: 2018-07
• 期刊: Proc. ACM Comput. Graph. Interact. Tech.
• 作者: Xiaoxu Meng;Ruofei Du;Matthias Zwicker;A. Varshney