FW-HTF: Collaborative Research: Enhancing Human Capabilities through Virtual Personal Embodied Assistants in Self-Contained Eyeglasses-Based Augmented Reality (AR) Systems

FW-HTF：协作研究：通过基于独立眼镜的增强现实 (AR) 系统中的虚拟个人助理增强人类能力

• 批准号: 1840131
• 负责人: Henry Fuchs
• 金额: $ 219万
• 依托单位: University of North Carolina at Chapel Hill
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-15 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1840131&HistoricalAwards=false
• 关键词: FW; HTF; Collaborative; Research; Enhancing

The Future of Work at the Human-Technology Frontier (FW-HTF) is one of 10 new Big Ideas for Future Investment announced by NSF. The FW-HTF cross-directorate program aims to respond to the challenges and opportunities of the changing landscape of jobs and work by supporting convergent research. This award fulfills part of that aim. This award supports basic research underpinning development of an eyeglass-based 3D mobile telepresence system with integrated virtual personal assistant. This technology will increase worker productivity and improve skills. The system automatically adjusts visual focus and places virtual elements in the image without eye strain. The user will be able to communicate to the system by speech. The system also uses sensors to keep track of the user's surroundings and provide the relevant information to the user automatically. The project will explore two of the many possible uses of the system: amplifying a workers capabilities (such as a physical therapist interacting with a remote patient), and accelerating post-injury return to work through telepresence (such as a burn victim reintegrating into his/her workplace). The project will advance the national interest by allowing the right person to be virtually in the right place at the right time. The project also includes an education and outreach component wherein undergraduate and graduate students shall receive training in engineering and research methods. Course curriculum at Stanford University and the University of North Carolina at Chapel Hill shall be updated to include project-related content and examples. This project comprises fundamental research activities needed to develop an embodied Intelligent Cognitive Assistant (GLASS-X) that will amplify the capabilities of workers in a way that will increase productivity and improve quality of life. GLASS-X is conceived of as an eyeglass-based 3D mobile telepresence system with integrated virtual personal assistant. Methods include: body and environment reconstruction (situation awareness) from a fusion of images provided by an eyeglass frame-based camera array and limb motion data provided by inertial measurement units; fundamental research on adaptive focus displays capable to reduce eye strain when using augmented reality displays; dialog-based communication with a virtual personal assistant, including transformations from visual input to dialog and vice versa; human subject evaluations of GLASS-X technology in two workplace domains (remote interactions between a physical therapist and his/her patient; burn survivor remote return-to-work). This research promises to push the state of the art in core areas including: computer vision; augmented reality; accommodating displays; and natural language and dialogue models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人类技术前沿的未来工作（FW-HTF）是NSF宣布的10个未来投资新想法之一。FW-HTF跨董事会计划旨在通过支持融合研究来应对不断变化的就业和工作环境的挑战和机遇。这个奖项实现了这个目标的一部分。该奖项支持基于眼镜的3D移动的远程呈现系统与集成的虚拟个人助理的开发基础研究。这项技术将提高工人的生产力和技能。该系统自动调整视觉焦点，并在图像中放置虚拟元素，而不会造成眼睛疲劳。 用户将能够通过语音与系统进行通信。 该系统还使用传感器来跟踪用户的周围环境，并自动向用户提供相关信息。 该项目将探索该系统的多种可能用途中的两种：增强工作人员的能力（例如物理治疗师与远程患者互动），并通过远程呈现加速受伤后重返工作岗位（例如烧伤受害者重新融入他/她的工作场所）。该项目将通过允许合适的人在正确的时间出现在正确的地点来促进国家利益。该项目还包括一个教育和外联部分，本科生和研究生将接受工程和研究方法方面的培训。应更新斯坦福大学和查佩尔山的北卡罗来纳州大学的课程设置，以纳入项目相关内容和示例。该项目包括开发嵌入式智能认知助理（GLASS-X）所需的基础研究活动，该智能认知助理将以提高生产力和改善生活质量的方式增强工人的能力。GLASS-X被认为是一个基于眼镜的3D移动的远程呈现系统，集成了虚拟个人助理。方法包括：身体与环境重建（态势感知）来自由基于双帧的相机阵列提供的图像和由惯性测量单元提供的肢体运动数据的融合;对能够在使用增强现实显示器时减少眼睛疲劳的自适应聚焦显示器的基础研究;与虚拟个人助理的基于对话的通信，包括从视觉输入到对话的转换，反之亦然; GLASS-X技术在两个工作场所领域（物理治疗师和他/她的病人之间的远程交互;烧伤幸存者远程返回工作岗位）的人类受试者评估。该研究有望推动核心领域的最新技术水平，包括：计算机视觉、增强现实、可调节显示器以及自然语言和对话模型。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Towards Fully Mobile 3D Face, Body, and Environment Capture Using Only Head-worn Cameras

• DOI: 10.1109/tvcg.2018.2868527
• 发表时间: 2018-09
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Young-Woon Cha;True Price;Zhen Wei;Xinran Lu;Nicholas Rewkowski;Rohan Chabra;Zihe Qin;Hyounghun Kim;Zhaoqi Su;Yebin Liu;A. Ilie;A. State;Zhenlin Xu;Jan-Michael Frahm;H. Fuchs

ZenCam: Context-Driven Control of Autonomous Body Cameras

• DOI: 10.1109/dcoss.2019.00029
• 发表时间: 2019-05
• 期刊: 2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS)
• 作者: Shiwei Fang;Ketan Mayer-Patel;S. Nirjon

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

• 发表时间: 2021-09
• 作者: Youngjoon Kwon;Dahun Kim;Duygu Ceylan;H. Fuchs

ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments

• DOI: 10.18653/v1/2020.findings-emnlp.348
• 发表时间: 2020-11
• 作者: Hyounghun Kim;Abhaysinh Zala;Graham Burri;Hao Tan;Mohit Bansal

FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality

• DOI: 10.1109/tvcg.2022.3203102
• 发表时间: 2021-03
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Nianchen Deng;Zhenyi He;Jiannan Ye;Budmonde Duinkharjav;Praneeth Chakravarthula;Xubo Yang;Qi Sun