CAREER: Performance-Guided Synthesis of Virtual Environments for Personalized Training

职业：用于个性化培训的虚拟环境的性能引导综合

• 批准号: 1942531
• 负责人: Lap Fai Yu
• 金额: $ 53.33万
• 依托单位: George Mason University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-07-01 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1942531&HistoricalAwards=false
• 关键词: CAREER; Performance; Guided; Synthesis; Virtual

Virtual Reality (VR) promises to provide a compelling, effective, and convenient means for training and reskilling the workforce. The objective of this project is to devise a computational design framework for guiding the synthesis of personalized virtual training environments for human performance. Designers and general users can intuitively apply this framework for synthesizing a variety of immersive and engaging VR training scenarios in a fully automatic, fast, scalable, and low-cost manner. As a showcase of the proposed framework, the project will demonstrate how to synthesize personalized virtual training environments for workforce training, such as safety inspection training for employees, workplace supervisors, and safety inspectors. This research project offers a novel interdisciplinary perspective for generating personalized VR training content by bringing together expertise and insights from human-computer interaction, virtual reality, computer graphics, machine learning, and optimization, as well as domain expertise in instructional design, safety, emergency management, and engineering. Designers and general users can use this framework to synthesize personalized VR training content for a variety of scenarios such as rehabilitation, safety training, disaster response training, as well as workforce training for different domains such as manufacturing, construction, logistics, transportation, retail management, and public safety.To achieve the project’s objectives, the researchers will address the following research question: (1) How to track human performance under virtual scenarios? A VR station will be set up which is capable of tracking multimodal body data such as gaze, body pose, hand movement, and the locomotion of a trainee in performing tasks in a virtual workplace. Based on such tracked data, machine learning models are trained to analyze and characterize the trainee’s skill levels. (2) How to synthesize virtual environments for personalized training? Optimization approaches will be formulated for synthesizing realistic and highly immersive virtual environments for adaptively training the trainee considering his/her skill levels, preferences, and personal training goals. (3) How to evaluate VR training effects? The investigator will evaluate the effectiveness of the synthesized VR training experiences by comparing with alternative training approaches quantitatively in terms of performance gain and knowledge retention; and qualitatively by obtaining domain experts’ and participants’ feedback about the effectiveness, enjoyment, and engagement of the training experience. To facilitate easy deployment and widespread adoption of personalized VR training, the investigator will publicly disseminate the software tools and toolkits devised through project websites, workshops, and research papers. In addition, the investigator will disseminate the research by (1) setting up an open-access VR station at the George Mason University’s Makerspace where faculty, students, and staff from various disciplines can conveniently experience personalized VR training; (2) organizing interdisciplinary VR training workshops and demo days to disseminate the VR training research findings, to showcase the VR training demos to the public, and to stimulate the research and adoption of VR training in different disciplines; (3) broadening the participation of first-generation underrepresented undergraduate students in computing and VR training research via a series of focused mentoring activities organized in collaboration with the Louis Stokes Alliance for Minority Participation program at the university.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

虚拟现实（VR）有望为培训和再培训员工提供一种引人注目、有效和方便的手段。这个项目的目标是设计一个计算设计框架，用于指导人类表现的个性化虚拟训练环境的合成。设计师和普通用户可以直观地应用该框架，以全自动、快速、可扩展和低成本的方式合成各种沉浸式和引人入胜的VR培训场景。作为拟议框架的展示，该项目将展示如何为劳动力培训综合个性化的虚拟培训环境，例如员工，工作场所主管和安全检查员的安全检查培训。该研究项目提供了一个新的跨学科视角，通过汇集来自人机交互，虚拟现实，计算机图形学，机器学习和优化的专业知识和见解，以及教学设计，安全，应急管理和工程领域的专业知识，生成个性化的VR培训内容。设计师和普通用户可以使用该框架来合成各种场景的个性化VR培训内容，如康复，安全培训，灾难应对培训，以及制造，建筑，物流，运输，零售管理和公共安全等不同领域的劳动力培训。为了实现项目目标，研究人员将解决以下研究问题：（1）如何在虚拟场景下跟踪人的表现？将建立一个VR站，它能够跟踪多模态身体数据，如凝视，身体姿势，手部运动和受训者在虚拟工作场所执行任务时的运动。基于这样的跟踪数据，机器学习模型被训练以分析和表征受训者的技能水平。(2)如何合成虚拟环境进行个性化训练？将制定优化方法来合成逼真且高度身临其境的虚拟环境，以便根据学员的技能水平、偏好和个人培训目标对他/她进行自适应培训。(3)如何评估VR培训效果？研究者将通过在绩效增益和知识保留方面与替代培训方法进行定量比较，并通过获取领域专家和参与者对培训体验的有效性、享受和参与度的反馈进行定性，来评估合成VR培训体验的有效性。为了方便部署和广泛采用个性化VR培训，研究人员将通过项目网站，研讨会和研究论文公开传播设计的软件工具和工具包。此外，研究者还将通过以下方式传播研究成果：（1）在乔治梅森大学的Makerspace建立一个开放式VR站，来自各个学科的教师、学生和工作人员可以方便地体验个性化的VR培训;（2）举办跨学科的虚拟现实培训工作坊及示范日，以宣传虚拟现实培训的研究成果，向公众展示虚拟现实培训示范，并鼓励研究和采用不同学科的虚拟现实培训;（3）扩大第一批参与者的参与--通过与大学的Louis Stokes少数民族参与联盟项目合作组织的一系列有针对性的指导活动，这一代在计算和VR培训研究方面代表性不足的本科生。该奖项反映了NSF的法定使命，并被认为是通过使用基金会的知识价值和更广泛的影响审查标准进行评估，

项目成果

A Review on Virtual Reality Skill Training Applications

• DOI: 10.3389/frvir.2021.645153
• 发表时间: 2021-04
• 作者: Biao Xie;Huimin Liu;Rawan Alghofaili;Yongqi Zhang;Yeling Jiang;F. Lobo;Changyang Li;Wanwan Li-Wan

Synthesizing scene-aware virtual reality teleport graphs

• DOI: 10.1145/3478513.3480478
• 发表时间: 2021-12
• 期刊: ACM Transactions on Graphics (TOG)
• 作者: Changyang Li;Haikun Huang;Jyh-Ming Lien;L. Yu

Designing Human-Robot Coexistence Space

• DOI: 10.1109/lra.2021.3097061
• 发表时间: 2020-11
• 期刊: IEEE Robotics and Automation Letters
• 影响因子: 5.2
• 作者: Jixuan Zhi;L. Yu;Jyh-Ming Lien

WFH-VR: Teleoperating a Robot Arm to set a Dining Table across the Globe via Virtual Reality

WFH-VR：通过虚拟现实远程操作机器人手臂在全球范围内设置餐桌

• DOI: 10.1109/iros47612.2022.9981729
• 发表时间: 2022
• 期刊: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS
• 作者: Yim, Lai Sum;Vo, Quang TN;Huang, Ching-I;Wang, Chi-Ruei;McQueary, Wren;Wang, Hsueh-Cheng;Huang, Haikun;Yu, Lap-Fai
• 通讯作者: Yu, Lap-Fai

Interactive Design of Gallery Walls via Mixed Reality

• DOI: 10.1109/aivr50618.2020.00013
• 发表时间: 2020-12
• 期刊: 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)
• 作者: Haikun Huang;Yuxuan Zhang;Tomer Weiss;R. Perry;L. Yu