CAREER: Performance-Guided Synthesis of Virtual Environments for Personalized Training

虚拟现实（VR）有望为培训和再培训员工提供一种引人注目、有效和方便的手段。这个项目的目标是设计一个计算设计框架，用于指导人类表现的个性化虚拟训练环境的合成。设计师和普通用户可以直观地应用该框架，以全自动、快速、可扩展和低成本的方式合成各种沉浸式和引人入胜的VR培训场景。作为拟议框架的展示，该项目将展示如何为劳动力培训综合个性化的虚拟培训环境，例如员工，工作场所主管和安全检查员的安全检查培训。该研究项目提供了一个新的跨学科视角，通过汇集来自人机交互，虚拟现实，计算机图形学，机器学习和优化的专业知识和见解，以及教学设计，安全，应急管理和工程领域的专业知识，生成个性化的VR培训内容。设计师和普通用户可以使用该框架来合成各种场景的个性化VR培训内容，如康复，安全培训，灾难应对培训，以及制造，建筑，物流，运输，零售管理和公共安全等不同领域的劳动力培训。为了实现项目目标，研究人员将解决以下研究问题：（1）如何在虚拟场景下跟踪人的表现？将建立一个VR站，它能够跟踪多模态身体数据，如凝视，身体姿势，手部运动和受训者在虚拟工作场所执行任务时的运动。基于这样的跟踪数据，机器学习模型被训练以分析和表征受训者的技能水平。(2)如何合成虚拟环境进行个性化训练？优化方法将制定合成现实和高度沉浸式的虚拟环境，自适应培训学员考虑他/她的技能水平，偏好和个人培训目标。(3)如何评估VR培训效果？研究者将通过在绩效增益和知识保留方面与替代培训方法进行定量比较，并通过获取领域专家和参与者对培训体验的有效性、享受和参与度的反馈进行定性，来评估合成VR培训体验的有效性。为了方便部署和广泛采用个性化VR培训，研究人员将通过项目网站，研讨会和研究论文公开传播设计的软件工具和工具包。此外，研究者还将通过以下方式传播研究成果：（1）在乔治梅森大学的Makerspace建立一个开放式VR站，来自各个学科的教师、学生和工作人员可以方便地体验个性化的VR培训;（2）举办跨学科的虚拟现实培训工作坊及示范日，以宣传虚拟现实培训的研究成果，向公众展示虚拟现实培训示范，并鼓励研究和采用不同学科的虚拟现实培训;（3）扩大第一批参与者的参与--通过与大学的Louis Stokes少数民族参与联盟项目合作组织的一系列有针对性的指导活动，这一代在计算和VR培训研究方面代表性不足的本科生。该奖项反映了NSF的法定使命，并被认为是通过使用基金会的知识价值和更广泛的影响审查标准进行评估，

Virtual Reality (VR) promises to provide a compelling, effective, and convenient means for training and reskilling the workforce. The objective of this project is to devise a computational design framework for guiding the synthesis of personalized virtual training environments for human performance. Designers and general users can intuitively apply this framework for synthesizing a variety of immersive and engaging VR training scenarios in a fully automatic, fast, scalable, and low-cost manner. As a showcase of the proposed framework, the project will demonstrate how to synthesize personalized virtual training environments for workforce training, such as safety inspection training for employees, workplace supervisors, and safety inspectors. This research project offers a novel interdisciplinary perspective for generating personalized VR training content by bringing together expertise and insights from human-computer interaction, virtual reality, computer graphics, machine learning, and optimization, as well as domain expertise in instructional design, safety, emergency management, and engineering. Designers and general users can use this framework to synthesize personalized VR training content for a variety of scenarios such as rehabilitation, safety training, disaster response training, as well as workforce training for different domains such as manufacturing, construction, logistics, transportation, retail management, and public safety.To achieve the project’s objectives, the researchers will address the following research question: (1) How to track human performance under virtual scenarios? A VR station will be set up which is capable of tracking multimodal body data such as gaze, body pose, hand movement, and the locomotion of a trainee in performing tasks in a virtual workplace. Based on such tracked data, machine learning models are trained to analyze and characterize the trainee’s skill levels. (2) How to synthesize virtual environments for personalized training? Optimization approaches will be formulated for synthesizing realistic and highly immersive virtual environments for adaptively training the trainee considering his/her skill levels, preferences, and personal training goals. (3) How to evaluate VR training effects? The investigator will evaluate the effectiveness of the synthesized VR training experiences by comparing with alternative training approaches quantitatively in terms of performance gain and knowledge retention; and qualitatively by obtaining domain experts’ and participants’ feedback about the effectiveness, enjoyment, and engagement of the training experience. To facilitate easy deployment and widespread adoption of personalized VR training, the investigator will publicly disseminate the software tools and toolkits devised through project websites, workshops, and research papers. In addition, the investigator will disseminate the research by (1) setting up an open-access VR station at the George Mason University’s Makerspace where faculty, students, and staff from various disciplines can conveniently experience personalized VR training; (2) organizing interdisciplinary VR training workshops and demo days to disseminate the VR training research findings, to showcase the VR training demos to the public, and to stimulate the research and adoption of VR training in different disciplines; (3) broadening the participation of first-generation underrepresented undergraduate students in computing and VR training research via a series of focused mentoring activities organized in collaboration with the Louis Stokes Alliance for Minority Participation program at the university.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.