CAREER: Performance-Guided Synthesis of Virtual Environments for Personalized Training

虚拟现实(VR)有望为培训和再培训劳动力提供一种令人信服、有效和方便的手段。本项目的目标是设计一个计算设计框架，以指导为人类表演而合成的个性化虚拟训练环境。设计师和普通用户可以直观地应用该框架，以全自动、快速、可扩展和低成本的方式合成各种身临其境、引人入胜的VR培训场景。作为拟议框架的展示，该项目将演示如何合成用于劳动力培训的个性化虚拟培训环境，例如针对员工、工作场所主管和安全检查员的安全检查培训。该研究项目提供了一种新的跨学科视角，通过汇集人机交互、虚拟现实、计算机图形学、机器学习和优化的专业知识和见解，以及教学设计、安全、应急管理和工程领域的专业知识，来生成个性化的VR培训内容。设计人员和普通用户可以使用该框架综合各种场景的个性化VR培训内容，如康复、安全培训、灾难应对培训，以及针对制造、建筑、物流、运输、零售管理和公共安全等不同领域的劳动力培训。为了实现项目目标，研究人员将解决以下研究问题：(1)如何在虚拟场景下跟踪人类的表现？将建立一个VR站，能够跟踪多模式的身体数据，如凝视、身体姿势、手部运动和学员在虚拟工作场所执行任务时的移动。基于这些跟踪的数据，机器学习模型被训练来分析和表征受训者的技能水平。(2)如何合成虚拟环境进行个性化培训？将制定优化方法，以合成逼真和高度身临其境的虚拟环境，以便根据受训者的技能水平、偏好和个人培训目标进行适应性培训。(3)如何评价VR培训效果？研究人员将通过在绩效收益和知识保留方面与其他培训方法进行定量比较，以及通过获取领域专家和参与者对培训体验的有效性、享受性和参与度的反馈，来评估综合VR培训体验的有效性。为了便于部署和广泛采用个性化虚拟现实培训，调查员将通过项目网站、研讨会和研究论文公开传播设计的软件工具和工具包。此外，调查员还将通过以下方式传播这项研究：(1)在乔治梅森大学的Makerspace设立一个开放的VR站点，让来自不同学科的教职员工可以方便地体验个性化的VR培训；(2)组织跨学科的VR培训工作坊和演示日，以传播VR培训的研究成果，向公众展示VR培训演示，并促进不同学科对VR培训的研究和采用；(3)通过与大学路易斯·斯托克斯少数群体参与计划合作组织的一系列重点指导活动，扩大第一代代表性不足的本科生对计算和虚拟现实培训研究的参与。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual Reality (VR) promises to provide a compelling, effective, and convenient means for training and reskilling the workforce. The objective of this project is to devise a computational design framework for guiding the synthesis of personalized virtual training environments for human performance. Designers and general users can intuitively apply this framework for synthesizing a variety of immersive and engaging VR training scenarios in a fully automatic, fast, scalable, and low-cost manner. As a showcase of the proposed framework, the project will demonstrate how to synthesize personalized virtual training environments for workforce training, such as safety inspection training for employees, workplace supervisors, and safety inspectors. This research project offers a novel interdisciplinary perspective for generating personalized VR training content by bringing together expertise and insights from human-computer interaction, virtual reality, computer graphics, machine learning, and optimization, as well as domain expertise in instructional design, safety, emergency management, and engineering. Designers and general users can use this framework to synthesize personalized VR training content for a variety of scenarios such as rehabilitation, safety training, disaster response training, as well as workforce training for different domains such as manufacturing, construction, logistics, transportation, retail management, and public safety.To achieve the project’s objectives, the researchers will address the following research question: (1) How to track human performance under virtual scenarios? A VR station will be set up which is capable of tracking multimodal body data such as gaze, body pose, hand movement, and the locomotion of a trainee in performing tasks in a virtual workplace. Based on such tracked data, machine learning models are trained to analyze and characterize the trainee’s skill levels. (2) How to synthesize virtual environments for personalized training? Optimization approaches will be formulated for synthesizing realistic and highly immersive virtual environments for adaptively training the trainee considering his/her skill levels, preferences, and personal training goals. (3) How to evaluate VR training effects? The investigator will evaluate the effectiveness of the synthesized VR training experiences by comparing with alternative training approaches quantitatively in terms of performance gain and knowledge retention; and qualitatively by obtaining domain experts’ and participants’ feedback about the effectiveness, enjoyment, and engagement of the training experience. To facilitate easy deployment and widespread adoption of personalized VR training, the investigator will publicly disseminate the software tools and toolkits devised through project websites, workshops, and research papers. In addition, the investigator will disseminate the research by (1) setting up an open-access VR station at the George Mason University’s Makerspace where faculty, students, and staff from various disciplines can conveniently experience personalized VR training; (2) organizing interdisciplinary VR training workshops and demo days to disseminate the VR training research findings, to showcase the VR training demos to the public, and to stimulate the research and adoption of VR training in different disciplines; (3) broadening the participation of first-generation underrepresented undergraduate students in computing and VR training research via a series of focused mentoring activities organized in collaboration with the Louis Stokes Alliance for Minority Participation program at the university.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.