CAREER: Non-Text-Based Smart Learning in Multi-Person Virtual Reality

非文本智能学习是指技术支持的学习，它使用（1）非文本特征（即可视化信息）和（2）基于个人需求的适应性学习材料。完全沉浸式多人虚拟现实（MVR）是指位于不同地点的人佩戴VR设备加入一个虚拟房间，一个无限大小的教室，向教师学习。学习环境即将经历一场重大的变革，虚拟现实（VR）将增强并可能取代传统的课堂学习环境。本研究的目的是发现新的智能学习方法在MVR环境中使用非侵入性多模态分析的生理措施，包括眼动特征，触觉交互和大脑活动。通过这项研究，该项目探索了在非基于文本的MVR学习环境中人类行为范围的新问题（例如，他们观察到什么以及他们如何互动）。非语言生理测量可以用来评估个人的参与学习和支持个人的学习需要，以及时的方式。此外，该项目还包括协助儿童，特别是代表性不足的少数民族和农村欠发达地区的儿童，以便调查儿童在学习或不学习时的身体变化，并提供机会探索儿童在特定环境下和采用特定教学方法可以学到什么。该项目将MVR中的智能学习教学法与被评估的人类行为联系起来，并将其转换为一个有凝聚力的多模态分析模型。MVR中的多模态分析包括两个主要方面。第一个重点是通过发现多个生理指标之间的关系来评估多人学习绩效，并利用这些关系来预测学习绩效。可以有效地解决数据复杂性和可变性的生理测量包括但不限于熵（即，视觉询问或触觉交互模式中的无序或随机性的程度）、视觉分组（即，比其他更频繁地被紧密询问或来回交互的对象或信息块）和脑血流动力学。第二个重点是开发MVR学习材料，并发现非基于文本的智能学习支架策略，以提高学习成绩。结合行为特征的预测措施的使用将导致开发更个性化的培训和创建更集成的智能学习系统，以满足表现不佳的学习者的各种需求。该项目的研究成果将通过俄克拉荷马州虚拟学术图书馆（OVAL）用于教师的专业发展和学生的最先进的学习机会，一个为教学和教学教师设计的虚拟会议场所。该项目由刺激竞争力研究的既定计划（EPSCoR）共同资助该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Non-text-based smart learning refers to technology-supported learning that uses (1) non-text features (i.e. visualized information) and (2) adapted learning materials based on the individual’s needs. Fully immersive multi-person virtual reality (MVR) refers to humans located at different places wearing VR devices to join a single virtual room, a classroom with unlimited size, to learn from an instructor. The learning environment is poised to undergo a major reformation, and Virtual Reality (VR) will augment, and possibly replace, the traditional classroom learning environment. The purpose of this research is to discover new smart learning methodologies within the MVR environment using nonintrusive multimodal analysis of physiological measures, including eye movement characteristics, haptic interactions, and brain activities. By doing this research, the project explores new questions on the range of human behaviors (e.g. what they observe and how they interact) within the non-text-based MVR learning environment. Non-verbal physiological measures can be used to assess the individuals’ engagement in learning and support the individuals’ learning needs in a timely manner. Furthermore, the project includes assisting the children, especially the underrepresented minorities and those in rural, under-developed areas, that will allow the investigation of the physical shifts/changes when a child is learning or not learning, and provide opportunities to explore what the child can learn in specific environments and with specific teaching methods. The project bridges smart learning pedagogy in MVR with the assessed humans’ behaviors and transforms them into a cohesive multimodal analysis model. Multimodal analyses in MVR are composed of two main thrusts. The first thrust is to evaluate the multi-person learning performance through discovering the relationships among the multiple physiological measures and to use the relationships to predict learning performance. The physiological measures that can effectively address the data complexity and variability include, but are not limited to, entropy (i.e. degree of disorder or randomness in the visual interrogation or haptic interaction patterns), visual groupings (i.e. objects or information blocks that were closely interrogated or interacted back and forth more often than others), and cerebral hemodynamics. The second thrust is to develop MVR learning materials and discover non-text-based smart learning scaffolding strategies to improve learning performance. The utilization of the predictive measures paired with behavioral traits will lead to the development of a more personalized training and creation of a more integrated smart learning system to meet the diverse needs of underperforming learners. The research results from the project will be used for professional development of teachers and state-of-the-art learning opportunities to the students, through the Oklahoma Virtual Academic Library (OVAL), a virtual meeting place designed for teaching and teaching teachers.This project is co-funded by the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.