CAREER: Non-Text-Based Smart Learning in Multi-Person Virtual Reality

非基于文本的智能学习是指使用(1)非文本特征(即可视化信息)和(2)基于个人需求的改编学习材料的技术支持的学习。完全沉浸式多人虚拟现实(MVR)是指位于不同地点的人们戴着VR设备加入单个虚拟房间，即不限大小的教室，向教师学习。学习环境正准备经历一场重大变革，虚拟现实(VR)将增强甚至可能取代传统的课堂学习环境。本研究的目的是通过对生理测量的非侵入性多模式分析，在MVR环境中发现新的智能学习方法，包括眼动特征、触觉交互和大脑活动。通过进行这项研究，该项目探索了在非基于文本的MVR学习环境中人类行为范围(例如，他们观察到什么以及他们如何互动)的新问题。非语言生理学测量可以用来评估个体的学习投入，并及时支持个体的学习需求。此外，该项目还包括帮助儿童，特别是代表人数不足的少数群体和农村欠发达地区的儿童，这将使他们能够调查儿童在学习或不学习时的身体变化，并提供机会探索儿童在特定环境和特定教学方法中能学到什么。该项目将MVR中的智能学习教学法与被评估的人类行为联系起来，并将其转化为一个连贯的多通道分析模型。MVR中的多模式分析由两个主要推力组成。第一个重点是通过发现多个生理指标之间的关系来评估多人学习绩效，并利用这些关系来预测学习绩效。能够有效地解决数据复杂性和可变性的生理测量包括但不限于，熵(即，视觉询问或触觉交互模式中的无序或随机程度)、视觉分组(即，比其他对象或信息块更频繁地被密切询问或交互的对象或信息块)，以及脑血流动力学。第二个重点是开发MVR学习材料，发现非基于文本的智能学习脚手架策略，以提高学习成绩。预测性措施与行为特征的结合将导致开发更个性化的培训，并创建更集成的智能学习系统，以满足表现不佳的学习者的多样化需求。该项目的研究成果将通过俄克拉荷马州虚拟学术图书馆(OVAL)用于教师的专业发展和为学生提供最先进的学习机会，OVAL是一个为教学和教学教师设计的虚拟会议场所。该项目由既定的激励竞争研究计划(EPSCoR)共同资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Non-text-based smart learning refers to technology-supported learning that uses (1) non-text features (i.e. visualized information) and (2) adapted learning materials based on the individual’s needs. Fully immersive multi-person virtual reality (MVR) refers to humans located at different places wearing VR devices to join a single virtual room, a classroom with unlimited size, to learn from an instructor. The learning environment is poised to undergo a major reformation, and Virtual Reality (VR) will augment, and possibly replace, the traditional classroom learning environment. The purpose of this research is to discover new smart learning methodologies within the MVR environment using nonintrusive multimodal analysis of physiological measures, including eye movement characteristics, haptic interactions, and brain activities. By doing this research, the project explores new questions on the range of human behaviors (e.g. what they observe and how they interact) within the non-text-based MVR learning environment. Non-verbal physiological measures can be used to assess the individuals’ engagement in learning and support the individuals’ learning needs in a timely manner. Furthermore, the project includes assisting the children, especially the underrepresented minorities and those in rural, under-developed areas, that will allow the investigation of the physical shifts/changes when a child is learning or not learning, and provide opportunities to explore what the child can learn in specific environments and with specific teaching methods. The project bridges smart learning pedagogy in MVR with the assessed humans’ behaviors and transforms them into a cohesive multimodal analysis model. Multimodal analyses in MVR are composed of two main thrusts. The first thrust is to evaluate the multi-person learning performance through discovering the relationships among the multiple physiological measures and to use the relationships to predict learning performance. The physiological measures that can effectively address the data complexity and variability include, but are not limited to, entropy (i.e. degree of disorder or randomness in the visual interrogation or haptic interaction patterns), visual groupings (i.e. objects or information blocks that were closely interrogated or interacted back and forth more often than others), and cerebral hemodynamics. The second thrust is to develop MVR learning materials and discover non-text-based smart learning scaffolding strategies to improve learning performance. The utilization of the predictive measures paired with behavioral traits will lead to the development of a more personalized training and creation of a more integrated smart learning system to meet the diverse needs of underperforming learners. The research results from the project will be used for professional development of teachers and state-of-the-art learning opportunities to the students, through the Oklahoma Virtual Academic Library (OVAL), a virtual meeting place designed for teaching and teaching teachers.This project is co-funded by the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.