Research on the Use of Real-Time Tracking and Eye-Tracking Technology for Integrating Metacognition and Augmented Reality into Undergraduate Engineering Laboratories

该项目旨在创新一个本科工程实验室的增强现实(AR)学习平台。最近，由于新冠肺炎疫情，面对面的工程实验室在提供动手练习方面面临着重大挑战。AR技术可能是应对当前挑战的一种解决方案。与虚拟现实不同，AR不覆盖物理世界，而是将3D虚拟对象混合到物理对象中，以改善学生的实验室体验和空间意识。然而，学生们在AR设备上导航时遇到了困难，在AR环境中学习时，他们经历了计算机生成的3D图像与实物之间的不匹配。为了将AR学习从静态和孤立的体验转变为动态的自主学习，本项目包括研究使用带有3D全身运动捕捉系统的实时跟踪传感器来提高AR的可用性，并利用元认知监控反馈和眼球跟踪技术检测学生提前终止学习的情况。该项目的主要目标是集成实时3D运动和位置跟踪系统，以满足一系列目标，以洞察基于位置的AR环境中的科学发展和技术创新。第一个目标是通过整合最先进的室内实时定位技术和3D全身运动捕捉系统来创建下一代AR学习平台。第二个目标是通过使用元认知监控反馈和眼动数据来检测学习的提前终止。研究人员将应用先进的凝视行为指标，如瞳孔放大和眼睛内盯着持续时间的分形分析，来识别代表学生提前终止学习的显著眼睛凝视行为。最后一个目标是实施研究结果，以促进工程实验室在接收元认知反馈对学习成绩的影响方面的教学改进。目前的项目将致力于综合研究和教育活动，以提高我们对有效的本科工程实验室教育的认识，使用实时3D运动和位置跟踪能力。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to innovate Augmented Reality (AR) learning platforms in an undergraduate engineering laboratory. Recently, due to the COVID-19 pandemic, in-person engineering laboratories faced significant challenges in providing hands-on exercise. AR technology might be a solution to the current challenge. Unlike virtual reality, AR does not cover a physical world but mixes 3D virtual objects into physical objects to improve students’ laboratory experiences and spatial awareness. However, students faced difficulties navigating the AR device and have experienced a mismatch between computer-generated 3D images and physical objects while learning in an AR environment. To transform AR learning from a static and isolated experience into a dynamic self-study, this project includes research on using a real-time tracking sensor with a 3D full-body motion capture system to improve AR usability and detect student’s premature termination of learning using metacognitive monitoring feedback and eye-tracking technology. The primary goal of this project is to integrate real-time 3D motion and location tracking systems to meet a series of objectives to gain insight into scientific development and technological innovation in a location-based AR environment. The first objective is to create the next generation of an AR learning platform by integrating state-of-the-art indoor real-time location technology and a 3D full-body motion capture system. The second objective is to detect the early termination of learning by using metacognitive monitoring feedback and eye movement data. The researchers will apply advanced gaze behavior metrics, such as fractal analysis of pupil dilation and eye inter-fixation duration, to identify significant eye gaze behavior representing student’s premature termination of learning. The last objective is to implement the research findings to promote instructional improvement in an engineering laboratory on the impact of receiving metacognitive feedback on learning performance. The current project will address integrated research and education activities to enhance our knowledge of effective undergraduate engineering laboratory education using real-time 3D motion and location tracking capabilities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.