Multimodal Visitor Analytics: Investigating Naturalistic Engagement with Interactive Tabletop Science Exhibits

参与是非正式科学教育中学习的基石。在博物馆和科学中心的自由选择学习中，游客的参与决定了学习者如何与展品互动，如何在展览空间中穿行，以及如何形成对科学的态度、兴趣和理解。多模式学习分析的最新进展为在自由选择的环境中扩大访问者参与度的范围和丰富程度创造了新的机会。特别是，多模式学习分析为整合多个数据源提供了巨大的潜力，可以设计出访问者认知、情感和行为参与的综合图像。该项目将集中于为个人参观者和小团体之间的交互式桌面科学展览提供有意义的游客参与的丰富经验，并揭示在整个展览空间中展开的更广泛的游客参与趋势模式。该项目的一个关键目标是创建模型和以从业者为中心的学习分析工具，为展览设计师和博物馆教育工作者提供最佳实践。该项目由推进非正式STEM学习（AISL）计划资助。作为在非正式环境中加强学习的总体战略的一部分，AISL资助研究和创新的方法和资源，用于各种环境。研究团队将通过多模态学习分析对参观者的学习体验进行数据丰富的调查，多模态学习分析将全仪器化展览空间产生的丰富的多渠道数据流与机器学习和教育数据挖掘的最新进展所提供的数据驱动建模功能融合在一起。研究小组将在参观者探索互动桌面科学展览时，对自然参与进行一系列的游客研究，包括单人、双人和团体互动。这些研究将利用眼动仪捕捉参观者的即时注意力，面部表情分析和定量现场观察来跟踪参观者的情绪状态，展览软件生成的跟踪日志，以及动作跟踪传感器和编码视频记录来捕捉参观者的行为互动。研究还将使用对话录音和前后评估措施来捕捉游客的科学理解和探究过程。将这些多模态数据流作为训练数据，研究团队将使用概率和神经机器学习技术来设计访问者参与的学习分析模型。该项目将由北卡罗莱纳州立大学和北卡罗莱纳州自然科学博物馆合作进行。研究团队将1)设计一个数据丰富的多模式游客研究方法，2)创建游客信息平台，一套用于多模式游客分析的开源软件工具，以及3)推出多模式游客数据仓库，一个精心策划的游客体验数据档案。多模式访客研究方法、访客信息平台和多模式访客数据仓库将使非正式科学教育界的研究人员和实践者能够在他们自己的非正式学习环境中利用多模式学习分析。预计该项目将通过扩展和丰富有意义的游客参与措施，扩大游客体验设计原则的证据基础，并提供学习分析工具来支持博物馆教育工作者，从而推动非正式STEM学习领域的发展。通过加强对科学博物馆访客体验背后的认知、情感和行为动态的理解，非正式的科学教育者将能够更好地设计更有效、更吸引人的学习体验。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Engagement is the cornerstone of learning in informal science education. During free-choice learning in museums and science centers, visitor engagement shapes how learners interact with exhibits, navigate through exhibit spaces, and form attitudes, interests, and understanding of science. Recent advances in multimodal learning analytics are creating novel opportunities for expanding the range and richness of measures of visitor engagement in free-choice settings. In particular, multimodal learning analytics offer significant potential for integrating multiple data sources to devise a composite picture of visitors' cognitive, affective, and behavioral engagement. The project will center on providing a rich empirical account of meaningful visitor engagement with interactive tabletop science exhibits among individual visitors and small groups, as well as uncovering broader tidal patterns in visitor engagement that unfold across exhibit spaces. A key objective of the project is creating models and practitioner-focused learning analytic tools that will inform the best practices of exhibit designers and museum educators. This project is funded by the Advancing Informal STEM Learning (AISL) program. As part of its overall strategy to enhance learning in informal environments, AISL funds research and innovative approaches and resources for use in a variety of settings.The research team will conduct data-rich investigations of visitors' learning experiences with multimodal learning analytics that fuse the rich multichannel data streams produced by fully-instrumented exhibit spaces with the data-driven modeling functionalities afforded by recent advances in machine learning and educational data mining. The research team will conduct a series of visitor studies of naturalistic engagement in solo, dyad, and group interactions as visitors explore interactive tabletop science exhibits. The studies will utilize eye trackers to capture visitors' moment-to-moment attention, facial expression analysis and quantitative field observations to track visitors' emotional states, trace logs generated by exhibit software, as well as motion-tracking sensors and coded video recordings to capture visitors' behavioral interactions. The studies will also use conversation recordings and pre-post assessment measures to capture visitors' science understanding and inquiry processes. With these multimodal data streams as training data, the research team will use probabilistic and neural machine learning techniques to devise learning analytic models of visitor engagement. The project will be conducted by a partnership between North Carolina State University and the North Carolina Museum of Natural Sciences. The research team will 1) design a data-rich multimodal visitor study methodology, 2) create the Visitor Informatics Platform, a suite of open source software tools for multimodal visitor analytics, and 3) launch the Multimodal Visitor Data Warehouse, a curated visitor experience data archive. Together, the multimodal visitor study methodology, the Visitor Informatics Platform, and the Multimodal Visitor Data Warehouse will enable researchers and practitioners in the informal science education community to utilize multimodal learning analytics in their own informal learning environments. It is anticipated that the project will advance the field of informal STEM learning by extending and enriching measures of meaningful visitor engagement, expanding the evidence base for visitor experience design principles, and providing learning analytic tools to support museum educators. By enhancing understanding of the cognitive, affective, and behavioral dynamics underlying visitor experiences in science museums, informal science educators will be well-positioned to design learning experiences that are more effective and engaging.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.