Multimodal Visitor Analytics: Investigating Naturalistic Engagement with Interactive Tabletop Science Exhibits

多模式访客分析：研究交互式桌面科学展览的自然参与

• 批准号: 1713545
• 负责人: James Lester
• 金额: $ 195.2万
• 依托单位: North Carolina State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-03-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1713545&HistoricalAwards=false
• 关键词: Multimodal; Visitor; Analytics; Investigating; Naturalistic

Engagement is the cornerstone of learning in informal science education. During free-choice learning in museums and science centers, visitor engagement shapes how learners interact with exhibits, navigate through exhibit spaces, and form attitudes, interests, and understanding of science. Recent advances in multimodal learning analytics are creating novel opportunities for expanding the range and richness of measures of visitor engagement in free-choice settings. In particular, multimodal learning analytics offer significant potential for integrating multiple data sources to devise a composite picture of visitors' cognitive, affective, and behavioral engagement. The project will center on providing a rich empirical account of meaningful visitor engagement with interactive tabletop science exhibits among individual visitors and small groups, as well as uncovering broader tidal patterns in visitor engagement that unfold across exhibit spaces. A key objective of the project is creating models and practitioner-focused learning analytic tools that will inform the best practices of exhibit designers and museum educators. This project is funded by the Advancing Informal STEM Learning (AISL) program. As part of its overall strategy to enhance learning in informal environments, AISL funds research and innovative approaches and resources for use in a variety of settings.The research team will conduct data-rich investigations of visitors' learning experiences with multimodal learning analytics that fuse the rich multichannel data streams produced by fully-instrumented exhibit spaces with the data-driven modeling functionalities afforded by recent advances in machine learning and educational data mining. The research team will conduct a series of visitor studies of naturalistic engagement in solo, dyad, and group interactions as visitors explore interactive tabletop science exhibits. The studies will utilize eye trackers to capture visitors' moment-to-moment attention, facial expression analysis and quantitative field observations to track visitors' emotional states, trace logs generated by exhibit software, as well as motion-tracking sensors and coded video recordings to capture visitors' behavioral interactions. The studies will also use conversation recordings and pre-post assessment measures to capture visitors' science understanding and inquiry processes. With these multimodal data streams as training data, the research team will use probabilistic and neural machine learning techniques to devise learning analytic models of visitor engagement. The project will be conducted by a partnership between North Carolina State University and the North Carolina Museum of Natural Sciences. The research team will 1) design a data-rich multimodal visitor study methodology, 2) create the Visitor Informatics Platform, a suite of open source software tools for multimodal visitor analytics, and 3) launch the Multimodal Visitor Data Warehouse, a curated visitor experience data archive. Together, the multimodal visitor study methodology, the Visitor Informatics Platform, and the Multimodal Visitor Data Warehouse will enable researchers and practitioners in the informal science education community to utilize multimodal learning analytics in their own informal learning environments. It is anticipated that the project will advance the field of informal STEM learning by extending and enriching measures of meaningful visitor engagement, expanding the evidence base for visitor experience design principles, and providing learning analytic tools to support museum educators. By enhancing understanding of the cognitive, affective, and behavioral dynamics underlying visitor experiences in science museums, informal science educators will be well-positioned to design learning experiences that are more effective and engaging.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

参与是非正式科学教育学习的基石。在博物馆和科学中心的自由选择学习期间，游客参与塑造了学习者如何与展品互动，浏览展览空间，并形成对科学的态度，兴趣和理解。多模态学习分析的最新进展正在为扩大自由选择环境中访问者参与度的范围和丰富性创造新的机会。特别是，多模态学习分析提供了整合多个数据源的巨大潜力，以设计一个游客的认知，情感和行为参与的综合图片。该项目将集中在提供一个丰富的经验帐户的有意义的游客参与与互动桌面科学展览之间的个人游客和小团体，以及揭示更广泛的潮汐模式，在游客的参与，在展览空间展开。该项目的一个关键目标是创建模型和以博物馆工作者为中心的学习分析工具，为展览设计师和博物馆教育工作者提供最佳实践。该项目由推进非正式STEM学习（AISL）计划资助。 作为其加强非正规环境中学习的总体战略的一部分，AISL资助研究和创新方法和资源，用于各种环境。研究团队将通过多模态学习分析对参观者的学习体验进行数据丰富的调查，该分析将由全仪器化的展览空间产生的丰富的多通道数据流与数据融合，机器学习和教育数据挖掘的最新进展提供的驱动建模功能。研究小组将进行一系列的游客研究的自然参与独奏，二分体，和群体的互动，游客探索互动桌面科学展览。这些研究将利用眼动仪捕捉参观者的每时每刻的注意力，面部表情分析和定量实地观察来跟踪参观者的情绪状态，利用展览软件生成的跟踪日志，以及运动跟踪传感器和编码视频记录来捕捉参观者的行为互动。研究亦会采用对话录音及事前事后评估的方法，以记录参观者对科学的理解及探究过程。有了这些多模态数据流作为训练数据，研究团队将使用概率和神经机器学习技术来设计游客参与度的学习分析模型。该项目将由北卡罗来纳州州立大学和北卡罗来纳州自然科学博物馆合作进行。该研究团队将1）设计一个数据丰富的多模态游客研究方法，2）创建游客信息平台，一套用于多模态游客分析的开源软件工具，以及3）推出多模态游客数据仓库，一个策划的游客体验数据档案。多模态访客研究方法，访客信息平台和多模态访客数据仓库将使非正式科学教育社区的研究人员和从业者能够在自己的非正式学习环境中利用多模态学习分析。预计该项目将通过扩展和丰富有意义的游客参与措施，扩大游客体验设计原则的证据基础，并提供学习分析工具来支持博物馆教育工作者，从而推动非正式STEM学习领域的发展。通过加强对认知，情感和行为动力学的理解，在科学博物馆的游客体验，非正式的科学教育工作者将处于有利地位，设计更有效和引人入胜的学习体验。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

What's Fair is Fair: Detecting and Mitigating Encoded Bias in Multimodal Models of Museum Visitor Attention

• DOI: 10.1145/3462244.3479943
• 发表时间: 2021-10
• 期刊: Proceedings of the 2021 International Conference on Multimodal Interaction
• 作者: Halim Acosta;Nathan L. Henderson;Jonathan P. Rowe;Wookhee Min;James Minogue;James C. Lester

Multimodal Trajectory Analysis of Visitor Engagement with Interactive Science Museum Exhibits

游客与互动科学博物馆展品互动的多模态轨迹分析

• DOI: 10.1007/978-3-030-78270-2_27
• 发表时间: 2021
• 期刊: Artificial Intelligence in Education
• 作者: Emmerson, Andrew;Henderson, Nathan;Min, Wookhee;Rowe, Jonathan;Minogue, James;Lester, James
• 通讯作者: Lester, James

Early Prediction of Visitor Engagement in Science Museums with Multimodal Adversarial Domain Adaptation

通过多模态对抗域适应对科学博物馆的游客参与度进行早期预测

• 发表时间: 2021
• 期刊: Educational Data Mining
• 作者: Henderson, Nathan;Min, Wookhee;Emerson, Andrew;Rowe, Jonathan;Lee, Seung;Minogue, James;Lester, James
• 通讯作者: Lester, James

Investigating Visitor Engagement in Interactive Science Museum Exhibits with Multimodal Bayesian Hierarchical Models

使用多模态贝叶斯分层模型调查游客对互动科学博物馆展览的参与度

• DOI: 10.1007/978-3-030-52237-7-14
• 发表时间: 2020
• 期刊: Proceedings of the Twenty-First International Conference on Artificial Intelligence in Education
• 作者: Emerson, Andrew;Henderson, Nathan;Rowe, Jonathan;Min, Wookhee;Lee, Seung;Minogu, James;Lester, James.
• 通讯作者: Lester, James.