A Remote Multimodal Learning Environment to Increase Graphical Information Access for Blind and Visually Impaired Students

对于盲人或视障学生（BVI）来说，在教室或工作场所获取复杂的STEM图形信息有很多限制。这个长期存在的问题是由于依赖于难以获取和过时的学习材料，需要昂贵的专业设备，以及坚持过时的教育服务模式而产生的。为了解决这些问题，该项目将研究开发和评估一种创新的远程学习系统，该系统基于使用多种感官渠道，从听觉、语言、触觉和增强的视觉感知中战略性地呈现信息。本研究将重点关注多模态信息呈现和感知的优化，根据其独特的信息处理特征分离感官输出，以传递不同类型的刺激。第一个项目目标是通过确定促进STEM图形和空间信息非视觉知识获取的感知动机学习支持（学习目标）来提高英属维尔京群岛学生的STEM教学质量。第二个项目目标是通过创建一个创新的远程多模式界面来交流视觉信息的概念意义，从而增加对图形和空间STEM内容的访问（技术目标）。项目成果将有助于非视觉学习和多感官处理的理论，并为盲人和视力正常的用户开发更高效、直观和可用的多模态界面提供明确的转化途径。通过提供一个新的、低成本的、可访问的技术平台来交流非视觉化的STEM材料，研究结果的应用将有助于解决英属维尔京群岛个人在STEM相关学科中代表性严重不足的问题，以及这一人口中70%的失业率。研究人员将回答以下相互关联的问题：1)为了最大限度地提高STEM图形材料的感知显著性、可学习性、解释和表示，不同模态输出所传达的最佳信息内容是什么？在实验室中优化后，2)优化后的多模态学习系统在远程部署环境中向BVI学习者传递图形化STEM材料的表现如何；3)与传统的BVI教学方法相比，远程学习系统是否提高了对STEM图形内容的理解水平？将收集和分析有关优化过程和远程技术系统的定量和定性数据，包括用户对速度/准确性的响应指标、用户体验数据以及适用于英属维尔京群岛学生的STEM图形评估工具。研究的第一阶段将研究多模态信息处理，以建立信息传递和非视觉图形学习效率的最佳实践，通过实验比较三个核心STEM图形主题：图形、图表和地图以不同方式呈现的图形信息。实验的第二阶段将调查远程学习系统的有效性，并评估用户在图形化STEM学习测量和关键可用性和满意度指标上的表现。这个项目有一个更广泛的目标，即提高英属维尔京群岛学习者和其他有残疾或没有残疾的学生的独立性，这些学生可能会从远程多模式学习环境中受益，并开发一种新的工具来支持英属维尔京群岛人类受试者的大规模研究和辅助技术评估，从而大大提高科学家招募和与更多英属维尔京群岛用户合作的能力，而不是目前基于实验室的研究。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

There are many limitations for students who are blind or visually impaired (BVI) in accessing complex STEM graphical information in the classroom or workplace. This longstanding problem arises due to reliance on inaccessible and outdated learning materials, the need for costly specialized devices, and an adherence to an outdated educational service model. To address these issues, this project will investigate the development and evaluation of an innovative remote learning system based on the use of multiple sensory channels to strategically present information from auditory, linguistic, touch, and enhanced visual sensing. The research will focus specifically on the optimization of multimodal information presentation and perception, separating sensory output based on its unique information processing characteristics for conveying different types of stimuli. The first project goal is to increase the quality of STEM instruction for BVI students by determining perceptually motivated learning supports that promote non-visual knowledge acquisition of STEM graphical and spatial information (learning goal). The second project goal is to increase access to graphical and spatial STEM content through creation of an innovative remote multimodal interface for communicating the conceptual meaning of visual information (technology goal). The project outcomes will contribute to theories of non-visual learning and multisensory processing, and a clear translational path to development of more efficient, intuitive, and usable multimodal interfaces for both blind and sighted users. The application of the results will help to address the severe under-representation of BVI individuals in STEM-related disciplines, and the 70% unemployment rate of this demographic, by providing a new, low-cost, and accessible technology platform for communicating non-visual graphical STEM materials. The researchers will answer the following inter-connected questions: 1) What is the best information content to be conveyed by different modal outputs for maximizing perceptual saliency, learnability, interpretation, and representation of STEM graphical materials? Once optimized in the lab, 2) How well does the optimized multimodal learning system perform in a remote deployment environment in conveying graphical STEM materials to BVI learners; and 3) Does the remote learning system increase the level of comprehension of STEM graphical content as compared to traditional BVI instructional methods? Both quantitative and qualitative data about the optimization process and the remote technology system will be collected and analyzed including user response metrics on speed/accuracy, user experience data, and STEM graphical assessment instruments adapted for BVI students. The first phase of the research will investigate multimodal information processing in order to establish best practices for information delivery and non-visual graphical learning efficiency with experiments comparing graphical information presented in different modalities for three core STEM graphical themes: graphs, diagrams, and maps. The second phase of experiments will investigate the remote learning system's efficacy as well as evaluating user performance on graphical STEM learning measures and key usability and satisfaction metrics. This project has the broader goals of increasing independence for BVI learners and other students with or without disabilities who might benefit from a remote multimodal learning environment, and the development of a new tool for supporting large-scale research and assistive technology evaluation with BVI human subjects, thereby dramatically increasing scientists' ability to recruit and work with a much larger population of BVI users than is currently possible from lab-based studies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.