RAPD: Analytical and Exploratory Approaches to Communicate Mathematics to Visually Impaired Students

人类视觉系统的特点是大空间带宽和并发处理能力，使个人能够从复杂视觉布局中的单个组件、项目组和模式中学习。视觉信息处理的启用能力在许多领域都是至关重要的，理解具有数学内容的文档就是这样的领域之一。传统的数学表示本质上是空间的和多维的（例如，矩阵、分数），它经常结合分析表示（例如，公式、定义）和直观的图形描述（例如，图表）。这给视力障碍者造成了一个根本的障碍；由于缺乏视觉访问，他们无法访问数学内容的传统表示形式，即使是以数字格式编码的（例如，在线数学教科书）。视觉障碍学生使用的传统辅助技术（如盲文显示器、屏幕阅读器）本质上是线性的，删除了上下文参考，对读者的工作记忆施加了非常高的负荷。实际上，数学的可及性已经成为信息和学习机会的数字障碍。这在教育方面尤其严重，因为强调远程教育和联机材料并没有适当地与采用无障碍和普遍可用性原则相匹配。该项目的目标是开发一套知识和工具，使有视觉障碍的学生能够访问具有重要数学内容的在线教育材料。这项工作的目的是使视障学生能够以与正常学生相同的效率和认知过程访问和学习数学。特别是，我们的目标是创建一个全面的框架，在分析层面（例如公式、定理、证明）、探索/直觉层面（例如内容探索、访问图表和图表）和实践层面（例如解决问题、练习）提供访问数学内容的途径。这项工作的指导原则是理解和形式化数学文档中存在的潜在语义结构，从其基本组成部分（例如，单个公式，图的组成部分）开始，向更复杂的组织（例如，逻辑步骤，证明，定理，图表）移动。采用语义结构指导多模态表示方案，在表示（即连续访问内容）和导航（即用户导向的内容遍历）之间提供自动调整的平衡，基于用户模型的表示适应，以及听觉和触觉相结合的表示。触觉呈现依赖于一种新型硬件设备（触觉手指）的设计和开发，作为该项目的一部分。我们提出的框架还将使视障学生能够通过一个可访问的练习册，将阅读“数学与练习”穿插在一起，直接与阅读过程互动。将采用不同级别的评估，在设计的早期让视障学生参与，并依靠正式的认知模型来评估进展。该项目组建了一支经验丰富的研究人员团队，他们在应用认知/感知研究、辅助技术软件和硬件研究以及可用性设计和评估等领域具有专业知识。研究人员之前也合作过，例如，Tran和Pontelli进行了一项由美国国家科学基金会资助的关于视障学生上网无障碍的研究。智力优势：(1)观察和分析视觉障碍学生与电子形式的数学内容的人机交互；(2)开发提取数学内容语义表示的方法；(3)设计一种用于数学概念图形化表示的触觉装置和控制软件；(4)数学内容的听觉呈现与导航方法研究；(4)以用户模型为驱动，结合触觉和听觉的数学呈现和导航，并由可访问的数学概念实践工作空间组成；(5)视障学生可用性评价。更广泛的影响：(1)促进教学、培训和学习，重点关注视力障碍的多样性；(2)扩大视障学生访问和浏览含有大量数学内容的文件（如数学教科书）的能力，从而增加他们在所有领域，特别是在科学和工程领域的教育机会；(3)通过提高可用性专家在教育环境中评估触觉和听觉界面的能力，加强教育基础设施。最终，社会将受益于教育的更大包容性和视力残疾学生的培训机会的改善，并通常通过使这些用户更广泛地获得信息。所有pi都积极参与为不同学生群体提供教育机会的倡议-例如，新密歇根州立大学是一所少数民族和西班牙裔服务机构，来自传统上代表性不足的群体的学生将参与研究和教育活动。关键词：数学无障碍，触觉导航，视觉障碍，可用性评价方法

PI: Pontelli, EnricoProposal Number: 0754525The human visual system is characterized by a large spatial bandwidth and concurrent processing capabilities, enabling individuals to learn from individual components, groups of items, and patterns in complex visual layouts. The enabling capabilities of visual information processing are vital in many areas the understanding of documents with mathematical content is one of such areas. The traditional representation of mathematics is inherently spatial and multi-dimensional (e.g., matrices, fractions), and it often combines analytical presentation (e.g., formulae, definitions) with intuitive graphical descriptions (e.g., diagrams).This creates a fundamental obstacle for people with visual disabilities; their lack of visual access prevents them from accessing traditional representations of mathematical content, even when encoded in digital format (e.g., on-line mathematics textbooks). The traditional assistive technologies employed by students with visual disabilities (e.g., Braille displays, screen readers) are inherently linear, removing contextual references and imposing very high load on the reader's working memory. Effectively, accessibility of mathematics has become a digital barriers to information and learning opportunities. This is particularly severe in the educational context, where the emphasis on distance education and on-line material has not been properly matched by the adoption of accessibility and universal usability principles. The goal of this project is to develop a body of knowledge and tools that will enable students with visual disabilities to access on-line educational material with significant mathematical content. The aim of this work is to enable visually impaired students to access and learn mathematics with a level of efficiency and following cognitive processes that are at par with those used by sighted students. In particular, we aim at creating a comprehensive framework which provides access to mathematical content at the analytical level (e.g., formulae, theorems, proofs), at the exploratory/intuitive level (e.g., content exploration, access to diagrams and charts), and at the practical level (e.g., problem solving, exercises).The guiding principle of this work is to understand and formalize the underlying semantic structure present in mathematical documents, starting from its basic components (e.g., individual formulae, components of a graph) and moving towards more complex organizations (e.g., logical steps, proofs, theorems, diagrams). Thesemantic structure is employed to guide multi-modal presentation schemes, which provide an automatically tuned balance between presentation (i.e., continuous access to the content) and navigation (i.e., user-directed traversal of the content), presentation adaptation based on user models, and combination of aural and haptic presentation. Haptic presentation relies on a novel hardware device (the haptic finger) to be designed and developed as part of this project. The framework we propose will also enable visually impaired students to interleave \reading" mathematics with \practicing" it, through an accessible workbook, which directly interacts with the reading process. Different levels of evaluation will be applied, involving visually impaired students early on in the design, and relying on formal cognitive models to assess progress.The project assembles a team of experienced researchers with expertise in the areas of applied cognitive/perceptual research, assistive technology software and hardware research, and usability design and evaluation. The researchers have collaborated previously, e.g., Tran and Pontelli conducted NSF-sponsored research in accessibility of the web for visually-impaired students.Intellectual Merit: (1) Observation and analysis of human-computer interaction by students with visual disabilities with mathematical content presented in electronic form, (2) Development of methodologies to extract semantic representations of mathematical content; (3) Design of a haptic device and control software for the presentation of graphical representations of mathematical concepts; (4) Study of approaches to auralpresentation and navigation of mathematical content; (4) Integration of haptic and aural presentation and navigation of mathematics, driven by user models, and composed with workspaces for accessible practice of mathematical concepts; (5) Usability evaluation with sighted and visually-impaired students.Broader Impact: (1) promote teaching, training, and learning, with focus on diversity due to visual disabilities; (2) broaden the representation of students with visual disabilities in their ability to access and navigate documents with heavy mathematical content (e.g., math textbooks), thereby increasing their educational opportunities in all areas, but especially in the sciences and engineering; (3) enhance infrastructure for education by improving the ability of usability experts to evaluate haptic and auditory interfaces in an educational setting. Ultimately, society will benefit from the greater inclusion in education and improved training opportunities for students with visual disabilities and generally by making information more widely accessible to these users. All PIs are actively involved in initiatives to advance the educational opportunitiesfor a diverse student population - e.g., NMSU is a Minority and Hispanic-serving Institution, and students from traditionally underrepresented groups will be involved in the research and educational activities.Keywords: accessibility of mathematics, haptic navigation, visual disabilities, usability evaluation methods