Next Generation Screen Magnification Technology for People with Low Vision

视力低下的人发现使用目前可用的屏幕放大镜是一项挑战，这是他们与计算设备交互的“首选”辅助技术。首先，这些放大镜不分青红皂白地放大屏幕内容，包括空白，作为一种毯子操作，通常会阻止重要的上下文信息，如视觉线索（例如，边界），以及用户视窗中不同用户界面元素之间的语义关系（例如，复选框及其标签），用户视窗是用户在给定时间可以看到的网页上的可见区域。从狭隘的视角在心理上重构上下文信息会极大地增加用户的负担。其次，低视力用户有广泛不同的需求，需要对界面元素进行一系列动态定制，这在目前可用的放大镜中是破坏性的，因为它会导致用户转移他们的焦点。第三，缺乏导航辅助工具来帮助用户浏览应用程序，获得快速概述，并轻松定位感兴趣的元素。拟议的项目将研究方向盘的设计和开发，这是一种变革性的下一代屏幕放大技术，以纠正这些限制。方向盘将基于几个新颖的想法。首先，它将以不同的方式放大空白和非空白用户界面（UI）元素，以便在放大后的视口中保持本地上下文。其次，它将光标移动限制在本地上下文中，从而限制了平移。第三，它将与物理表盘连接，支持简单的旋转和按下手势，带有听觉-触觉（触觉）反馈，这将使用户能够快速浏览不同的内容部分，轻松定位所需内容，快速浏览，并无缝地定制界面。该项目的一个副产品是创建标准化基准数据集，以衡量当前和未来屏幕放大技术的性能。预计SteeringWheel将使低视力用户更容易感知和消费数字信息，从而提高生产力。该项目将作为创建项目驱动的研究生和本科生无障碍计算课程的启动板。该项目将研究、设计和制造SteeringWheel，这是一种革命性的下一代屏幕放大技术，基于基于语义的位置保持放大（SLM）的基本思想。为了保留上下文信息，SLM结合了关于不同UI元素的语义和元素间关系的知识，这些知识植根于逻辑段的概念，逻辑段是相关UI元素的集合，在表示样式和空间位置上表现出一致性（例如，表单字段文本框及其相关的边界和标签）。SteeringWheel克服了现有屏幕放大技术的局限性，它基于两个科学理念，即将语义融入放大过程，辅以基于简单旋转和按压手势的触觉反馈交互范式，作为所有放大相关操作的“一体化”界面。研究计划有六个目标。目标1：针对桌面和移动设备不同屏幕尺寸的位置保存算法。提取算法将用于分析应用程序布局，识别语义上有意义的逻辑段，并通过以面向对象的方式组织这些段来生成语义层次结构。提取之后，将设计位置保持算法，在语义层次中对这些片段的不同类型的内容进行差异化放大。位置保留算法将在放大后将大部分（如果不是全部）上下文信息保留在用户的视口内，以最大限度地减少平移工作，从而减少相关的认知负担。将方向盘手势映射和集成到桌面的输入设备上。针对桌面平台的SteeringWheel界面将设计一套基于旋转和按压等简单动作的输入手势。这套手势将在微软的Surface Dial和游戏鼠标上实现。这些手势将使用户能够快速导航并轻松探索应用程序的不同部分，并根据需要无缝地进行放大调整。在移动设备的输入设备上映射和集成方向盘手势。一组输入手势是简单的旋转和按压的变化，将为针对移动平台的方向盘设计。这组手势将在苹果和安卓手表上实现。这些手势将使用户能够轻松地调整放大设置，方便地探索移动应用程序内容，而无需经历令人沮丧和耗时的交互过程，使用智能手机中默认的触摸手势（例如三重按）。OBJ 4：用于放大设置半自动个性化的基础设施。技术将被设计成支持半自动化的个性化，使用户只需进行一次定制，并且SteeringWheel将在每次用户重新访问应用程序时记住并自动应用这些用户的首选项。在单个段的粒度定制将得到支持，这将进一步减少用户的繁琐工作，自动应用自定义为一个段到所有其他类似的段。目标5：方向盘的口语对话助手。该助手的设计目的是帮助用户使用语音命令，如“带我到菜单栏”、“移动到第一个表单字段”等，轻松定位和转移导航焦点到感兴趣的部分或UI元素。该助手还允许用户指定命令，如“增加亮度”、“反转颜色”等，以定制界面。语音命令有可能将这种认知负担从用户转移到放大界面，从而让用户专注于他们的任务，而不是花费不必要的时间和精力来定位和手动配置各个部分和UI元素。OBJ 6：用于跨不同设备移植用户放大配置文件的基础设施。我们将设计不同的方法，以便将用户的配置文件（包含他们对不同应用程序的首选放大率设置）移植到不同的设备上，这样低视力用户就不需要在不同的设备上对相同的应用程序进行相同的放大率调整。还将设计机制，让用户安全地共享彼此的设置，这将进一步减少交互过载，因为具有相似眼睛状况的不同用户可能需要相同应用程序的相似放大设置。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

People with low vision find it challenging to use currently available screen magnifiers, their "go-to" assistive technology for interacting with computing devices. Firstly, these magnifiers indiscriminately magnify screen content, including white space, as a blanket operation, often blocking important contextual information such as visual cues (e.g., borders), and semantic relationships between different user interface elements (e.g., a checkbox and its label) from the user's viewport, which is the visible area on a web page that the user can see at a given time. Mentally reconstructing contextual information from exclusively narrow views dramatically increases the burden on the user. Secondly, low vision users have widely varying needs requiring a range of dynamic customizations for interface elements, which in currently available magnifiers is disruptive as it causes users to shift their focus. Thirdly, navigation aids to help users explore the application, obtain quick overviews, and easily locate elements of interest are lacking. The proposed project will research the design and development of SteeringWheel, a transformative next generation screen magnification technology to rectify these limitations. SteeringWheel will be based on several novel ideas. First, it will magnify the white space and non-white space user interface (UI) elements differently in order to keep the local context in the viewport post-magnification. Second, it will confine the cursor movement to the local context, thereby restricting panning. Third, it will interface with a physical dial, supporting simple rotate-and-press gestures with audio-haptic (sense of touch) feedback, that will enable users to quickly navigate different content sections, easily locate desired content, get a quick overview, and seamlessly customize the interface. A byproduct of the project will be the creation of standardized benchmark data sets to gauge the performance of current and future screen magnification technologies. It is anticipated that SteeringWheel will make it far easier for low vision users to perceive and consume digital information, leading to improved productivity. The project will serve as a launching board for creating project-driven graduate and undergraduate courses in Accessible Computing.The project will research, design and engineer SteeringWheel, a transformative next generation screen magnification technology that is predicated on the fundamental idea of Semantics-based Locality-Preserving Magnification (SLM). For retaining contextual information, SLM incorporates knowledge about the semantics of different UI elements and inter-element relationships, rooted in the concept of a logical segment, which is a collection of related UI elements exhibiting consistency in presentation style and spatial locality (e.g. a form-field textbox and its associated border and label). SteeringWheel, which overcomes the limitations of extant screen magnification technologies, rests on two scientific ideas, namely, incorporating semantics into the magnification process, complemented by an interaction paradigm based on simple rotate and press gestures with haptic feedback, that serves as an "all-in-one" interface for all magnification-related operations. The Research Plan is organized under six objectives. OBJ 1: Algorithms for locality-preservation targeting different screen-sizes for Desktops and Mobiles. Extraction algorithms will be designed to analyze the application layout, identify the semantically meaningful logical segments, and generate a semantic hierarchy by organizing these segments in an object-oriented fashion. Extraction will be followed by the design of locality-preserving algorithms that differentially magnify different types of content of these segments in the semantic hierarchy. Locality-preserving algorithms will keep most (if not all) of the contextual information within the users' viewport after magnification, to minimize the panning effort and hence the associated cognitive burden. OBJ 2: Mapping and integration of SteeringWheel gestures onto input devices for Desktops. A set of input gestures based on simple actions such as rotation and press will be designed for the SteeringWheel interface targeted at the Desktop platform. The set of gestures will be implemented on a Microsoft's Surface Dial and a Gaming Mouse. These gestures will enable users to quickly navigate and easily explore different segments of the application as well as seamlessly make magnification adjustments as needed. OBJ 3: Mapping and integration of SteeringWheel gestures onto input devices for Mobiles. A set of input gestures that are variations of simple rotation and press will be designed for the SteeringWheel targeted at the Mobile platform. The set of gestures will be implemented on Apple and Android Watches. These gestures will enable the users to easily adjust the magnification settings on-the-fly and conveniently explore the mobile application content without having to go through a frustrating and time-consuming interaction process using the default touch-based gestures (e.g. triple press) available in smart phones. OBJ 4: Infrastructure for semi-automated personalization of magnification settings. Techniques will be designed to support semi-automated personalization that enables users to make customizations only once, and SteeringWheel will remember and automatically apply these users' preferences each time they revisit the application. Customization at the granularity of individual segments will be supported, which will further reduce the user's tedious efforts by automatically applying the customizations made for one segment to all other similar segments. OBJ 5: Spoken dialog assistant for SteeringWheel. The Assistant will be designed to helps users easily locate and shift the navigation focus to segments or UI elements of interest using speech commands such as "take me to menu bar", "move to the first form field", etc. The Assistant will also allow users to specify commands such as "increase brightness", "invert colors", etc., for customizing the interface. Speech commands have the potential to transfer this cognitive burden from users to the magnification interface, thereby letting users focus on their tasks instead of expending needless time and effort locating and manually configuring individual segments and UI elements. OBJ 6: Infrastructure for porting users' magnification profile across different devices. Different methods will be designed to facilitate porting of users' profiles containing their preferred magnification settings for different applications to different devices, so that the low vision users need not make the same magnification adjustments for the same applications on different devices. Mechanisms will also be designed for letting users securely share their settings with each other, which will further reduce their interaction overload, as different users with similar eye conditions may need similar magnification settings for the same application.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.