Space-Compacting Magnification Augmented with Natural Gestures and Keyboardless Text Entry for Low Vision Smartphone Interaction

通过自然手势和无键盘文本输入增强空间压缩放大倍率，实现低视力智能手机交互

• 批准号: 10368141
• 负责人: I.V. Ramakrishnan
• 金额: $ 35.78万
• 依托单位: STATE UNIVERSITY NEW YORK STONY BROOK
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-03-01 至 2025-02-28
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10368141
• 关键词: Accelerometer; Acute; Address; Affect; Age; Algorithms; Android; Area; Awareness; Cellular Phone; Complex; Controlled Study; Cues; Dependence; Development; Devices; Disabled Persons; Disadvantaged; Elderly; Elements; Employment Opportunities; Eye; Fingers; Friends; General Population; Gestures; Hand; In Situ; Information Technology; Learning; Manuals; Microscope; Motion; Outcome; Persons; Plant Roots; Population; Productivity; Psyche structure; Reader; Research; Self-Help Devices; Services; Sum; Surface; Taxes; Techniques; Technology; Telephone; Text; Touch sensation; Travel; Use Effectiveness; Vision; Vision Disorders; Visual; Visual impairment; barometric pressure; base; disability; experience; falls; graphical user interface; next generation; novel; operation; preservation; pressure sensor; recruit; sensor; synergism; usability; virtual; vision aid

Project Summary For people with low-vision impairments, the smartphone has become inextricably tied to their daily lives just like the general population. It is their go-to assistive device, relying on the smartphone’s built-in screen magnifier - Zoom on the iPhone and Magnifier on Android - to interact with it. But the usability of these screen magnifiers falls woefully short, adversely affecting productivity. First, the magnifier indiscriminately magnifies the raw screen pixels, including whitespace, as a blanket operation, causing occlusion of important contextual information such as visual cues from the user’s viewport. This necessitates panning over these occluded portions and mentally reconstructing the contextual information necessary for interacting with the content elements. Second, magnification gestures such as the bimanual multi-tap and multi-finger touch gestures tend to be more complex than the basic 1-finger swipe. The complexity of these touch gestures makes them cumbersome and tiring to use. Remembering the entire repertoire of gestures is also difficult. Since all these touch gestures involve some subset of finger combinations, it is easy to mix up one for the other. Third, virtual keyboards, which takes up significant screen real estate as-is, is difficult to use for text entry and editing in magnified view. Either the entire screen area is magnified including the virtual keyboard or only the display area. In the former, some of the keys are occluded from the view whereas in the latter the keys remain unmagnified. Regardless, key presses are hard to do in either cases. In sum, all these usability issues contribute to a vastly disproportionate gap in user experience and productivity between people with and without low-vision impairments. This proposal seeks to develop the next generation screen magnifier that will bridge this wide gap in user experience. It is rooted on three novel ideas. First, instead of indiscriminately magnifying the screen content as is done now, it will do object-aware magnification by identifying the objects in the graphical interface and compacting the space between the objects so as to keep contextually related objects close together in the magnified view. Second, by leveraging the untapped built-in sensors such as accelerometer, geometric field and barometric pressure sensors, it will expand the default surface gestures to include surfaceless natural gestures for magnification operations that can be done with only one hand, thus freeing the other hand for other tasks. More importantly, these gestures will be easy-to-do and easy-to-learn and recall. Third, it will incorporate a novel keyboardless gesture-based text entry and editing technique to eliminate the difficulties that arise with virtual keyboards for text entry in magnification mode. These three ideas will inform the development of CxZoom, a transformative, next-generation smartphone screen reader for low vision. CxZoom will make interaction with smartphones far more usable for people with low vision, thereby eliminating any barriers to their productivity and empower them to utilize the power and connectivity of these devices to fully participate in this digitized economy.

项目摘要 对于低视力障碍的人来说，智能手机已经与他们的日常生活密不可分 就像普通民众一样。这是他们的首选辅助设备，依靠智能手机的内置屏幕 放大镜--在iPhone上放大，在Android上放大镜--与它互动。但这些屏幕的可用性 可悲的是，放大镜不够用，对生产率产生了不利影响。首先，放大镜不加区别地放大 将包括空白在内的原始屏幕像素作为毯子操作，造成重要的上下文遮挡 信息，如来自用户视区的视觉提示。这就需要对这些遮挡部分进行平移 以及在心理上重建与内容元素交互所必需的上下文信息。 其次，放大手势，如双手多次敲击和多指触摸手势，往往更多 比基本的1指滑动更复杂。这些触摸手势的复杂性使它们变得笨拙和 使用起来很累。记住所有的手势也很困难。因为所有这些触摸手势都涉及到 手指组合的一些子集，很容易混淆一个和另一个。第三，虚拟键盘，这需要 在放大的情况下，很难使用显著的屏幕区域来进行文本输入和编辑。要么是 包括虚拟键盘或仅显示区域的整个屏幕区域被放大。在前者中，一些 关键点被遮挡在视图之外，而在后者中，关键点保持不放大。无论如何，按键都是 在这两种情况下都很难做到。总而言之，所有这些可用性问题都造成了极大的用户差距 低视力障碍患者和非低视力障碍患者之间的体验和工作效率。 这一提议旨在开发下一代屏幕放大镜，以弥合这一巨大的用户差距 经验。它植根于三个新颖的想法。首先，不是不加区别地将屏幕内容放大为 现在完成，它将通过识别图形界面中的对象来进行对象感知放大 压缩对象之间的空间，以使上下文相关的对象在 放大的视野。第二，利用未开发的内置传感器，如加速度计、几何场和 气压传感器，它将扩展默认的表面手势，包括无表面自然 仅用一只手即可完成放大操作的手势，从而释放另一只手 用手来完成其他任务。更重要的是，这些手势将易于操作，易于学习和回忆。第三，它 将结合一种新的基于手势的无键盘文本输入和编辑技术，以消除 在放大模式下用于文本输入的虚拟键盘出现的困难。这三个想法将给我们提供信息 CxZoom的开发，这是一款变革性的下一代智能手机屏幕阅读器，用于低视力。 CxZoom将使视力低下的人更容易与智能手机互动，从而消除 任何影响他们工作效率的障碍，并使他们能够利用这些设备的能力和连接性来 全面参与这个数字化经济。

项目成果

Semantic Table-of-Contents for Efficient Web Screen Reading.

• DOI: 10.1145/3412841.3442066
• 发表时间: 2021-03
• 期刊: Proceedings of the ... Symposium on Applied Computing. Symposium on Applied Computing
• 作者: Ferdous J;Uddin S;Ashok V
• 通讯作者: Ashok V

Modeling Touch Point Distribution with Rotational Dual Gaussian Model.

• DOI: 10.1145/3472749.3474816
• 发表时间: 2021-10
• 期刊: Proceedings of the ACM Symposium on User Interface Software and Technology. ACM Symposium on User Interface Software and Technology
• 作者: Ma Y;Zhai S;Ramakrishnan IV;Bi X
• 通讯作者: Bi X

TableView: Enabling Efficient Access to Web Data Records for Screen-Magnifier Users

• DOI: 10.1145/3373625.3417030
• 发表时间: 2020-10
• 期刊: Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility
• 作者: H. Lee;S. Uddin;V. Ashok

Variance and Distribution Models for Steering Tasks.

• DOI: 10.1145/3472749.3474811
• 发表时间: 2021-10
• 期刊: Proceedings of the ACM Symposium on User Interface Software and Technology. ACM Symposium on User Interface Software and Technology
• 作者: Wang M;Zhao H;Zhou X;Ren X;Bi X
• 通讯作者: Bi X