Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
复制标题
DOI：
10.1145/3411764.3445186
复制
发表时间：
2021-01
期刊：
影响因子：
--
通讯作者：
Xiaoyi Zhang;Lilian de Greef;Amanda Swearngin;Samuel White;Kyle I. Murray;Lisa Yu;Qi Shan;Jeffrey Nichols;Jason Wu;Chris Fleizach;Aaron Everitt;Jeffrey P. Bigham
中科院分区：
文献类型：
--
作者：
Xiaoyi Zhang;Lilian de Greef;Amanda Swearngin;Samuel White;Kyle I. Murray;Lisa Yu;Qi Shan;Jeffrey Nichols;Jason Wu;Chris Fleizach;Aaron Everitt;Jeffrey P. Bigham
移动平台上的许多可访问功能都需要应用程序（应用程序）来描述用户界面（UI）组件的完整元素。来自其像素的移动应用程序的可访问性元数据，因为视觉接口通常最好反映出应用程序的完整功能。记忆效率的，设备的模型使用77,637个屏幕（来自4,068个iPhone应用程序）检测UI元素，我们收集并注释了，以进一步改善UI检测并添加语义信息，我们引入了启发式启发式信息（例如，UI分组和订购，订购，订购，订购和订购，订购，订购和订购信息）和其他模型（例如，识别UI内容，状态，交互性）使用9个屏幕读取器用户进行研究，我们验证了我们的方法改善了现有移动应用程序的可访问性，甚至可以使用以前无法访问的应用程序。
Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app’s full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.