CAREER: Pixel-Based Interpretation and Modification of Graphical User Interfaces

职业：图形用户界面的基于像素的解释和修改

• 批准号: 1053868
• 负责人: James Fogarty
• 金额: $ 56.51万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-03-01 至 2017-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1053868&HistoricalAwards=false
• 关键词: CAREER; Pixel; Based; Interpretation; Modification

Nearly every graphical user interface (GUI) is implemented using some form of GUI toolkit. While these toolkits have enabled many successes of the past forty years of research and practice in human-computer interaction (HCI), they have unfortunately also become stifling. The rigidity and fragmentation of current GUI toolkits makes it difficult or impossible to deploy new ideas in complex existing applications, so researchers are instead often limited to demonstrating and evaluating small toy applications. This is a major challenge for the field, as it limits both the progress and impact of HCI research. The PI's goal is to transcend the fragmentation of modern GUIs by building upon their single largest commonality, namely that all GUIs ultimately consist of pixels painted to a display. He argues that if it were possible to quickly and robustly interpret those pixels, without application source code and independent of its underlying toolkit implementation, we could use that understanding to modify any interface. The PI's prior work on the Prefab system has shown promising preliminary results, enabled by a strategy of reverse engineering widget appearance. In the current project the PI will develop methods for pixel-based interpretation of graphical user interfaces, including methods for identifying complex widgets, for extracting widget content, and for recovering widget relationships, and he will build upon these to model widgets state and to address occlusion of portions of interfaces. He will create methods for modifying graphical user interfaces, including semantic views on pixel-based interpretation, methods for manipulating underlying interfaces, methods for translation between presentations, visual programming methods, and end-user tools for interface customization. He will implement new tools for informing, deploying, and evaluating interaction research in the field, and he will partner with leading researchers to apply these tools to target-aware pointing and automatic interface generation. And he will validate his pixel-based methods in real-world datasets collected from broad deployments.Broader Impacts: In this project the PI will pursue a vision of a transformative democratization of every aspect of interaction, in which instead of being limited to a single provided interface anybody can modify any interface of any application. For example, an HCI researcher might implement and evaluate a new interaction technique in several existing applications; a practitioner or hobbyist who sees the publication might add the technique to their favorite applications. Web communities might organize around causes, such as translating interfaces into new languages, improving the accessibility of existing applications, or updating interfaces to support gestures, speech access, or other interactions. End-users might browse libraries of extensions, vote on their favorites, and use visual programming tools to create their own. Project outcomes, including tools and dataset, will be openly disseminated. The PI will make special efforts to recruit students from under-represented groups, to integrate the research into existing and new courses, and to use a high school design competition as a channel for recruiting high school students for internships.

几乎每个图形用户界面(GUI)都是使用某种形式的图形用户界面工具包实现的。虽然这些工具包使过去40年来在人机交互(HCI)方面的研究和实践取得了许多成功，但不幸的是，它们也变得令人窒息。当前图形用户界面工具包的僵化和碎片化使得很难或不可能在复杂的现有应用程序中部署新的想法，因此研究人员往往仅限于演示和评估小型玩具应用程序。这是该领域的一个重大挑战，因为它限制了人机界面研究的进展和影响。PI的目标是通过构建现代图形用户界面最大的共同点，即所有图形用户界面最终由绘制到显示器上的像素组成，从而超越现代图形用户界面的碎片化。他认为，如果有可能快速而可靠地解释这些像素，而不需要应用程序源代码，并且独立于其底层工具包实现，我们就可以使用这种理解来修改任何接口。PI之前在预制系统上的工作已经显示出有希望的初步结果，这是通过反向工程小部件外观的策略实现的。在当前项目中，PI将开发基于像素的图形用户界面解释方法，包括识别复杂小部件、提取小部件内容和恢复小部件关系的方法，他将在这些方法的基础上对小部件状态进行建模，并解决界面部分的遮挡问题。他将创建修改图形用户界面的方法，包括基于像素的解释的语义视图、操作底层界面的方法、演示之间的转换方法、可视化编程方法以及用于界面定制的最终用户工具。他将实施新的工具来通知、部署和评估该领域的交互研究，并将与领先的研究人员合作，将这些工具应用于目标感知指向和自动界面生成。他将在从广泛部署收集的真实世界数据集中验证他的基于像素的方法。广泛影响：在这个项目中，PI将追求交互的各个方面的变革性民主化的愿景，其中不限于单一提供的界面，任何人都可以修改任何应用程序的任何界面。例如，人机界面研究人员可能会在几个现有的应用程序中实现和评估一种新的交互技术；看到该出版物的从业者或爱好者可能会将该技术添加到他们最喜欢的应用程序中。Web社区可能会围绕各种原因组织起来，例如将界面翻译成新的语言、改进现有应用程序的可访问性或更新界面以支持手势、语音访问或其他交互。最终用户可以浏览扩展库，投票选择他们最喜欢的，并使用可视化编程工具创建他们自己的扩展库。项目成果，包括工具和数据集，将公开分发。PI将特别努力从代表不足的群体中招收学生，将研究整合到现有和新的课程中，并将高中设计竞赛作为招募高中生实习的渠道。

项目成果

Epidemiology as a Framework for Large-Scale Mobile Application Accessibility Assessment

流行病学作为大规模移动应用程序可访问性评估的框架

• DOI: 10.1145/3132525.3132547
• 发表时间: 2017
• 期刊: Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2017
• 作者: Ross, Anne Spencer;Zhang, Xiaoyi;Fogarty, James;Wobbrock, Jacob O.
• 通讯作者: Wobbrock, Jacob O.