CAREER: Pixel-Based Interpretation and Modification of Graphical User Interfaces

职业：图形用户界面的基于像素的解释和修改

• 批准号: 1053868
• 负责人: James Fogarty
• 金额: $ 56.51万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-03-01 至 2017-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1053868&HistoricalAwards=false
• 关键词: CAREER; Pixel; Based; Interpretation; Modification

Nearly every graphical user interface (GUI) is implemented using some form of GUI toolkit. While these toolkits have enabled many successes of the past forty years of research and practice in human-computer interaction (HCI), they have unfortunately also become stifling. The rigidity and fragmentation of current GUI toolkits makes it difficult or impossible to deploy new ideas in complex existing applications, so researchers are instead often limited to demonstrating and evaluating small toy applications. This is a major challenge for the field, as it limits both the progress and impact of HCI research. The PI's goal is to transcend the fragmentation of modern GUIs by building upon their single largest commonality, namely that all GUIs ultimately consist of pixels painted to a display. He argues that if it were possible to quickly and robustly interpret those pixels, without application source code and independent of its underlying toolkit implementation, we could use that understanding to modify any interface. The PI's prior work on the Prefab system has shown promising preliminary results, enabled by a strategy of reverse engineering widget appearance. In the current project the PI will develop methods for pixel-based interpretation of graphical user interfaces, including methods for identifying complex widgets, for extracting widget content, and for recovering widget relationships, and he will build upon these to model widgets state and to address occlusion of portions of interfaces. He will create methods for modifying graphical user interfaces, including semantic views on pixel-based interpretation, methods for manipulating underlying interfaces, methods for translation between presentations, visual programming methods, and end-user tools for interface customization. He will implement new tools for informing, deploying, and evaluating interaction research in the field, and he will partner with leading researchers to apply these tools to target-aware pointing and automatic interface generation. And he will validate his pixel-based methods in real-world datasets collected from broad deployments.Broader Impacts: In this project the PI will pursue a vision of a transformative democratization of every aspect of interaction, in which instead of being limited to a single provided interface anybody can modify any interface of any application. For example, an HCI researcher might implement and evaluate a new interaction technique in several existing applications; a practitioner or hobbyist who sees the publication might add the technique to their favorite applications. Web communities might organize around causes, such as translating interfaces into new languages, improving the accessibility of existing applications, or updating interfaces to support gestures, speech access, or other interactions. End-users might browse libraries of extensions, vote on their favorites, and use visual programming tools to create their own. Project outcomes, including tools and dataset, will be openly disseminated. The PI will make special efforts to recruit students from under-represented groups, to integrate the research into existing and new courses, and to use a high school design competition as a channel for recruiting high school students for internships.

几乎每个图形用户界面（GUI）都是使用某种形式的GUI工具包实现的。 虽然这些工具包在过去四十年的人机交互（HCI）研究和实践中取得了许多成功，但不幸的是，它们也变得令人窒息。 当前GUI工具箱的僵化和碎片化使得在复杂的现有应用程序中部署新想法变得困难或不可能，因此研究人员通常仅限于演示和评估小型玩具应用程序。 这是该领域的一个重大挑战，因为它限制了HCI研究的进展和影响。 PI的目标是通过构建它们最大的共同点来超越现代GUI的碎片化，即所有GUI最终都由绘制到显示器上的像素组成。 他认为，如果能够快速、鲁棒地解释这些像素，而无需应用程序源代码并且独立于其底层工具包实现，我们就可以使用这种理解来修改任何接口。 PI先前在Prefab系统上的工作已经显示出有希望的初步结果，这是通过逆向工程部件外观的策略实现的。 在目前的项目中，PI将开发基于像素的图形用户界面解释方法，包括识别复杂小部件的方法，提取小部件内容的方法，以及恢复小部件关系的方法，他将在这些基础上建立小部件状态模型并解决界面部分的遮挡问题。 他将创建用于修改图形用户界面的方法，包括基于像素的解释的语义视图，操纵底层界面的方法，演示文稿之间的转换方法，可视化编程方法和界面定制的最终用户工具。 他将实施新的工具，用于通知，部署和评估该领域的交互研究，他将与领先的研究人员合作，将这些工具应用于目标感知指向和自动界面生成。 更广泛的影响：在这个项目中，PI将追求交互的每个方面的变革民主化的愿景，在这个愿景中，任何人都可以修改任何应用程序的任何接口，而不是局限于一个单一的接口。 例如，HCI研究人员可能会在几个现有的应用程序中实现和评估一种新的交互技术;看到出版物的从业者或爱好者可能会将该技术添加到他们最喜欢的应用程序中。 网络社区可能会围绕一些原因组织起来，比如将界面翻译成新的语言，提高现有应用程序的可访问性，或者更新界面以支持手势、语音访问或其他交互。 最终用户可以浏览扩展库，投票选出他们最喜欢的扩展，并使用可视化编程工具创建自己的扩展。 将公开传播项目成果，包括工具和数据集。 PI将特别努力从代表性不足的群体中招收学生，将研究纳入现有和新课程，并利用高中设计竞赛作为招募高中生实习的渠道。

项目成果

Epidemiology as a Framework for Large-Scale Mobile Application Accessibility Assessment

流行病学作为大规模移动应用程序可访问性评估的框架

• DOI: 10.1145/3132525.3132547
• 发表时间: 2017
• 期刊: Proceedings of the ACM Conference on Computers and Accessibility (ASSETS 2017
• 作者: Ross, Anne Spencer;Zhang, Xiaoyi;Fogarty, James;Wobbrock, Jacob O.
• 通讯作者: Wobbrock, Jacob O.