CAREER: Pixel-Based Interpretation and Modification of Graphical User Interfaces

几乎每个图形用户界面（GUI）都是使用某种形式的GUI工具包实现的。虽然这些工具包在过去四十年的人机交互（HCI）研究和实践中取得了许多成功，但不幸的是，它们也变得令人窒息。当前GUI工具包的刚性和碎片性使得在复杂的现有应用程序中部署新想法变得困难或不可能，因此研究人员通常限于演示和评估小型玩具应用程序。这是该领域面临的一个重大挑战，因为它限制了HCI研究的进展和影响。PI的目标是超越现代gui的碎片化，建立在它们最大的共性之上，即所有gui最终都由绘制到显示器上的像素组成。他认为，如果有可能快速而健壮地解释这些像素，而不需要应用程序源代码并且独立于其底层工具包实现，我们就可以使用这种理解来修改任何接口。PI之前在Prefab系统上的工作已经显示出有希望的初步结果，这是通过逆向工程小部件外观策略实现的。在当前的项目中，PI将开发基于像素的图形用户界面解释方法，包括识别复杂小部件、提取小部件内容和恢复小部件关系的方法，他将在这些方法的基础上建立小部件状态模型，并解决界面部分遮挡的问题。他将创建用于修改图形用户界面的方法，包括基于像素的解释的语义视图、操作底层接口的方法、表示之间的转换方法、可视化编程方法和用于界面定制的最终用户工具。他将实现用于通知、部署和评估该领域交互研究的新工具，并将与领先的研究人员合作，将这些工具应用于目标感知指向和自动界面生成。他将在从广泛部署中收集的真实数据集中验证他的基于像素的方法。更广泛的影响：在这个项目中，PI将追求交互的每个方面的变革民主化的愿景，在这个愿景中，任何人都可以修改任何应用程序的任何接口，而不是局限于提供的单个接口。例如，HCI研究人员可能会在几个现有的应用程序中实现和评估新的交互技术；看到出版物的从业者或业余爱好者可能会将该技术添加到他们喜欢的应用程序中。Web社区可能会围绕一些原因进行组织，比如将接口翻译成新的语言、改进现有应用程序的可访问性，或者更新接口以支持手势、语音访问或其他交互。最终用户可以浏览扩展库，投票选出他们最喜欢的，并使用可视化编程工具创建自己的扩展库。项目成果，包括工具和数据集，将公开传播。企划团还将特别努力，从弱势群体中招收学生，将研究成果与现有课程和新课程相结合，并利用高中设计竞赛作为招收高中生实习的渠道。

Nearly every graphical user interface (GUI) is implemented using some form of GUI toolkit. While these toolkits have enabled many successes of the past forty years of research and practice in human-computer interaction (HCI), they have unfortunately also become stifling. The rigidity and fragmentation of current GUI toolkits makes it difficult or impossible to deploy new ideas in complex existing applications, so researchers are instead often limited to demonstrating and evaluating small toy applications. This is a major challenge for the field, as it limits both the progress and impact of HCI research. The PI's goal is to transcend the fragmentation of modern GUIs by building upon their single largest commonality, namely that all GUIs ultimately consist of pixels painted to a display. He argues that if it were possible to quickly and robustly interpret those pixels, without application source code and independent of its underlying toolkit implementation, we could use that understanding to modify any interface. The PI's prior work on the Prefab system has shown promising preliminary results, enabled by a strategy of reverse engineering widget appearance. In the current project the PI will develop methods for pixel-based interpretation of graphical user interfaces, including methods for identifying complex widgets, for extracting widget content, and for recovering widget relationships, and he will build upon these to model widgets state and to address occlusion of portions of interfaces. He will create methods for modifying graphical user interfaces, including semantic views on pixel-based interpretation, methods for manipulating underlying interfaces, methods for translation between presentations, visual programming methods, and end-user tools for interface customization. He will implement new tools for informing, deploying, and evaluating interaction research in the field, and he will partner with leading researchers to apply these tools to target-aware pointing and automatic interface generation. And he will validate his pixel-based methods in real-world datasets collected from broad deployments.Broader Impacts: In this project the PI will pursue a vision of a transformative democratization of every aspect of interaction, in which instead of being limited to a single provided interface anybody can modify any interface of any application. For example, an HCI researcher might implement and evaluate a new interaction technique in several existing applications; a practitioner or hobbyist who sees the publication might add the technique to their favorite applications. Web communities might organize around causes, such as translating interfaces into new languages, improving the accessibility of existing applications, or updating interfaces to support gestures, speech access, or other interactions. End-users might browse libraries of extensions, vote on their favorites, and use visual programming tools to create their own. Project outcomes, including tools and dataset, will be openly disseminated. The PI will make special efforts to recruit students from under-represented groups, to integrate the research into existing and new courses, and to use a high school design competition as a channel for recruiting high school students for internships.