Collaborative Research: FMitF: Track I: Usable Synthesis-based End-User Programming with Rich Interaction Modalities

合作研究：FMitF：第一轨：具有丰富交互方式的可用的基于综合的最终用户编程

• 批准号: 2123654
• 负责人: Xinyu Wang
• 金额: $ 40万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2123654&HistoricalAwards=false
• 关键词: Collaborative; Research; FMitF; Track; Usable

Programming, defined as the process of transforming a mental plan of actions into a representation that can be executed by a computer, has been recognized as a powerful, desirable, and even necessary skill for an increasing number of people beyond professional programmers. In parallel, it has been a longstanding challenge in computer science to provide easier, more supportive interfaces for programming, which may be especially useful for computer users who want to leverage the power of programming but have little or no formal training. This project aims to develop software development tools and methods that help people to interactively create programs for data science and data visualization, using suggestions and feedback provided by the tools. The tools will allow users to express their goals through gestures, voice, and other familiar ways people interact with the world, providing rapid and ongoing feedback to help people zero in on the programs they are looking to create. To achieve this goal, this project is combining techniques from human-computer interaction, programming languages, and formal methods. On the one hand, this project is contributing a program-synthesis framework that supports rich interaction modalities, including accepting multi-modal specifications and providing feedback during the synthesis process. On the other hand, this project is also contributing novel user interfaces that allow end-users to naturally express their intent in different modalities, to better understand the characteristics of the program(s) generated on their behalf, and to more easily recognize and update their expression of intent. By synergistically combining strengths of these two areas, the project aims to lay the formal methods foundation for usable human-centered synthesis-based interactive systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

编程，定义为将心理行动计划转换为可由计算机执行的表示的过程，已被越来越多的专业程序员以外的人认为是一种强大的，理想的，甚至是必要的技能。与此同时，为编程提供更容易、更支持的接口一直是计算机科学中的一个长期挑战，这对于那些想要利用编程能力但很少或没有受过正式培训的计算机用户来说可能特别有用。该项目旨在开发软件开发工具和方法，帮助人们使用工具提供的建议和反馈，交互式地创建数据科学和数据可视化程序。 这些工具将允许用户通过手势、语音和其他人们熟悉的与世界互动的方式来表达他们的目标，提供快速和持续的反馈，帮助人们专注于他们想要创建的程序。 为了实现这一目标，该项目结合了人机交互，编程语言和形式化方法的技术。一方面，该项目正在提供一个程序合成框架，支持丰富的交互方式，包括接受多模式规范并在合成过程中提供反馈。另一方面，该项目还提供了新颖的用户界面，使最终用户能够以不同的方式自然地表达他们的意图，更好地理解代表他们生成的程序的特征，并更容易地识别和更新他们的意图表达。通过协同结合这两个领域的优势，该项目旨在为可用的以人为中心的基于合成的交互系统奠定正式的方法基础。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

INTENT: Interactive Tensor Transformation Synthesis

意图：交互式张量变换合成

• DOI: 10.1145/3526113.3545653
• 发表时间: 2022
• 期刊: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
• 作者: Zhou, Zhanhui;Tang, Man To;Pan, Qiping;Tan, Shangyin;Wang, Xinyu;Zhang, Tianyi
• 通讯作者: Zhang, Tianyi

DiLogics: Creating Web Automation Programs with Diverse Logics

• DOI: 10.1145/3586183.3606822
• 发表时间: 2023-08
• 期刊: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
• 作者: Kevin Pu;Jim Yang;Angel Yuan;Minyi Ma;Rui Dong;Xinyu Wang;Yuanchun Chen;Tovi Grossman

SemanticOn: Specifying Content-Based Semantic Conditions for Web Automation Programs

SemanticOn：为 Web 自动化程序指定基于内容的语义条件

• DOI: 10.1145/3526113.3545691
• 发表时间: 2022
• 期刊: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
• 作者: Pu, Kevin;Fu, Rainey;Dong, Rui;Wang, Xinyu;Chen, Yan;Grossman, Tovi
• 通讯作者: Grossman, Tovi

MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence

MIWA：混合主动 Web 自动化可实现更好的用户控制和信心

• DOI: 10.1145/3586183.3606720
• 发表时间: 2023
• 期刊: ACM
• 作者: Chen, Weihao;Liu, Xiaoyu;Zhang, Jiacheng;Lam, Ian Iong;Huang, Zhicheng;Dong, Rui;Wang, Xinyu;Zhang, Tianyi
• 通讯作者: Zhang, Tianyi

WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration

WebRobot：使用交互式演示编程的网络机器人流程自动化

• DOI: 10.1145/3519939.3523711
• 发表时间: 2022
• 期刊: Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation
• 作者: Dong, Rui;Huang, Zhicheng;Lam, Ian Iong;Chen, Yan;Wang, Xinyu
• 通讯作者: Wang, Xinyu