CAREER: Interactive Program Synthesis for Web Automation

职业：Web 自动化的交互式程序综合

• 批准号: 2236233
• 负责人: Xinyu Wang
• 金额: $ 52.49万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-03-01 至 2028-02-29
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2236233&HistoricalAwards=false
• 关键词: CAREER; Interactive; Program; Synthesis; Web

Process Automation continues to be a main driver of digital transformation. Yet, it is still technically very demanding to create such automation programs. This project aims to significantly lower the technical barrier of creating web automation programs. The project’s novelties are a suite of new rewrite-based program synthesis algorithms that can automatically generate programs from user demonstrations. The project’s impacts are to enable non-experts in need of performing a tedious but programmatic web-related task to create a program to automate the work, even if they have little or no background in web programming. At the core, this project is developing algorithms that take as input a user demonstration – in the form of a trace A of user actions (e.g., clicking buttons, scraping data) – and synthesize a parameterized program P with control-flow structures by rewriting A to P. The approach is based on finite tree automata and involves both neural and symbolic elements in the underlying web automation language, with the goal of being able to effectively reason about the webpage contents while still leveraging the underlying webpage structure. The project is curating a new suite of web automation tasks that are independently useful for future research beyond this project. The investigator is collaborating with partners at Michigan (e.g., Women in Science and Engineering (WISE), Michigan Louis Stokes Alliance for Minority Participation (MI-LSAMP) and Engineering Center for Academic Success (ECAS)) to increase participation of students from underrepresented groups in his research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

过程自动化仍然是数字化转型的主要驱动力。然而，创建这样的自动化程序在技术上仍然非常苛刻。该项目旨在显著降低创建Web自动化程序的技术障碍。该项目的新颖之处是一套新的基于重写的程序合成算法，可以从用户演示中自动生成程序。该项目的影响是使需要执行繁琐但程序化的Web相关任务的非专家能够创建一个程序来自动化工作，即使他们在Web编程方面几乎没有背景。在核心，这个项目正在开发算法，作为输入的用户演示-在用户操作的痕迹A的形式（例如，点击按钮，抓取数据）-并通过将A重写为P来合成具有控制流结构的参数化程序P。该方法基于有限树自动机，并涉及底层web自动化语言中的神经和符号元素，目标是能够有效地推理网页内容，同时仍然利用底层网页结构。该项目正在策划一套新的Web自动化任务，这些任务对本项目以外的未来研究独立有用。研究人员正在与密歇根州的合作伙伴合作（例如，该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

DiLogics: Creating Web Automation Programs with Diverse Logics

• DOI: 10.1145/3586183.3606822
• 发表时间: 2023-08
• 期刊: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
• 作者: Kevin Pu;Jim Yang;Angel Yuan;Minyi Ma;Rui Dong;Xinyu Wang;Yuanchun Chen;Tovi Grossman

Efficient Bottom-Up Synthesis for Programs with Local Variables

具有局部变量的程序的高效自下而上综合

• DOI: 10.1145/3632894
• 发表时间: 2024
• 期刊: Proceedings of the ACM on Programming Languages
• 作者: Li, Xiang;Zhou, Xiangyu;Dong, Rui;Zhang, Yihong;Wang, Xinyu
• 通讯作者: Wang, Xinyu

MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence

MIWA：混合主动 Web 自动化可实现更好的用户控制和信心

• DOI: 10.1145/3586183.3606720
• 发表时间: 2023
• 期刊: ACM
• 作者: Chen, Weihao;Liu, Xiaoyu;Zhang, Jiacheng;Lam, Ian Iong;Huang, Zhicheng;Dong, Rui;Wang, Xinyu;Zhang, Tianyi
• 通讯作者: Zhang, Tianyi