Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps
Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps
复制标题
DOI：
10.1109/tse.2018.2844788
复制
发表时间：
2020-02-01
影响因子：
7.4
通讯作者：
Poshyvanyk, Denys
中科院分区：
文献类型：
--
作者：
Moran, Kevin;Bernal-Cardenas, Carlos;Poshyvanyk, Denys
面向用户的软件开发人员通常会将图形用户界面（GUI）的模型转换为代码。这个过程发生在应用程序的初始阶段，也发生在演化的上下文中，因为GUI的变化与演化的特性保持同步。不幸的是，这种做法既具有挑战性又耗时。在本文中，我们提出了一种方法，通过三个任务：检测，分类和组装，使准确的原型GUI自动化这一过程。首先，使用计算机视觉技术或实体模型元数据从实体模型工件检测GUI的逻辑组件。然后，利用软件存储库挖掘、自动化动态分析和深度卷积神经网络来将GUI组件准确地分类为域特定类型（例如，切换按钮）。最后，一个数据驱动的，K-近邻算法生成一个合适的层次化的GUI结构，原型应用程序可以自动组装。我们在一个名为ReDraw的系统中为Android实现了这种方法。我们的评估表明，ReDraw实现了91%的平均GUI组件分类准确率，并组装了原型应用程序，在视觉亲和力方面密切反映了目标模型，同时表现出合理的代码结构。对行业从业者的采访说明了ReDraw改进真实的开发工作流程的潜力。
It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application's inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called ReDraw. Our evaluation illustrates that ReDraw achieves an average GUI-component classification accuracy of 91 percent and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw's potential to improve real development workflows.