Learning Design Semantics for Mobile Apps
Learning Design Semantics for Mobile Apps
复制标题
DOI：
10.1145/3242587.3242650
复制
发表时间：
2018-10
期刊：
影响因子：
--
通讯作者：
Thomas F. Liu;Mark Craft;Jason Situ;Ersin Yumer;R. Mech;Ranjitha Kumar
中科院分区：
文献类型：
--
作者：
Thomas F. Liu;Mark Craft;Jason Situ;Ersin Yumer;R. Mech;Ranjitha Kumar
最近，研究人员开发了一种黑盒方法，用于矿山设计和来自移动应用程序的交互数据。尽管在此交互挖掘过程中捕获的数据是描述性的，但它并不公开ui的设计语义：屏幕上的元素意味着什么以及如何使用它们。本文介绍了一种自动生成移动应用界面语义注释的方法。通过73k UI元素和720个屏幕的迭代开放编码，我们贡献了一个包含25种UI组件、197个文本按钮概念和135个跨应用程序共享的图标类的词汇数据库。我们使用这些标记的数据来学习基于代码的模式来检测UI组件，并训练一个卷积神经网络，以94%的准确率区分图标类别。为了证明我们的方法在规模上的有效性，我们为Rico数据集中的72k唯一ui计算语义注释，为78%的可见非冗余元素分配标签。
Recently, researchers have developed black-box approaches to mine design and interaction data from mobile apps. Although the data captured during this interaction mining is descriptive, it does not expose the design semantics of UIs: what elements on the screen mean and how they are used. This paper introduces an automatic approach for generating semantic annotations for mobile app UIs. Through an iterative open coding of 73k UI elements and 720 screens, we contribute a lexical database of 25 types of UI components, 197 text button concepts, and 135 icon classes shared across apps. We use this labeled data to learn code-based patterns to detect UI components and to train a convolutional neural network that distinguishes between icon classes with 94% accuracy. To demonstrate the efficacy of our approach at scale, we compute semantic annotations for the 72k unique UIs in the Rico dataset, assigning labels for 78% of the total visible, non-redundant elements.