Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
复制标题
DOI：
10.1145/3411764.3445049
复制
发表时间：
2021-01
期刊：
影响因子：
--
通讯作者：
Toby Jia-Jun Li;Lindsay Popowski;Tom Michael Mitchell;B. Myers
中科院分区：
文献类型：
--
作者：
Toby Jia-Jun Li;Lindsay Popowski;Tom Michael Mitchell;B. Myers
表示图形用户界面和组件的语义对于用于建模用户-图形用户界面交互和挖掘图形用户界面设计的数据驱动计算方法至关重要。现有的图形用户界面语义表示仅限于对文本内容、视觉设计和布局模式或应用程序上下文进行编码。许多表示技术还需要大量的手动数据注释工作。本文提出了Screen2Vec，这是一种新的自监督技术，用于在图形用户界面屏幕和组件的嵌入向量中生成表示，这些图形用户界面和组件编码了上述所有图形用户界面特征，而不需要使用用户交互跟踪上下文进行手动注释。Screen2Vec的灵感来自单词嵌入方法word2vec，但它使用了一个新的两层管道，以图形用户界面和交互跟踪的结构为信息，并结合了特定于屏幕和应用程序的元数据。通过几个示例下游任务，我们演示了Screen2Vec的关键有用属性：通过最近的邻居表示屏幕间的相似性、可组合性和表示用户任务的能力。
Representing the semantics of GUI screens and components is crucial to data-driven computational methods for modeling user-GUI interactions and mining GUI designs. Existing GUI semantic representations are limited to encoding either the textual content, the visual design and layout patterns, or the app contexts. Many representation techniques also require significant manual data annotation efforts. This paper presents Screen2Vec, a new self-supervised technique for generating representations in embedding vectors of GUI screens and components that encode all of the above GUI features without requiring manual annotation using the context of user interaction traces. Screen2Vec is inspired by the word embedding method Word2Vec, but uses a new two-layer pipeline informed by the structure of GUIs and interaction traces and incorporates screen- and app-specific metadata. Through several sample downstream tasks, we demonstrate Screen2Vec’s key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks.