An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation
An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation
复制标题
DOI：
10.1109/saner53432.2022.00069
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Kevin Moran;Ali Yachnes;George Purnell;Juanyed Mahmud;Michele Tufano;Carlos Bernal Cardenas;D. Poshyvanyk;Zach H’Doubler
中科院分区：
文献类型：
--
作者：
Kevin Moran;Ali Yachnes;George Purnell;Juanyed Mahmud;Michele Tufano;Carlos Bernal Cardenas;D. Poshyvanyk;Zach H’Doubler
用于软件文档的现有自动化技术通常会尝试在两个主要信息来源之间推理：代码和自然语言。但是，这种推理过程通常会因更抽象的自然语言和更结构化的编程语言之间的词汇差距而变得复杂。该差距的一个潜在桥梁是图形用户界面（GUI），因为GUI固有地编码了有关基础程序功能的显着信息，以计入富的基于像素的数据表示。本文对GUIS与软件的功能自然语言描述之间的联系进行了首次全面的实证研究之一。首先，我们收集，分析和开源一个功能性GUI描述的大型数据集，其中包括来自流行的Android应用程序的10,204个屏幕截图的45,998个描述。这些描述是从人类标记中获得的，并经历了多种质量控制机制。为了深入了解GUIS的代表性潜力，我们研究了四个神经图像字幕模型预测自然语言描述的能力，当提供屏幕快照作为输入时。我们使用通用的机器翻译指标定量评估这些模型，并通过大规模的用户研究定性地评估这些模型。最后，我们提供了学习的课程，并讨论了多模型模型所显示的潜力，以增强自动化软件文档的未来技术。
Existing automated techniques for software documentation typically attempt to reason between two main sources of information: code and natural language. However, this reasoning process is often complicated by the lexical gap between more abstract natural language and more structured programming languages. One potential bridge for this gap is the Graphical User Interface (GUI), as GUIs inherently encode salient information about underlying program functionality into rich, pixel-based data representations. This paper offers one of the first comprehensive empirical investigations into the connection between GUIs and functional, natural language descriptions of software. First, we collect, analyze, and open source a large dataset of functional GUI descriptions consisting of 45,998 descriptions for 10,204 screenshots from popular Android applications. The descriptions were obtained from human labelers and underwent several quality control mechanisms. To gain insight into the representational potential of GUIs, we investigate the ability of four Neural Image Captioning models to predict natural language descriptions of varying granularity when provided a screenshot as input. We evaluate these models quantitatively, using common machine translation metrics, and qualitatively through a large-scale user study. Finally, we offer learned lessons and a discussion of the potential shown by multimodal models to enhance future techniques for automated software documentation.