A Human Study of Comprehension and Code Summarization
A Human Study of Comprehension and Code Summarization
复制标题
DOI：
10.1145/3387904.3389258
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Sean Stapleton;Yashmeet Gambhir;Alexander LeClair;Zachary Eberhart;Westley Weimer;Kevin Leach;Yu Huang-Y
中科院分区：
文献类型：
--
作者：
Sean Stapleton;Yashmeet Gambhir;Alexander LeClair;Zachary Eberhart;Westley Weimer;Kevin Leach;Yu Huang-Y
软件开发人员花费大量时间阅读和理解文档记录不佳、由其他开发人员编写或使用不同风格开发的代码。在过去的十年中，研究人员研究了自动记录代码以提高可理解性的技术。特别是，深度学习的最新进展导致了复杂的摘要生成技术，可以将函数或方法转换为简单的英文字符串，简洁地描述代码的行为。然而，自动摘要技术是使用内部指标来评估的，比如BLEU分数，它测量翻译模型中的自然语言属性，或者ROUGE分数，它测量与人类书面文本的重叠。不幸的是，这些指标并不一定能够捕捉到机器生成的代码摘要实际上是如何影响人类的理解能力或开发人员的生产力的。我们进行了一项涉及大学生和专业开发人员的人体研究（n = 45）。参与者回顾了Java方法和摘要，并回答了既定的程序理解问题。此外，参与者还完成了给出摘要作为规范的编码任务。关键是，实验控制了摘要的来源：对于给定的方法，一些参与者看到的是人类编写的文本，一些参与者看到的是机器生成的文本。我们发现，与机器生成的摘要相比，使用人工编写的摘要的参与者表现明显更好（p = 0.029）。然而，我们没有发现证据支持参与者认为人类和机器生成的摘要具有不同的质量。此外，参与者的表现与BLEU和ROUGE分数没有相关性，BLEU和ROUGE分数通常用于评估机器生成摘要的质量。这些结果表明需要修订度量来评估和指导自动总结技术。
Software developers spend a great deal of time reading and understanding code that is poorly-documented, written by other developers, or developed using differing styles. During the past decade, researchers have investigated techniques for automatically documenting code to improve comprehensibility. In particular, recent advances in deep learning have led to sophisticated summary generation techniques that convert functions or methods to simple English strings that succinctly describe that code's behavior. However, automatic summarization techniques are assessed using internal metrics such as BLEU scores, which measure natural language properties in translational models, or ROUGE scores, which measure overlap with human-written text. Unfortunately, these metrics do not necessarily capture how machine-generated code summaries actually affect human comprehension or developer productivity. We conducted a human study involving both university students and professional developers (n = 45). Participants reviewed Java methods and summaries and answered established program comprehension questions. In addition, participants completed coding tasks given summaries as specifications. Critically, the experiment controlled the source of the summaries: for a given method, some participants were shown human-written text and some were shown machine-generated text. We found that participants performed significantly better (p = 0.029) using human-written summaries versus machine-generated summaries. However, we found no evidence to support that participants perceive human- and machine-generated summaries to have different qualities. In addition, participants' performance showed no correlation with the BLEU and ROUGE scores often used to assess the quality of machine-generated summaries. These results suggest a need for revised metrics to assess and guide automatic summarization techniques.