Docable: Evaluating the Executability of Software Tutorials
Docable: Evaluating the Executability of Software Tutorials
复制标题
Docable：评估软件教程的可执行性
DOI：
10.1145/3368089.3409706
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Parnin, Chris
中科院分区：
文献类型：
--
作者：
Mirhosseini, Samim;Parnin, Chris
典型的软件教程包括安装开发人员工具、编辑文件和代码以及运行命令的分步说明。当这些软件教程无法执行时，无论是由于缺少说明，模糊的步骤，还是简单的命令，它们的价值都会降低。不可执行的教程在几个方面影响开发人员，包括令人沮丧的学习体验，并限制开发人员工具的可用性。为了了解软件教程在多大程度上是可执行的-以及为什么它们可能失败-我们对600多个教程进行了实证研究，其中包括近15，000个代码块。我们发现一个天真的执行策略实现了只有26%的整体可执行率。即使是一个基于人工注释的执行策略-虽然可执行性加倍-仍然不能产生可以成功执行所有步骤的教程。我们确定了几个常见的可执行性障碍，从潜在的无害的原因，如需要人工响应的交互式提示，到潜在的错误，如缺少步骤和无法访问的资源。我们与技术文档中的主要利益相关者验证了我们的研究结果，并讨论了改进软件教程的可能策略，例如为教程学习者提供可访问的替代方案，并投资于自动化教程测试以确保软件教程的持续质量。
The typical software tutorial includes step-by-step instructions for installing developer tools, editing files and code, and running commands. When these software tutorials are not executable, either due to missing instructions, ambiguous steps, or simply broken commands, their value is diminished. Non-executable tutorials impact developers in several ways, including frustrating learning experiences, and limiting usability of developer tools.To understand to what extent software tutorials are executable---and why they may fail---we conduct an empirical study on over 600 tutorials, including nearly 15,000 code blocks. We find a naive execution strategy achieves an overall executability rate of only 26%. Even a human-annotation-based execution strategy---while doubling executability---still yields no tutorial that can successfully execute all steps. We identify several common executability barriers, ranging from potentially innocuous causes, such as interactive prompts requiring human responses, to insidious errors, such as missing steps and inaccessible resources. We validate our findings with major stakeholders in technical documentation and discuss possible strategies for improving software tutorials, such as providing accessible alternatives for tutorial takers, and investing in automated tutorial testing to ensure continuous quality of software tutorials.