UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior
UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior
复制标题
DOI：
10.1145/3411764.3445538
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
E. Schoop;Forrest Huang;Bjoern Hartmann
中科院分区：
文献类型：
--
作者：
E. Schoop;Forrest Huang;Bjoern Hartmann
训练深度神经网络可能会生成非描述性错误消息或产生不寻常的输出，而根本没有任何显式错误。虽然专家依靠隐性知识来应用调试策略，但非专家缺乏解释模型输出和纠正深度学习（DL）程序所需的经验。在这项工作中，我们确定了专家使用的DL调试技巧和策略，在这项工作中，我们对新手在编写ML代码时遇到的错误类型进行了分类，并将它们映射到工具可以提供帮助的机会上。我们用它们来指导变音的设计。Umlaut根据这些语法检查DL程序结构和模型行为;向用户提供人类可读的错误消息;并注释错误的模型输出以促进错误纠正。Umlaut将代码、模型输出和教程驱动的错误消息链接到一个界面中。我们在一项有15名参与者的研究中评估了Umlaut，以确定其在帮助开发人员发现和修复DL程序中错误方面的有效性。与基线条件相比，使用Umlaut的参与者发现并修复了更多的错误，并且能够修复更多的错误。
Training deep neural networks can generate non-descriptive error messages or produce unusual output without any explicit errors at all. While experts rely on tacit knowledge to apply debugging strategies, non-experts lack the experience required to interpret model output and correct Deep Learning (DL) programs. In this work, we identify DL debugging heuristics and strategies used by experts, andIn this work, we categorize the types of errors novices run into when writing ML code, and map them onto opportunities where tools could help. We use them to guide the design of Umlaut. Umlaut checks DL program structure and model behavior against these heuristics; provides human-readable error messages to users; and annotates erroneous model output to facilitate error correction. Umlaut links code, model output, and tutorial-driven error messages in a single interface. We evaluated Umlaut in a study with 15 participants to determine its effectiveness in helping developers find and fix errors in their DL programs. Participants using Umlaut found and fixed significantly more bugs and were able to implement fixes for more bugs compared to a baseline condition.