On-the-fly Improving Performance of Deep Code Models via Input Denoising
On-the-fly Improving Performance of Deep Code Models via Input Denoising
复制标题
DOI：
10.1109/ase56229.2023.00166
复制
发表时间：
2023-08
期刊：
影响因子：
--
通讯作者：
Zhao Tian;Junjie Chen;Xiangyu Zhang
中科院分区：
文献类型：
--
作者：
Zhao Tian;Junjie Chen;Xiangyu Zhang
深度学习已被广泛采用，通过基于大量代码片段构建深度代码模型来处理各种基于代码的任务。虽然这些深度代码模型取得了巨大的成功，但即使是最先进的模型也会受到输入中存在的噪声的影响，从而导致错误的预测。虽然可以通过重新训练/微调来增强模型，但这不是一劳永逸的方法，并且会产生显著的开销。特别是，这些技术不能即时提高（部署的）模型的性能。目前在其他领域（如图像处理）也有一些输入去噪的技术，但由于代码输入是离散的，必须严格遵守复杂的语法和语义约束，其他领域的输入去噪技术几乎不适用。在这项工作中，我们提出了第一个输入去噪技术（即，CodeDenoise）用于深度代码模型。其核心思想是定位（可能）错误预测输入中的噪声标识符，并通过清洗定位的标识符来消除这些输入。它不需要重新训练或重建模型，而只需要在运行中清理输入以提高性能。我们在18个深度代码模型上的实验（即，三个预先训练的模型，六个基于代码的数据集）证明了CodeDenoise的有效性和效率。例如，平均而言，CodeDenoise成功地对21.91%的错误预测输入进行了去噪，并在每个输入平均花费0.48秒的时间内将所有受试者的模型准确性提高了2.04%，大大优于广泛使用的微调策略。
Deep learning has been widely adopted to tackle various code-based tasks by building deep code models based on a large amount of code snippets. While these deep code models have achieved great success, even state-of-the-art models suffer from noise present in inputs leading to erroneous predictions. While it is possible to enhance models through retraining/fine-tuning, this is not a once-and-for-all approach and incurs significant overhead. In particular, these techniques cannot on-the-fly improve performance of (deployed) models. There are currently some techniques for input denoising in other domains (such as image processing), but since code input is discrete and must strictly abide by complex syntactic and semantic constraints, input denoising techniques in other fields are almost not applicable. In this work, we propose the first input denoising technique (i.e., CodeDenoise) for deep code models. Its key idea is to localize noisy identifiers in (likely) mispredicted inputs, and denoise such inputs by cleansing the located identifiers. It does not need to retrain or reconstruct the model, but only needs to cleanse inputs on-the-fly to improve performance. Our experiments on 18 deep code models (i.e., three pre-trained models with six code-based datasets) demonstrate the effectiveness and efficiency of CodeDenoise. For example, on average, CodeDenoise successfully denoises 21.91% of mispredicted inputs and improves the original models by 2.04% in terms of the model accuracy across all the subjects in an average of 0.48 second spent on each input, substantially outperforming the widely-used fine-tuning strategy.