Empirical analyses of the factors affecting confirmation bias and the effects of confirmation bias on software developer/tester performance
Empirical analyses of the factors affecting confirmation bias and the effects of confirmation bias on software developer/tester performance
复制标题
影响确认偏差的因素以及确认偏差对软件开发人员/测试人员绩效的影响的实证分析
DOI：
--
复制
发表时间：
2010
期刊：
影响因子：
--
通讯作者：
A. Bener
中科院分区：
文献类型：
--
作者：
G. Çalikli;A. Bener
背景：在所有级别的软件测试中，目标应该是使代码失败。然而，由于被称为确认偏差的现象，软件开发人员和测试人员更有可能选择正面测试，而不是负面测试。确认偏差是指人们倾向于验证自己的假说，而不是反驳假说。在文献中，有关于确认偏差对软件开发和测试的可能影响的理论。由于倾向于阳性测试，大多数软件缺陷仍然没有被检测到，这反过来又导致软件缺陷密度的增加。
目的：在本研究中，我们分析了影响确认偏差的因素，以寻找规避确认偏差的方法。我们调查的因素是软件开发/测试的经验和可以通过教育获得的推理技能。此外，我们还分析了确认偏差对软件开发人员和测试人员绩效的影响。
方法：为了测量和量化软件开发人员/测试人员的确认偏向水平，我们根据认知心理学文献中的两个任务编制了纸笔测试和互动测试。这些测试是对欧洲一家大型电信公司的36名员工以及博加齐奇大学28名计算机工程专业的研究生进行的，结果总共产生了对象。
除了我们从认知心理学文献中继承的一些基本方法外，我们还使用我们提出的测量方法来评估这些测试的结果。
结果：无论有没有软件开发/测试经验，逻辑推理和策略假设检验等能力都是低确认偏倚水平下的区分因素。此外，对软件开发人员和测试人员的代码缺陷密度与确认偏差水平之间的关系的分析结果表明，确认偏差与代码的缺陷倾向之间存在直接关联。
结论：我们的研究结果表明，拥有强大的逻辑推理和假设检验技能是软件开发人员/测试人员在缺陷率方面表现的不同因素。我们建议，公司应该通过设计培训计划，专注于提高员工的逻辑推理和假设检验技能。作为未来的工作，我们计划将这项研究复制到其他软件开发公司。此外，除了产品和过程度量外，我们还将在软件缺陷预测中使用确认偏差度量。我们相信，确认偏差度量将改善我们十多年来一直在构建的基于学习的缺陷预测模型的预测性能。
Background: During all levels of software testing, the goal should be to fail the code. However, software developers and testers are more likely to choose positive tests rather than negative ones due to the phenomenon called confirmation bias. Confirmation bias is defined as the tendency of people to verify their hypotheses rather than refuting them. In the literature, there are theories about the possible effects of confirmation bias on software development and testing. Due to the tendency towards positive tests, most of the software defects remain undetected, which in turn leads to an increase in software defect density.
Aims: In this study, we analyze factors affecting confirmation bias in order to discover methods to circumvent confirmation bias. The factors, we investigate are experience in software development/testing and reasoning skills that can be gained through education. In addition, we analyze the effect of confirmation bias on software developer and tester performance.
Method: In order to measure and quantify confirmation bias levels of software developers/testers, we prepared pen-and-paper and interactive tests based on two tasks from cognitive psychology literature. These tests were conducted on the 36 employees of a large scale telecommunication company in Europe as well as 28 graduate computer engineering students of Bogazici University, resulting in a total of 64 subjects.
We evaluated the outcomes of these tests using the metrics we proposed in addition to some basic methods which we inherited from the cognitive psychology literature.
Results: Results showed that regardless of experience in software development/testing, abilities such as logical reasoning and strategic hypotheses testing are differentiating factors in low confirmation bias levels. Moreover, the results of the analysis to investigate the relationship between code defect density and confirmation bias levels of software developers and testers showed that there is a direct correlation between confirmation bias and defect proneness of the code.
Conclusions: Our findings show that having strong logical reasoning and hypothesis testing skills are differentiating factors in the software developer/tester performance in terms of defect rates. We recommend that companies should focus on improving logical reasoning and hypothesis testing skills of their employees by designing training programs. As future work, we plan to replicate this study in other software development companies. Moreover, we will use confirmation bias metrics in addition to product and process metrics in for software defect prediction. We believe that confirmation bias metrics would improve the prediction performance of learning based defect prediction models which we have been building over a decade.