Empirical software engineering experts on the use of students and professionals in experiments
Empirical software engineering experts on the use of students and professionals in experiments
复制标题
DOI：
10.1007/s10664-017-9523-3
复制
发表时间：
2018-02-01
影响因子：
4.1
通讯作者：
Oivo, Markku
中科院分区：
文献类型：
--
作者：
Falessi, Davide;Juristo, Natalia;Oivo, Markku
控制实验是产生和验证理论的重要经验方法。许多软件工程实验是与学生一起进行的。人们经常声称，使用学生作为实验参与者是以低外部效度为代价的，而使用专业人士则不是。[目的]对软件工程实验的外部效度进行深入研究。我们的目标是深入了解在实验中使用学生和专业人士的利弊。[方法]采用非常规的焦点小组调查方法和跟踪调查。首先，在2014年ISERN的一次会议上，包括七位作者在内的65位实证研究人员以开放的心态辩论和讨论了在实验中使用学生的问题。随后，我们再次讨论了这个问题，并听取了专家的意见，以促进讨论。然后，我们得出了14个声明，并要求ISERN与会者（不包括作者）提供他们对这些声明的同意程度。最后，我们分析了研究者的观点，并利用研究结果进一步讨论了这些陈述。[结果]调查结果显示，总体上，受访者对专业人士的缺点不认同。相反，我们坚信，没有任何群体（学生，专业人士或其他人）可以被认为在绝对意义上比另一个群体更好。[结论]以学生为参与者仍然是一种有效的简化现实需要在实验室环境中。它是推进软件工程理论和技术的有效方法，但与研究设置的任何其他方面一样，在设计，执行，解释和报告实验期间应仔细考虑。关键是要理解实验中的参与者代表了哪一部分开发人员。因此，提出了描述实验参与者的建议。
[Context] Controlled experiments are an important empirical method to generate and validate theories. Many software engineering experiments are conducted with students. It is often claimed that the use of students as participants in experiments comes at the cost of low external validity while using professionals does not. [Objective] We believe a deeper understanding is needed on the external validity of software engineering experiments conducted with students or with professionals. We aim to gain insight about the pros and cons of using students and professionals in experiments. [Method] We performed an unconventional, focus group approach and a follow-up survey. First, during a session at ISERN 2014, 65 empirical researchers, including the seven authors, argued and discussed the use of students in experiments with an open mind. Afterwards, we revisited the topic and elicited experts' opinions to foster discussions. Then we derived 14 statements and asked the ISERN attendees excluding the authors, to provide their level of agreement with the statements. Finally, we analyzed the researchers' opinions and used the findings to further discuss the statements. [Results] Our survey results showed that, in general, the respondents disagreed with us about the drawbacks of professionals. We, on the contrary, strongly believe that no population (students, professionals, or others) can be deemed better than another in absolute terms. [Conclusion] Using students as participants remains a valid simplification of reality needed in laboratory contexts. It is an effective way to advance software engineering theories and technologies but, like any other aspect of study settings, should be carefully considered during the design, execution, interpretation, and reporting of an experiment. The key is to understand which developer population portion is being represented by the participants in an experiment. Thus, a proposal for describing experimental participants is put forward.