Squib: Reproducibility in Computational Linguistics: Are We Willing to Share?
Squib: Reproducibility in Computational Linguistics: Are We Willing to Share?
复制标题
哑炮：计算语言学的可重复性：我们愿意分享吗？
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Gertjan van Noord
中科院分区：
文献类型：
--
作者：
Martijn Wieling;J. Rawee;Gertjan van Noord
本研究的重点是计算语言学中再现性的一个基本前提：作者愿意分享相关的源代码和数据。在Ted Pedersen在计算语言学中有影响力的“最后的话”贡献十年后，我们调查了计算语言学研究人员愿意并能够在多大程度上共享他们的数据和代码。我们调查了2011年和2016年ACL年会上提交的所有395篇论文全文，并确定是否提供了数据和代码的链接。如果没有提供工作链接，则要求作者提供此信息。虽然数据经常可用，但代码共享的频率较低。当论文中没有提供代码或数据的工作链接时，作者提供了大约三分之一的代码。对于十篇论文的选择，我们试图使用提供的数据和代码再现结果。我们能够复制大约六篇论文的结果。只有一篇论文我们得到了完全相同的结果。我们的研究结果表明，尽管2016年与2011年相比，情况似乎有所改善，但计算语言学中的语言主义在很大程度上仍然是一个信仰问题。尽管如此，我们对未来还是有些乐观的。确保可重复性不仅对整个领域很重要，而且对个体研究人员来说也是值得的：与源代码有工作链接的研究的引用数中位数更高。
This study focuses on an essential precondition for reproducibility in computational linguistics: the willingness of authors to share relevant source code and data. Ten years after Ted Pedersen’s influential “Last Words” contribution in Computational Linguistics, we investigate to what extent researchers in computational linguistics are willing and able to share their data and code. We surveyed all 395 full papers presented at the 2011 and 2016 ACL Annual Meetings, and identified whether links to data and code were provided. If working links were not provided, authors were requested to provide this information. Although data were often available, code was shared less often. When working links to code or data were not provided in the paper, authors provided the code in about one third of cases. For a selection of ten papers, we attempted to reproduce the results using the provided data and code. We were able to reproduce the results approximately for six papers. For only a single paper did we obtain the exact same results. Our findings show that even though the situation appears to have improved comparing 2016 to 2011, empiricism in computational linguistics still largely remains a matter of faith. Nevertheless, we are somewhat optimistic about the future. Ensuring reproducibility is not only important for the field as a whole, but also seems worthwhile for individual researchers: The median citation count for studies with working links to the source code is higher.