Continuous quality control for research data to ensure reproducibility: an institutional approach (CONQUAIRE)

可重复性是科学进程的基石。虽然实验的复制可能非常困难，但对支持某一结论(例如，验证假设)的数据进行复制(计算)分析的能力应该是对已发表的每一项研究的最低要求。复制某项研究的分析结果的能力至少要求：i)原始或次要数据可用，ii)数据的句法结构良好且随时可用，iii)数据被适当地记录，iv)用于处理或分析数据的分析程序(例如脚本)可用，v)这些分析程序可在数据上运行以重现发表在论文中的实际结果。分析的重现性常常受到上述要求之一不能满足的事实的影响。该项目的目标是扩大比勒费尔德大学现有的管理数据和出版物的基础设施，建立一个支持研究人员满足上述要求的框架，从而使他们的工作在分析上具有可重复性。与目前在研究项目结束时公布数据和软件的做法不同，如果有的话，我们打算将数据托管转移到科学过程的最开始。借鉴计算机科学和持续集成的思想，我们打算实施一个持续质量控制框架，从一开始就鼓励研究人员公布他们的数据和分析程序，以便这些数据和分析程序可以很容易地重复使用和验证。因此，我们理解项目中数据质量的方式是准备好被重用和验证。为了实现这一目标，我们将与比勒费尔德大学精选的一组研究人员互动，他们承诺定义用例、提供需求、实施试点，并持续使用基础设施并提供定期反馈。研究人员来自心理学、体育科学、生物学、化学、认知语言学、计算语言学、机器人学以及经济学等多个学科。通过涉及一组不同的学科，我们的目标是确定对基础设施的共同需求，该基础设施支持数据质量作为一个连续的过程，并支持研究结果的共享和外部验证。除了扩大我们的基础设施，该项目预计将产生远远超出比勒费尔德大学的影响。通过分享我们的经验和确定的要求，我们希望让其他大学和政策制定者了解努力和投资回报之间的权衡，以及采取哪些政策来支持更高的研究透明度。

Reproducibility is a cornerstone of the scientific process. While the reproduction of an experiment can be extremely difficult, the ability to reproduce the (computational) analysis of the data that supported a certain conclusion (e.g. the validation of a hypothesis) should be a minimum requirement on every piece of published research. We call this type of reproducibility "analytical reproducibility".The ability to reproduce the analytic results of a certain piece of research requires, as a minimum, that: i) the primary or secondary data is available, ii) the data is syntactically well-formed and ready-to-use, iii) the data is appropriately documented, iv) the analysis procedures (e.g. scripts) that were used to process or analyze the data are available, and v) these analytic procedures can be run on the data to reproduce the actual result published in a paper. Analytical reproducibility is often hampered by the fact that one of the above requirements is not met. The goal of this project is to extend the infrastructure available at Bielefeld University for the management of data and publications by a framework that supports researchers in meeting the above mentioned requirements and thus to make their work analytically reproducible. Departing from current practices where data and software is published at the end of a research project, if at all, we intend to move the hosting of data to the very beginning of the scientific process. Borrowing ideas from computer science and from continuous integration, we intend to implement a continuous quality control framework that from early on encourages researchers to publish their data and analytic procedures so that these can easily be re-used and verified. The way we understand quality of data in the project is thus in the sense of readiness to be re-used and validated. Towards this goal, we will interact with a selected group of researchers at Bielefeld University that have committed themselves to define a use case, provide requirements, implement pilots, and continuously work with the infrastructure and provide regular feedback. The researchers come from disciplines as varied as psychology, sports sciences, biology, chemistry, cognitive linguistics, computational linguistics, robotics as well as economics. By involving a varied set of disciplines, our goal is to identify common requirements on an infrastructure that supports data quality as a continuous process, and supports sharing and external validation of research results. Besides extending our infrastructure, the project can be expected to have an impact way beyond Bielefeld University. By sharing our experiences and requirements identified, we hope to inform other universities and policy makers on the trade-off between effort and return-on-investment and which policies to adopt to support higher transparency in research.