PEER: A computerized platform for authoring structured peer reviews

PEER：用于撰写结构化同行评审的计算机化平台

• 批准号: 440185223
• 负责人: Professorin Dr. Iryna Gurevych
• 金额: --
• 依托单位: Fachgebiet Ubiquitäre Wissensverarbeitung
• 依托单位国家: 德国
• 项目类别: Research data and software (Scientific Library Services and Information Systems)
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/440185223?language=en
• 关键词: PEER; computerized; platform; authoring; structured

In this project we propose a novel approach and a tool that use the latest developments in digital annotation and natural language processing to fundamentally change and improve the peer reviewing of scientific manuscripts. Despite being at the very core of the quality assurance system in academia, peer reviewing is a rather informal practice and varies greatly depending on the field, research community and the experience level of the reviewers. The lack of quality assurance and training in peer review results in inconsistent evaluation, especially in emerging fields where the reviewing pool often includes junior and cross-disciplinary researchers. This leads to perceived randomness of the peer reviewing process, jeopardizing quality control and resulting in publication delays and dissemination of spurious results. The recent trend for openness in scientific publishing and evaluation – manifested by the growing popularity of preprint servers, open access journals and public discussion platforms – makes the need for high-quality peer reviewing even more pronounced.While digitization has significantly sped up the communication between authors, reviewers and editors, peer reviewing per se has not seen much development in the past decades. To adapt peer reviewing to the pace of modern research and scientific publishing, we combine the existing best practices in the areas of peer reviewing, discourse theory and annotation-based collaboration in a novel peer reviewing approach - structured peer review - and develop a dedicated writing assistance tool. The tool builds upon the informal note-taking that accompanies reading of the scientific manuscripts, and guides the reviewers towards authoring comprehensive and concise review reports based on the annotations they make and the reviewing schemata provided by the editors of the target venue. The tool is highly configurable and can be used to create structured peer review reports for actual submissions, as part of research training, and as a platform that allows experimenting with reviewing schemata. The resulting structured reports can be submitted as is or transformed into drafts of a traditional essay-like review.To make the manuscript assessment more efficient, we introduce assistance models that use natural language processing to help users perform routine reviewing operations without biasing their evaluation. Our assistance models automatically suggest the aspect of the manuscript a commentary belongs to, help grouping similar commentaries together to make the review reports more compact, and allow merging structured reports from several reviewers into a single meta-report to support the editors in final acceptance decisions. The project paves the way towards machine-assisted evaluation of scientific manuscripts, and aims to foster the collaboration between meta-science, digital annotation and natural language processing communities.

在这个项目中，我们提出了一种新的方法和工具，利用数字注释和自然语言处理的最新发展来从根本上改变和改进科学手稿的同行评审。尽管同行评议在学术界是质量保证体系的核心，但它是一种相当非正式的做法，而且根据领域、研究社区和评审员的经验水平有很大的不同。缺乏同行审查的质量保证和培训导致评价不一致，特别是在新兴领域，审查对象往往包括初级和跨学科的研究人员。这导致同行审查过程的随机性，危及质量控制，并导致出版延迟和虚假结果的传播。最近科学出版和评估的开放趋势--体现在预印本服务器、开放获取期刊和公共讨论平台的日益流行--使得对高质量同行评议的需求更加明显。虽然数字化显著加快了作者、审稿人和编辑之间的交流，但同行评议本身在过去几十年中并没有看到太大的发展。为了使同行评议适应现代研究和科学出版的步伐，我们结合了同行评议、话语理论和基于注释的协作领域的现有最佳实践，形成了一种新的同行评议方法--结构化同行评议--并开发了专门的写作辅助工具。该工具建立在阅读科学手稿时附带的非正式笔记的基础上，并指导审查员根据他们所作的注释和目标地点编辑提供的审查方案编写全面而简洁的审查报告。该工具具有高度的可配置性，可用于为实际提交的报告创建结构化同行审查报告，作为研究培训的一部分，并作为允许试验审查模式的平台。为了提高稿件评估的效率，我们引入了使用自然语言处理的辅助模型，帮助用户执行常规的审稿操作，而不会对他们的评估产生偏见。我们的辅助模型自动建议评论所属的手稿方面，帮助将类似的评论分组在一起，使审查报告更加紧凑，并允许将来自多个评审者的结构化报告合并为单个元报告，以支持编辑做出最终接受决定。该项目为科学手稿的机器辅助评估铺平了道路，旨在促进元科学、数字注释和自然语言处理社区之间的合作。