PEER: A computerized platform for authoring structured peer reviews

在这个项目中，我们提出了一种新的方法和工具，使用数字注释和自然语言处理的最新发展，从根本上改变和改善科学手稿的同行评审。尽管同行审评是学术界质量保证体系的核心，但同行审评是一种相当非正式的做法，因领域、研究界和审评人员的经验水平而有很大差异。同行审查缺乏质量保证和培训，导致评价不一致，特别是在审查人才库往往包括初级和跨学科研究人员的新兴领域。这导致人们认为同行审查过程具有随意性，危及质量控制，并导致出版延迟和虚假结果的传播。近年来，科学出版和评价的开放趋势--预印本服务器、开放获取期刊和公共讨论平台的日益普及--使得对高质量同行评议的需求变得更加明显。虽然数字化大大加快了作者、审稿人和编辑之间的沟通，但同行评议本身在过去几十年里并没有太大的发展。为了使同行评审适应现代研究和科学出版的步伐，我们将同行评审，话语理论和基于注释的合作领域的现有最佳实践结合在一种新颖的同行评审方法中-结构化同行评审-并开发专用的写作辅助工具。该工具以阅读科学手稿时进行的非正式笔记为基础，指导审评员根据他们所作的注释和目标地点编辑提供的审评图表编写全面、简明的审评报告。该工具是高度可配置的，可用于为实际提交创建结构化的同行评审报告，作为研究培训的一部分，并作为一个平台，允许试验审查图式。生成的结构化报告可以按原样提交，也可以转换为传统的论文式评论草稿。为了使手稿评估更有效，我们引入了使用自然语言处理的辅助模型，以帮助用户执行常规的审查操作，而不会影响他们的评估。我们的辅助模型自动提示评论所属的稿件方面，帮助将相似的评论分组在一起，使评论报告更加紧凑，并允许将来自多个评论者的结构化报告合并为一个元报告，以支持编辑最终的接受决定。该项目为科学手稿的机器辅助评估铺平了道路，旨在促进元科学，数字注释和自然语言处理社区之间的合作。

In this project we propose a novel approach and a tool that use the latest developments in digital annotation and natural language processing to fundamentally change and improve the peer reviewing of scientific manuscripts. Despite being at the very core of the quality assurance system in academia, peer reviewing is a rather informal practice and varies greatly depending on the field, research community and the experience level of the reviewers. The lack of quality assurance and training in peer review results in inconsistent evaluation, especially in emerging fields where the reviewing pool often includes junior and cross-disciplinary researchers. This leads to perceived randomness of the peer reviewing process, jeopardizing quality control and resulting in publication delays and dissemination of spurious results. The recent trend for openness in scientific publishing and evaluation – manifested by the growing popularity of preprint servers, open access journals and public discussion platforms – makes the need for high-quality peer reviewing even more pronounced.While digitization has significantly sped up the communication between authors, reviewers and editors, peer reviewing per se has not seen much development in the past decades. To adapt peer reviewing to the pace of modern research and scientific publishing, we combine the existing best practices in the areas of peer reviewing, discourse theory and annotation-based collaboration in a novel peer reviewing approach - structured peer review - and develop a dedicated writing assistance tool. The tool builds upon the informal note-taking that accompanies reading of the scientific manuscripts, and guides the reviewers towards authoring comprehensive and concise review reports based on the annotations they make and the reviewing schemata provided by the editors of the target venue. The tool is highly configurable and can be used to create structured peer review reports for actual submissions, as part of research training, and as a platform that allows experimenting with reviewing schemata. The resulting structured reports can be submitted as is or transformed into drafts of a traditional essay-like review.To make the manuscript assessment more efficient, we introduce assistance models that use natural language processing to help users perform routine reviewing operations without biasing their evaluation. Our assistance models automatically suggest the aspect of the manuscript a commentary belongs to, help grouping similar commentaries together to make the review reports more compact, and allow merging structured reports from several reviewers into a single meta-report to support the editors in final acceptance decisions. The project paves the way towards machine-assisted evaluation of scientific manuscripts, and aims to foster the collaboration between meta-science, digital annotation and natural language processing communities.