PEER: A computerized platform for authoring structured peer reviews

在这个项目中，我们提出了一种新的方法和工具，它使用了数字注释和自然语言处理的最新发展，从根本上改变和改善了科学手稿的同行评审。尽管同行评议是学术界质量保证体系的核心，但它是一种相当非正式的做法，并且因领域、研究团体和审稿人的经验水平而有很大差异。缺乏同行评议的质量保证和培训导致评价不一致，特别是在评议人员通常包括初级和跨学科研究人员的新兴领域。这导致同行评议过程的随机性，危及质量控制，并导致出版延迟和虚假结果的传播。最近科学出版和评估的开放趋势——预印本服务器、开放获取期刊和公共讨论平台的日益普及——使得对高质量同行评议的需求更加明显。虽然数字化大大加快了作者、审稿人和编辑之间的沟通，但同行评议本身在过去几十年里并没有多大发展。为了使同行评议适应现代研究和科学出版的步伐，我们将同行评议、话语理论和基于注释的协作等领域的现有最佳实践结合在一起，形成了一种新的同行评议方法——结构化同行评议——并开发了一种专门的写作辅助工具。该工具建立在阅读科学手稿的非正式笔记的基础上，并指导审稿人根据他们所做的注释和目标地点的编辑提供的审稿图式编写全面而简洁的审稿报告。该工具是高度可配置的，可用于为实际提交创建结构化的同行评审报告，作为研究培训的一部分，并作为允许对评审模式进行实验的平台。生成的结构化报告可以按原样提交，也可以转换为传统的类似论文的审查草稿。为了提高稿件评估的效率，我们引入了使用自然语言处理的辅助模型，以帮助用户执行日常审查操作，而不会影响他们的评估。我们的辅助模型自动提示评论所属的手稿方面，帮助将类似的评论分组在一起，使审查报告更加紧凑，并允许将来自几个审稿人的结构化报告合并到一个元报告中，以支持编辑最终的接受决定。该项目为科学手稿的机器辅助评估铺平了道路，旨在促进元科学、数字注释和自然语言处理社区之间的合作。

In this project we propose a novel approach and a tool that use the latest developments in digital annotation and natural language processing to fundamentally change and improve the peer reviewing of scientific manuscripts. Despite being at the very core of the quality assurance system in academia, peer reviewing is a rather informal practice and varies greatly depending on the field, research community and the experience level of the reviewers. The lack of quality assurance and training in peer review results in inconsistent evaluation, especially in emerging fields where the reviewing pool often includes junior and cross-disciplinary researchers. This leads to perceived randomness of the peer reviewing process, jeopardizing quality control and resulting in publication delays and dissemination of spurious results. The recent trend for openness in scientific publishing and evaluation – manifested by the growing popularity of preprint servers, open access journals and public discussion platforms – makes the need for high-quality peer reviewing even more pronounced.While digitization has significantly sped up the communication between authors, reviewers and editors, peer reviewing per se has not seen much development in the past decades. To adapt peer reviewing to the pace of modern research and scientific publishing, we combine the existing best practices in the areas of peer reviewing, discourse theory and annotation-based collaboration in a novel peer reviewing approach - structured peer review - and develop a dedicated writing assistance tool. The tool builds upon the informal note-taking that accompanies reading of the scientific manuscripts, and guides the reviewers towards authoring comprehensive and concise review reports based on the annotations they make and the reviewing schemata provided by the editors of the target venue. The tool is highly configurable and can be used to create structured peer review reports for actual submissions, as part of research training, and as a platform that allows experimenting with reviewing schemata. The resulting structured reports can be submitted as is or transformed into drafts of a traditional essay-like review.To make the manuscript assessment more efficient, we introduce assistance models that use natural language processing to help users perform routine reviewing operations without biasing their evaluation. Our assistance models automatically suggest the aspect of the manuscript a commentary belongs to, help grouping similar commentaries together to make the review reports more compact, and allow merging structured reports from several reviewers into a single meta-report to support the editors in final acceptance decisions. The project paves the way towards machine-assisted evaluation of scientific manuscripts, and aims to foster the collaboration between meta-science, digital annotation and natural language processing communities.