EAGER: Can Data Mining and Crowd Sourcing Revolutionize the Study of Scientific Peer Review? Generating a National, Open Depository of Grant Review Outcomes from Federal Agencies

同行评议研究的一个主要障碍是数据难以获取。联邦机构被要求保护审稿人和申请人的身份，这使得数据共享具有挑战性，并阻止机构利用在科学界的帮助下改进其系统的机会。这个项目用一个新颖的自动化程序来解决这个问题，这个程序直接从申请人那里收集同行评审结果，比如拨款申请的评论和分数，并将它们去识别并存储在一个科学界可以访问的大型存储库中。对联邦机构的审查程序进行更深入的研究和改进，将有助于为联邦政策提供信息，以确保美国纳税人的钱以产生最广泛利益的方式分配。开展这项工作还将有助于扩大对科学的参与，因为它很有可能阐明为什么来自历史上代表性不足的群体的个人在同行评议过程中面临劣势的原因。从生物医学研究的最大联邦资助机构美国国立卫生研究院（NIH）开始，该项目的自动数据收集程序使用网络爬虫从NIH的公共访问数据库RePORTER中识别主要研究者（pi）及其电子邮件地址；然后，该计划向个人意见发送电子邮件，邀请他们捐赠他们的评论材料（即评论和分数），这些材料将被去识别并放置在一个受保护的数据库中（所有个人意见都有“选择退出”的选项）。该项目要求参与项目的pi提供人口统计信息，并将人口统计信息和去识别审查结果信息合并到可用于数据和文本分析的数据集中。这个自动化程序可以通过针对更广泛的电子邮件列表，从未资助的应用程序以及其他联邦资助机构收集审查结果。

One major barrier to research on peer review is the inaccessibility of data. Federal agencies are required to protect the identities of reviewers and applicants, which makes sharing data challenging, and prevents agencies from capitalizing on the opportunity to improve their systems with the help of the scientific community. This project addresses this problem with a novel automated program that collects peer review outcomes, such as grant application critiques and scores, directly from applicants, and de-identifies and stores them in a large repository that the scientific community can access. More intensive study and refinement of federal agencies' review processes will help to inform federal policies to ensure that U.S. tax dollars are allocated in ways that yield the broadest benefit. Conducting this work will also help broaden participation in science because it has high potential to illuminate reasons why individuals from historically underrepresented groups face disadvantage in peer review processes.Beginning with the National Institutes of Health (NIH), the largest federal funder of biomedical research, this project's automated data collection program uses web crawlers to identify Principal Investigators (PIs) and their email addresses from NIH's public access database, RePORTER; the program then sends PIs email invitations to donate their review materials (i.e., critiques and scores), which are de-identified and placed in a protected database (with the option to "opt-out" for all PIs). The program asks participating PIs to provide demographic information, and merges demographic and de-identified review outcome information into datasets that can be used for data and text analysis. This automated program can be adapted to collect review outcomes from unfunded applications, as well as from other federal funding agencies, by targeting broader email lists.