EAGER: Can Data Mining and Crowd Sourcing Revolutionize the Study of Scientific Peer Review? Generating a National, Open Depository of Grant Review Outcomes from Federal Agencies

EAGER：数据挖掘和众包能否彻底改变科学同行评审研究？

• 批准号: 1747445
• 负责人: Anna Kaatz
• 金额: $ 14.98万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2018-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1747445&HistoricalAwards=false
• 关键词: EAGER; Data; Mining; Crowd; Sourcing

One major barrier to research on peer review is the inaccessibility of data. Federal agencies are required to protect the identities of reviewers and applicants, which makes sharing data challenging, and prevents agencies from capitalizing on the opportunity to improve their systems with the help of the scientific community. This project addresses this problem with a novel automated program that collects peer review outcomes, such as grant application critiques and scores, directly from applicants, and de-identifies and stores them in a large repository that the scientific community can access. More intensive study and refinement of federal agencies' review processes will help to inform federal policies to ensure that U.S. tax dollars are allocated in ways that yield the broadest benefit. Conducting this work will also help broaden participation in science because it has high potential to illuminate reasons why individuals from historically underrepresented groups face disadvantage in peer review processes.Beginning with the National Institutes of Health (NIH), the largest federal funder of biomedical research, this project's automated data collection program uses web crawlers to identify Principal Investigators (PIs) and their email addresses from NIH's public access database, RePORTER; the program then sends PIs email invitations to donate their review materials (i.e., critiques and scores), which are de-identified and placed in a protected database (with the option to "opt-out" for all PIs). The program asks participating PIs to provide demographic information, and merges demographic and de-identified review outcome information into datasets that can be used for data and text analysis. This automated program can be adapted to collect review outcomes from unfunded applications, as well as from other federal funding agencies, by targeting broader email lists.

同行评议研究的一个主要障碍是无法获得数据。联邦机构被要求保护审查者和申请者的身份，这使得共享数据具有挑战性，并阻止机构利用机会在科学界的帮助下改进其系统。该项目通过一种新颖的自动化程序来解决这个问题，该程序直接从申请人那里收集同行评审结果，例如拨款申请批评和分数，并将其去识别并存储在科学界可以访问的大型存储库中。对联邦机构的审查程序进行更深入的研究和改进，将有助于为联邦政策提供信息，以确保美国税收的分配方式产生最广泛的利益。开展这项工作还将有助于扩大对科学的参与，因为它很有可能阐明为什么历史上代表性不足的群体的个人在同行评审过程中处于不利地位。从美国国立卫生研究院（NIH）开始，它是生物医学研究的最大联邦资助者，该项目的自动数据收集程序使用网络爬虫从NIH的公共访问数据库中识别主要研究者（PI）及其电子邮件地址，记者;该程序然后向PI发送电子邮件邀请以捐赠他们的评论材料（即，评论和分数），这些被去识别并放置在受保护的数据库中（对于所有PI，可以选择“退出”）。该计划要求参与的PI提供人口统计信息，并将人口统计和去识别的审查结果信息合并到可用于数据和文本分析的数据集中。这一自动化程序可以通过针对更广泛的电子邮件列表，从未资助的申请以及其他联邦资助机构收集审查结果。