IGF::OT::IGF : An Innovative Approach for Peer-Review. Period of Performance: September 15, 2017 - March 14, 2018. N43DA-17-1214. BASE AWARD.

IGF::OT::IGF：同行评审的创新方法。

• 批准号: 9582275
• 负责人: YINFENG ZHANG
• 金额: $ 22.5万
• 依托单位: PEVALS, LLC
• 依托单位国家: 美国
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-15 至 2018-03-14
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9582275
• 关键词: Advocacy; Agreement; Algorithms; Award; Bioinformatics; Biological Sciences; Contractor; Decision Making; Disease; Educational workshop; Effectiveness; Electronics; Engineering; Evaluation; Florida; Foundations; Future; Goals; Individual; Institutes; Legal patent; Methodology; Methods; Modification; Online Systems; Paper; Participant; Peer Review; Performance; Phase; Pilot Projects; Policy Maker; Process; Reporting; Research; Science; Small Business Innovation Research Grant; Solid; Students; Testing; Universities; Work; Workload; base; design; essays; graduate student; improved; indexing; innovation; instructor; novel; novel strategies; programs; simulation; symposium

Under this Small Business Innovation Research (SBIR) Phase I project, Research Topic 162, the contractor will investigate the effectiveness of new peer-review methods, Concordance Index based Global Ranking (CIGR) and Modified Borda Count (MCB), to enhance professional evaluation and decision making in life sciences management and administration. The intended result is a novel set of metrics that can be used by NGOs/disease foundations, advocacy groups, research funders, policy makers, and by academic institutional bodies. This SBIR phase I project has the following objectives: 1. Simulation studies to investigate the effectiveness of the method used in the NSF pilot study and the contractor's new method, CIGR. The contractor will perform simulation studies to investigate the effectiveness of the new review method in terms of how it can recover the true ranking of proposals or papers. The contractor will also study which parameter settings are optimal for the new review method. 2. Testing the new approach in real evaluation settings The contractor has implemented the method used in the NSF’s pilot study as an online platform at http://pevals.com. The contractor will implement our new ranking method, CIGR, in the phase I project and test the new peer-review approach in graduate student course project/essay evaluation at Florida State University. In these tests, the true rankings and scores of the project reports or essays will be obtained from the course instructor and used as the true rankings. After collecting the ranking from individuals, we will use both CIGR and the modified Borda Count (MBC) score, proposed originally for the new peer-review method and also used in the NSF’s pilot study, to obtain the final rankings and compare the accuracy of the two different ranking methods based on their agreement with the true rankings. This test is unique in the sense that it has the true rankings obtained from the course instructors, which will help us to evaluate the effectiveness of different ranking algorithms. This test of using graduate student course projects will establish solid basis for our future tests on scientific papers and proposals. The contractor will also test our method for a conference paper evaluation for a workshop in IEEE BIBM 2017 conference.

在这个小企业创新研究（SBIR）第一阶段项目，研究主题162，承包商将调查新的同行评审方法的有效性，一致性指数为基础的全球排名（CIGR）和修正的Borda计数（MCB），以加强生命科学管理和行政的专业评估和决策。预期的结果是一套新的指标，可用于非政府组织/疾病基金会，倡导团体，研究资助者，政策制定者和学术机构机构。SBIR第一阶段项目有以下目标： 1.模拟研究，以调查NSF试点研究中使用的方法和承包商的新方法CIGR的有效性。 承包者将进行模拟研究，调查新的审查方法在如何恢复提案或文件的真实排名方面的有效性。承包者还将研究哪些参数设置最适合新的审查方法。 2.在真实的评估环境中测试新方法 承包商已将NSF试点研究中使用的方法作为在线平台（http://pevals.com）实施。承包商将在第一阶段项目中实施我们的新排名方法CIGR，并在佛罗里达州立大学的研究生课程项目/论文评估中测试新的同行评审方法。在这些测试中，项目报告或论文的真实排名和分数将从课程讲师处获得，并用作真实排名。在收集个人排名后，我们将使用CIGR和修改后的Borda Count（MBC）得分，最初为新的同行评审方法提出，也用于NSF的试点研究，以获得最终排名，并根据与真实排名的一致性比较两种不同排名方法的准确性。这个测试的独特之处在于它拥有从课程讲师那里获得的真实排名，这将有助于我们评估不同排名算法的有效性。本次研究生课程设计测试为今后的科研论文和科研提案测试奠定了坚实的基础。 承包商还将测试我们的方法，用于IEEE BIBM 2017会议研讨会的会议论文评估。