CAREER: Using network analysis to assess confidence in research synthesis

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。最佳科学是为保护，能源，医疗保健和可持续发展等领域的政策提供信息的重要因素。确定最好的科学需要综合多个科学结果，以衡量科学共识的水平和研究的可靠性。然而，在一些与政策相关的主题上，不同的综合得出了不一致的结论。在综合证据方面的这种不一致性浪费了资金，产生了误导性的结果，并可能导致影响许多人的错误决定。通过研究，教育和推广，这个职业项目旨在开发和测试一个新的工具和工作流程框架，揭示专家文献中的潜在偏见来源。该框架将使利益相关者能够快速了解哪些个人、机构和资助者为证据的创建做出了贡献。它将评估产生偏见风险的其他因素以及专家群体对所提供证据的信心程度。研究成果可以促进在广泛领域中以数据为驱动的决策。例子包括能源和环境科学以及健康科学的主题，如各种形式的食品生产的碳足迹，群体免疫力和疫苗有效性。该项目还将通过从服务不足的人群中雇用学生助理，以及通过开发两个与政策相关的STEM大学课程和一个中学职业视频来吸引代表性不足的学生，帮助科学劳动力多样化。该项目探讨如何改善大规模研究信心的评估。它将使证据寻求者能够快速了解文献中的共识水平，沿着可能影响研究可靠性的风险因素，为稳健性和可重复性提供关键资源。这个框架可以应用于任何书目，包括同行评审的手稿，已发表的文章和数据库搜索结果。项目输出将有利于识别文献综述中的风险，例如申办者偏倚或避免引用矛盾证据，这将有助于减少错误信息的传播。这个项目是可能的网络科学和文本挖掘方法的最新进展，以及在适当的数据科学许可证下的摘要，隶属关系，引文和资金数据的可用性。这项工作是新颖的，汇集了以前没有结合互补的方法：论证理论和争议的研究;方法综合证据;和文献计量学和科学计量学的方法来结构性地看待一个领域。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The best available science is an important factor that informs policy in areas such as conservation, energy, healthcare, and sustainable. Determining the best available science requires synthesizing multiple scientific results to gauge both the level of scientific consensus and the reliability of the research. However, on some policy-relevant topics, different syntheses come to incompatible conclusions. Such inconsistency in the synthesis of evidence wastes money, generates misleading results, and can lead to poor decisions impacting large numbers of people. Through research, education, and outreach, this CAREER project aims to develop and test a novel framework of tools and workflows that will reveal potential sources of bias in expert literature. The framework will enable stakeholders to quickly understand which individuals, institutions and funders contributed to the creation of the evidence. It will assess other factors that create risk of bias as well as the degree of confidence an expert community has in the evidence presented. Research outcomes could facilitate data-driven decision-making in a broad range of areas. Examples include topics in energy and environmental sciences and health sciences, like the carbon footprint of various forms of food production, herd immunity, and vaccine effectiveness. This project will also help diversify the science workforce by employing student assistants from underserved populations and by developing two policy-relevant STEM university courses and a middle school career video to attract underrepresented students.This project explores how to improve the assessment of confidence in research at scale. It will enable evidence-seekers to quickly understand the level of consensus within a body of literature, along with risk factors that might impact reliability of research, providing a key resource for robustness and reproducibility. This framework can be applied to any bibliography, including manuscripts under peer review, published articles, and database search results. Project outputs will be beneficial for identifying risks in literature reviews, such as sponsor bias or the avoidance of citation of contradictory evidence, which will help reduce the spread of misinformation. This project is made possible by recent advances in network science and text mining methods, as well as the availability of abstracts, affiliation, citations, and funding data under suitable licenses for data science. The work is novel in bringing together complementary approaches that have not previously been combined: argumentation theory and the study of controversies; approaches for synthesizing evidence; and bibliometric and scientometric approaches for looking structurally at a field.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.