CAREER: Using network analysis to assess confidence in research synthesis

该奖项全部或部分由《2021年美国救援计划法案》（公法117-2）资助。可获得的最佳科学是为保护、能源、卫生保健和可持续发展等领域的政策提供信息的一个重要因素。确定现有的最佳科学需要综合多种科学结果，以衡量科学共识的水平和研究的可靠性。然而，在一些与政策相关的主题上，不同的综合得出了不相容的结论。在综合证据方面的这种不一致浪费了资金，产生了误导性的结果，并可能导致影响大量人的错误决定。通过研究、教育和推广，这个CAREER项目旨在开发和测试一个新的工具和工作流程框架，以揭示专家文献中潜在的偏见来源。该框架将使利益攸关方能够迅速了解哪些个人、机构和资助者为证据的创造做出了贡献。它将评估产生偏见风险的其他因素，以及专家团体对所提供证据的信心程度。研究成果可以在广泛的领域促进数据驱动的决策。例子包括能源和环境科学以及健康科学的主题，如各种形式的粮食生产的碳足迹、群体免疫和疫苗有效性。该项目还将通过从服务不足的人群中雇用学生助理，开发两门与政策相关的STEM大学课程和一段中学职业视频来吸引代表性不足的学生，从而帮助实现科学劳动力的多样化。本项目探讨如何在规模研究中提高信心评估。它将使证据寻求者能够快速了解文献中的共识水平，以及可能影响研究可靠性的风险因素，为稳健性和可重复性提供关键资源。这个框架可以应用于任何参考书目，包括同行评审的手稿、发表的文章和数据库搜索结果。项目产出将有助于识别文献综述中的风险，如赞助者偏见或避免引用相互矛盾的证据，这将有助于减少错误信息的传播。由于网络科学和文本挖掘方法的最新进展，以及在适当的数据科学许可下的摘要、隶属关系、引用和资助数据的可用性，该项目成为可能。这项工作的新颖之处在于将以前从未结合过的互补方法结合在一起：论证理论和争议研究；证据合成方法；以及文献计量学和科学计量学方法来从结构上观察一个领域。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The best available science is an important factor that informs policy in areas such as conservation, energy, healthcare, and sustainable. Determining the best available science requires synthesizing multiple scientific results to gauge both the level of scientific consensus and the reliability of the research. However, on some policy-relevant topics, different syntheses come to incompatible conclusions. Such inconsistency in the synthesis of evidence wastes money, generates misleading results, and can lead to poor decisions impacting large numbers of people. Through research, education, and outreach, this CAREER project aims to develop and test a novel framework of tools and workflows that will reveal potential sources of bias in expert literature. The framework will enable stakeholders to quickly understand which individuals, institutions and funders contributed to the creation of the evidence. It will assess other factors that create risk of bias as well as the degree of confidence an expert community has in the evidence presented. Research outcomes could facilitate data-driven decision-making in a broad range of areas. Examples include topics in energy and environmental sciences and health sciences, like the carbon footprint of various forms of food production, herd immunity, and vaccine effectiveness. This project will also help diversify the science workforce by employing student assistants from underserved populations and by developing two policy-relevant STEM university courses and a middle school career video to attract underrepresented students.This project explores how to improve the assessment of confidence in research at scale. It will enable evidence-seekers to quickly understand the level of consensus within a body of literature, along with risk factors that might impact reliability of research, providing a key resource for robustness and reproducibility. This framework can be applied to any bibliography, including manuscripts under peer review, published articles, and database search results. Project outputs will be beneficial for identifying risks in literature reviews, such as sponsor bias or the avoidance of citation of contradictory evidence, which will help reduce the spread of misinformation. This project is made possible by recent advances in network science and text mining methods, as well as the availability of abstracts, affiliation, citations, and funding data under suitable licenses for data science. The work is novel in bringing together complementary approaches that have not previously been combined: argumentation theory and the study of controversies; approaches for synthesizing evidence; and bibliometric and scientometric approaches for looking structurally at a field.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.