CAREER: Using network analysis to assess confidence in research synthesis

该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。最佳可用科学是为节能、能源、医疗保健和可持续发展等领域的政策提供信息的重要因素。确定最好的可用科学需要综合多个科学结果，以衡量科学共识的水平和研究的可靠性。然而，在一些与政策相关的话题上，不同的合成得出了互不相容的结论。证据合成中的这种不一致浪费金钱，产生误导性的结果，并可能导致影响大量人的糟糕决定。通过研究、教育和推广，这个职业项目旨在开发和测试一个新的工具和工作流程框架，以揭示专家文献中潜在的偏见来源。该框架将使利益攸关方能够迅速了解哪些个人、机构和资助者参与了证据的创建。它将评估造成偏见风险的其他因素，以及专家群体对所提出证据的信任程度。研究成果可以促进在广泛领域进行数据驱动的决策。例子包括能源和环境科学以及健康科学的主题，如各种形式的食品生产的碳足迹、群体免疫和疫苗有效性。该项目还将通过从服务不足的人群中聘请学生助理，并通过开发两门与政策相关的STEM大学课程和一部中学职业视频来吸引代表性不足的学生，从而帮助实现科学劳动力的多样化。该项目探索如何在规模上改进对研究的信心评估。它将使寻求证据的人能够迅速了解一系列文献中的共识水平，以及可能影响研究可靠性的风险因素，为稳健性和重复性提供关键资源。这一框架可以应用于任何书目，包括同行评审的手稿、已发表的文章和数据库搜索结果。项目产出将有助于确定文献审查中的风险，如赞助者偏见或避免引用相互矛盾的证据，这将有助于减少错误信息的传播。这个项目是由于网络科学和文本挖掘方法的最新进展，以及摘要、从属关系、引文和数据科学适当许可证下的资助数据的可用性而成为可能的。这项工作在将以前从未结合在一起的互补方法结合在一起方面是新颖的：论证理论和争议研究；合成证据的方法；以及从结构上看领域的文献计量学和科学计量学方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The best available science is an important factor that informs policy in areas such as conservation, energy, healthcare, and sustainable. Determining the best available science requires synthesizing multiple scientific results to gauge both the level of scientific consensus and the reliability of the research. However, on some policy-relevant topics, different syntheses come to incompatible conclusions. Such inconsistency in the synthesis of evidence wastes money, generates misleading results, and can lead to poor decisions impacting large numbers of people. Through research, education, and outreach, this CAREER project aims to develop and test a novel framework of tools and workflows that will reveal potential sources of bias in expert literature. The framework will enable stakeholders to quickly understand which individuals, institutions and funders contributed to the creation of the evidence. It will assess other factors that create risk of bias as well as the degree of confidence an expert community has in the evidence presented. Research outcomes could facilitate data-driven decision-making in a broad range of areas. Examples include topics in energy and environmental sciences and health sciences, like the carbon footprint of various forms of food production, herd immunity, and vaccine effectiveness. This project will also help diversify the science workforce by employing student assistants from underserved populations and by developing two policy-relevant STEM university courses and a middle school career video to attract underrepresented students.This project explores how to improve the assessment of confidence in research at scale. It will enable evidence-seekers to quickly understand the level of consensus within a body of literature, along with risk factors that might impact reliability of research, providing a key resource for robustness and reproducibility. This framework can be applied to any bibliography, including manuscripts under peer review, published articles, and database search results. Project outputs will be beneficial for identifying risks in literature reviews, such as sponsor bias or the avoidance of citation of contradictory evidence, which will help reduce the spread of misinformation. This project is made possible by recent advances in network science and text mining methods, as well as the availability of abstracts, affiliation, citations, and funding data under suitable licenses for data science. The work is novel in bringing together complementary approaches that have not previously been combined: argumentation theory and the study of controversies; approaches for synthesizing evidence; and bibliometric and scientometric approaches for looking structurally at a field.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.