Collaborative Research: SOS-DCI / HNDS-R: Advancing Semantic Network Analysis to Better Understand How Evaluative Exchanges Shape Scientific Arguments

同行评议是科学过程的核心，但其中很多内容对参与评议的人来说是隐藏的。审稿人关注哪些问题？作者是否比其他人更多地反驳和/或实现这些评论？同行评议是一个社会过程，在这个过程中，审稿人和作者就新科学工作的优点交换不同的观点。在这些交流中，作者在提交的材料中提出新的论点，这些论点由同行审稿人进行批判性评估，审稿人对其中一些论点提出异议，并肯定其他论点。然后作者们用反驳来回应这些评论，有时是承认，有时是反驳。这种同行评议的过程——交换论点、评价和回应——是形成作为科学知识发表的内容的一种社会的、制度性的手段。该项目将系统地描述这种交流过程及其变化，揭示不同领域如何建立社会接受的知识主张。我们打算这样做：表示科学论证的层次逻辑依赖性；辨别认识论价值的不同表达；并开发一种方法来确定这些评估在何时何地被互文表示。当这些交流和评估模式被放在一起看时，它们将为作者、审稿人、编辑和领域参与者提供一种方法，通过这种方法可以观察这些隐藏的认识论讨论，反思它们的优点，并潜在地帮助参与者倡导改进同行评审。我们开发的技术也应该使社会科学家能够更广泛地系统地研究评价性交换，这种方法可以用来观察评价性规范在不同的社会和制度背景下是如何实践的。利用OpenReview和JSTOR的数万篇评价性交流和近10万篇学术文章，我们将系统地研究同行学者之间的评价性交流，分为三个相互关联的阶段：作者如何提出论点，同行如何评估和审查他们，作者如何反驳和修改他们的手稿以回应这些评论。我们将运用自然语言处理技术，包括语篇分析、修辞结构理论和网络分析来调查私人同行评议和公共评议-回应中的评议交流。这项工作有三个目的：第一，发现科学论证的语义结构，以及它们在不同学术领域内部和不同学术领域之间的差异。这需要理解科学家用来辩论的各种主张、授权和证据；它们在网络形式或结构中是如何相互关联的；以及不同的科学文化对特定形式的影响。其次，我们的目标是发现科学论证的潜在文本间结构，特别是科学家如何做出回应，形成论据，重新排列、省略甚至强调文本中其他科学家论证的语义关系。这需要创新“文本即数据”的方法，以便观察和测量不同文本之间的离散联系。最后，我们的目标是了解科学家改变想法的条件：他们的论点如何得到修正，以及如何达成解决方案。这需要理解参数结构是否以及在多大程度上被反参数或响应参数更新或对齐。该项目由科学科学：发现、通信和影响以及人类网络和数据科学项目共同资助。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Peer review is central to the scientific process, but much of it is hidden from those who engage in it. What issues do reviewers focus on? Do authors rebut and/or implement some of these comments more than others? Peer review is a social process where reviewers and authors exchange competing arguments about the merits of new scientific work. In these exchanges, authors advance new arguments in their submissions, which are critically evaluated by peer reviewers who contest some of these arguments and affirm others. Authors then respond to these reviews with counter-arguments, sometimes conceding and other times refuting. This process of peer review - the exchange of argument, evaluation, and response – is a social, institutional means of shaping what is published as scientific knowledge. This project will systematically depict this exchange process and its variation, revealing how different fields establish socially accepted knowledge claims. We intend to do this by: representing the hierarchical logical dependencies of scientific arguments; discerning distinct expressions of epistemological value; and developing a method that identifies where and when such evaluations are intertextually represented. When these patterns of exchange and evaluation are viewed in the aggregate, they will offer authors, reviewers, editors, and field participants a means by which to observe these hidden epistemological deliberations, to reflect on their merits, and to potentially help participants advocate for improvements in peer review. The techniques we develop should also enable social scientists to systematically study evaluative exchanges more generally, and this approach can be used to see how evaluative norms are practiced in different social and institutional contexts.Using tens of thousands of evaluative exchanges and nearly 100,000 scholarly texts of OpenReview and JSTOR, we will systematically study evaluative exchanges among peer scholars over the course of three interrelated phases: how authors advance arguments, how peers evaluate and review them, and how authors counterargue and revise their manuscript in response to those reviews. We will apply natural language processing techniques including discourse parsing, rhetorical structural theory, and network analysis to investigate evaluative exchanges in both private peer reviews and public review-and-responses. The work has three aims: first, to discover the semantic structure of scientific arguments and how they vary within and across distinct scholarly fields. This entails understanding the kinds of claims, warrants, and evidence scientists use to debate; how these are interrelated in network forms or structures; and how certain forms in turn are specific to different scientific cultures. Second, we aim to discover the latent inter-textual structure of scientific arguments and, in particular, how responding scientists forge arguments that rearrange, omit, or even emphasize the semantic relations of other scientists’ arguments as laid out in texts. This entails innovating “text as data” methods in order to observe and measure discrete connections across disparate texts. Finally, we aim to understand the conditions under which scientists change their minds: how their arguments get revised and how resolution is achieved. This entails understanding whether and to what extent argument structures are updated or aligned by counter- or responding arguments. The project is co-funded by the Science of Science: Discovery, Communications, and Impact and the Human Networks and Data Science Programs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.