Collaborative Research: SOS-DCI / HNDS-R: Advancing Semantic Network Analysis to Better Understand How Evaluative Exchanges Shape Scientific Arguments

同行评议是科学过程的核心，但它的大部分内容对参与其中的人来说是隐藏的。作者是否反驳和/或实施了其中一些评论？同行评议是一个社会过程，在这个过程中，评议者和作者就新的科学工作的优点交换相互竞争的论点。在这些交流中，作者在提交的材料中提出新的论点，这些论点受到同行评审员的批判性评价，他们对其中一些论点提出质疑，并肯定其他论点。然后，作者用反驳来回应这些评论，有时承认，有时反驳。这种同行评议的过程--交换论点、评价和回应--是一种社会的、制度化的手段，用来塑造作为科学知识发表的东西。这个项目将系统地描述这种交换过程及其变化，揭示不同领域如何建立社会接受的知识主张。为此，我们打算：代表科学论证的层次逻辑依赖性;辨别认识论价值的不同表达;并开发一种方法，确定在何处以及何时以互文方式表示这些评价。当这些模式的交流和评价被视为在整体上，他们将提供作者，审稿人，编辑和现场参与者的一种手段，通过观察这些隐藏的认识论审议，反思其优点，并可能帮助参与者倡导改进同行评议。我们开发的技术还应该使社会科学家能够更普遍地系统地研究评价性交流，这种方法可以用来观察评价性规范在不同的社会和制度背景下是如何实践的。使用数以万计的评价性交流和近10万篇OpenReview和JSTOR的学术文本，我们将在三个相互关联的阶段系统地研究同行学者之间的评价交流：作者如何提出论点，同行如何评价和评论它们，以及作者如何反驳和修改他们的手稿以回应这些评论。我们将应用自然语言处理技术，包括话语分析，修辞结构理论和网络分析，以调查私人同行评审和公共评审和回应中的评估交流。这项工作有三个目标：首先，发现科学论证的语义结构，以及它们在不同学术领域内和不同学术领域之间的变化。这就需要理解科学家用来辩论的主张、论据和证据的种类;这些在网络形式或结构中是如何相互关联的;以及某些形式又是如何特定于不同的科学文化的。第二，我们的目标是发现潜在的互文结构的科学论点，特别是，如何回应科学家伪造的论点，重新安排，省略，甚至强调语义关系的其他科学家的论点，在文本中。这就需要创新“文本即数据”的方法，以观察和衡量不同文本之间的离散联系。最后，我们的目标是了解科学家改变主意的条件：他们的论点如何得到修改，以及如何实现解决方案。这就需要理解论点结构是否以及在多大程度上被反驳或回应论点更新或调整。该项目由科学的科学：发现，通信和影响以及人类网络和数据科学计划共同资助。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Peer review is central to the scientific process, but much of it is hidden from those who engage in it. What issues do reviewers focus on? Do authors rebut and/or implement some of these comments more than others? Peer review is a social process where reviewers and authors exchange competing arguments about the merits of new scientific work. In these exchanges, authors advance new arguments in their submissions, which are critically evaluated by peer reviewers who contest some of these arguments and affirm others. Authors then respond to these reviews with counter-arguments, sometimes conceding and other times refuting. This process of peer review - the exchange of argument, evaluation, and response – is a social, institutional means of shaping what is published as scientific knowledge. This project will systematically depict this exchange process and its variation, revealing how different fields establish socially accepted knowledge claims. We intend to do this by: representing the hierarchical logical dependencies of scientific arguments; discerning distinct expressions of epistemological value; and developing a method that identifies where and when such evaluations are intertextually represented. When these patterns of exchange and evaluation are viewed in the aggregate, they will offer authors, reviewers, editors, and field participants a means by which to observe these hidden epistemological deliberations, to reflect on their merits, and to potentially help participants advocate for improvements in peer review. The techniques we develop should also enable social scientists to systematically study evaluative exchanges more generally, and this approach can be used to see how evaluative norms are practiced in different social and institutional contexts.Using tens of thousands of evaluative exchanges and nearly 100,000 scholarly texts of OpenReview and JSTOR, we will systematically study evaluative exchanges among peer scholars over the course of three interrelated phases: how authors advance arguments, how peers evaluate and review them, and how authors counterargue and revise their manuscript in response to those reviews. We will apply natural language processing techniques including discourse parsing, rhetorical structural theory, and network analysis to investigate evaluative exchanges in both private peer reviews and public review-and-responses. The work has three aims: first, to discover the semantic structure of scientific arguments and how they vary within and across distinct scholarly fields. This entails understanding the kinds of claims, warrants, and evidence scientists use to debate; how these are interrelated in network forms or structures; and how certain forms in turn are specific to different scientific cultures. Second, we aim to discover the latent inter-textual structure of scientific arguments and, in particular, how responding scientists forge arguments that rearrange, omit, or even emphasize the semantic relations of other scientists’ arguments as laid out in texts. This entails innovating “text as data” methods in order to observe and measure discrete connections across disparate texts. Finally, we aim to understand the conditions under which scientists change their minds: how their arguments get revised and how resolution is achieved. This entails understanding whether and to what extent argument structures are updated or aligned by counter- or responding arguments. The project is co-funded by the Science of Science: Discovery, Communications, and Impact and the Human Networks and Data Science Programs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.