Collaborative Research: SOS-DCI / HNDS-R: Advancing Semantic Network Analysis to Better Understand How Evaluative Exchanges Shape Scientific Arguments

同行评议是科学过程的核心，但它的大部分都是对那些参与其中的人隐藏的。评论者关注的问题是什么？作者是否比其他人更多地反驳和/或执行其中的一些评论？同行评议是一种社会过程，评价者和作者就新的科学工作的价值交换相互竞争的论点。在这些交流中，作者在他们的提交中提出了新的论点，并由同行评审员进行批判性评估，他们对其中一些论点提出异议，并确认另一些论点。然后，作者用反驳论据来回应这些评论，有时让步，有时反驳。这种同行评审的过程--交换论点、评估和回应--是一种社会的、制度的手段，可以塑造作为科学知识发表的东西。这个项目将系统地描述这种交流过程及其变化，揭示不同领域如何建立社会接受的知识主张。我们打算通过以下方式来做到这一点：表现科学论证的层级逻辑依赖关系；辨别认识论价值的不同表达；并开发一种方法，确定在哪里以及何时以互文性方式表示这种评价。当这些交流和评价模式被综合看待时，它们将为作者、审稿人、编辑和实地参与者提供一种观察这些隐藏的认识论审议的手段，反思它们的优点，并潜在地帮助参与者倡导改进同行审查。我们开发的技术也应该使社会科学家能够更广泛地系统地研究评价性交流，这种方法可以用来看看评价性规范是如何在不同的社会和制度背景下实践的。我们将使用OpenReview和JSTOR的数万个评价性交流和近10万篇学术文本，在三个相互关联的阶段系统地研究同行学者之间的评价性交流：作者如何提出论点，同行如何评价和评论他们，以及作者如何反驳和修改他们的手稿以回应这些评论。我们将应用自然语言处理技术，包括语篇分析、修辞结构理论和网络分析来调查私人同行评议和公共评议与回应中的评价性交流。这项工作有三个目标：第一，发现科学论点的语义结构，以及它们在不同的学术领域内和之间是如何变化的。这需要了解科学家用来辩论的主张、授权和证据的种类；这些在网络形式或结构中是如何相互关联的；以及某些形式又是如何针对不同的科学文化的。第二，我们的目标是发现科学论点潜在的文本间结构，特别是，回应的科学家如何伪造论点，重新排列、省略甚至强调其他科学家论点在语篇中的语义关系。这需要创新“文本即数据”的方法，以便观察和测量不同文本之间的离散联系。最后，我们的目标是了解科学家在什么情况下改变主意：他们的论点如何被修改，以及如何实现解决方案。这需要了解论元结构是否以及在多大程度上被反论元或响应性论元更新或对齐。该项目由科学科学：发现、通信和影响以及人类网络和数据科学计划共同资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Peer review is central to the scientific process, but much of it is hidden from those who engage in it. What issues do reviewers focus on? Do authors rebut and/or implement some of these comments more than others? Peer review is a social process where reviewers and authors exchange competing arguments about the merits of new scientific work. In these exchanges, authors advance new arguments in their submissions, which are critically evaluated by peer reviewers who contest some of these arguments and affirm others. Authors then respond to these reviews with counter-arguments, sometimes conceding and other times refuting. This process of peer review - the exchange of argument, evaluation, and response – is a social, institutional means of shaping what is published as scientific knowledge. This project will systematically depict this exchange process and its variation, revealing how different fields establish socially accepted knowledge claims. We intend to do this by: representing the hierarchical logical dependencies of scientific arguments; discerning distinct expressions of epistemological value; and developing a method that identifies where and when such evaluations are intertextually represented. When these patterns of exchange and evaluation are viewed in the aggregate, they will offer authors, reviewers, editors, and field participants a means by which to observe these hidden epistemological deliberations, to reflect on their merits, and to potentially help participants advocate for improvements in peer review. The techniques we develop should also enable social scientists to systematically study evaluative exchanges more generally, and this approach can be used to see how evaluative norms are practiced in different social and institutional contexts.Using tens of thousands of evaluative exchanges and nearly 100,000 scholarly texts of OpenReview and JSTOR, we will systematically study evaluative exchanges among peer scholars over the course of three interrelated phases: how authors advance arguments, how peers evaluate and review them, and how authors counterargue and revise their manuscript in response to those reviews. We will apply natural language processing techniques including discourse parsing, rhetorical structural theory, and network analysis to investigate evaluative exchanges in both private peer reviews and public review-and-responses. The work has three aims: first, to discover the semantic structure of scientific arguments and how they vary within and across distinct scholarly fields. This entails understanding the kinds of claims, warrants, and evidence scientists use to debate; how these are interrelated in network forms or structures; and how certain forms in turn are specific to different scientific cultures. Second, we aim to discover the latent inter-textual structure of scientific arguments and, in particular, how responding scientists forge arguments that rearrange, omit, or even emphasize the semantic relations of other scientists’ arguments as laid out in texts. This entails innovating “text as data” methods in order to observe and measure discrete connections across disparate texts. Finally, we aim to understand the conditions under which scientists change their minds: how their arguments get revised and how resolution is achieved. This entails understanding whether and to what extent argument structures are updated or aligned by counter- or responding arguments. The project is co-funded by the Science of Science: Discovery, Communications, and Impact and the Human Networks and Data Science Programs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.