A 'gold standard' of institutional assessment? Operationalizing and explaining political biases in large numbers of international organization evaluation reports

在过去几十年中，评价已成为国际组织决策的一个重要组成部分。作为一种看似技术官僚的公共管理工具，评价被认为是服务于循证政策调整和学习的目的。然而，现有文献强调，评价结果基本上证实了强大的利益攸关方预先确定的政治利益。虽然基于看法的证据（调查、访谈）似乎证实了这种期望，但实际上没有对评价报告本身的实质内容进行经验研究。这个项目的目的就是填补这个空白。它的概念化，措施，并通过询问利益攸关方（IO官僚机构或成员国）控制独立评价单位的资源（预算，议程设置）的差异在多大程度上导致评价报告中的政治偏见解释的政治偏见。政治偏见被定义为评价报告内容和形式的系统模式（如评价结果的正面或负面），反映预先确定的利益攸关方利益。通过创建、编码和分析一个包含8,000多份评价报告的大型文本语料库，该项目提供了19个国际组织评价报告中政治偏见的系统性经验数据。在方法上，该项目建立在文本即数据方法的最新发展基础上，采用定量和定性内容分析。如果这些数据证实了关于在信息和组织评价中存在政治偏见的主要假设，这将导致一个强有力的主张，即评价实际上错过了它在政策周期中的目标;也就是说，对政策执行中什么起作用，什么不起作用进行批判性反思。然而，对主要论点的证伪将强化评价作为机构评估的“黄金标准”的想法。因此，该项目的研究结果将对我们理解评价在政策进程中的作用产生重大的理论和实际影响。

Over the past decades, evaluation has become a crucial part of international organizations’ (IOs) policy-making. As a seemingly technocratic public management tool, evaluation is thought to serve the purpose of evidence-based policy adjustment and learning. However, existing literature has highlighted that evaluation results essentially confirm the pre-defined political interests of powerful stakeholders. While perception-based evidence (surveys, interviews) seems to confirm such expectations, there is virtually no empirical research on the substance of evaluation reports themselves. The aim of this project is to fill this gap. It conceptualizes, measures, and explains political biases in IO evaluation reports by asking to what extent differences in stakeholder (IO bureaucracy or member states) control over the resources (budget, agenda setting) of independent evaluation units lead to political biases in evaluation reports. Political biases are defined as systematic patterns in the content and form of evaluation reports (e.g. positivity or negativity of findings) that reflect pre-defined stakeholder interests. By creating, coding, and analyzing a large text corpus database with over 8,000 evaluation reports, the project provides systematic empirical data on political biases in evaluation reports across 19 IOs. Methodologically, the project builds on recent developments in text-as-data approaches, employing both quantitative and qualitative content analysis. If the data confirms the main hypothesis about the presence of political biases in IO evaluations, this would result in a strong claim that evaluation actually misses its goal in the policy-cycle; that is, critical reflection about what works and what not in policy implementation. Falsification of the main thesis, however, would reinforce the idea of evaluation as a ‘gold standard’ of institutional assessment. The project’s findings will therefore have significant theoretical and practical implications for our understanding of the role of evaluation in policy processes.