A 'gold standard' of institutional assessment? Operationalizing and explaining political biases in large numbers of international organization evaluation reports

点击翻译按钮获取中文摘要

Over the past decades, evaluation has become a crucial part of international organizations’ (IOs) policy-making. As a seemingly technocratic public management tool, evaluation is thought to serve the purpose of evidence-based policy adjustment and learning. However, existing literature has highlighted that evaluation results essentially confirm the pre-defined political interests of powerful stakeholders. While perception-based evidence (surveys, interviews) seems to confirm such expectations, there is virtually no empirical research on the substance of evaluation reports themselves. The aim of this project is to fill this gap. It conceptualizes, measures, and explains political biases in IO evaluation reports by asking to what extent differences in stakeholder (IO bureaucracy or member states) control over the resources (budget, agenda setting) of independent evaluation units lead to political biases in evaluation reports. Political biases are defined as systematic patterns in the content and form of evaluation reports (e.g. positivity or negativity of findings) that reflect pre-defined stakeholder interests. By creating, coding, and analyzing a large text corpus database with over 8,000 evaluation reports, the project provides systematic empirical data on political biases in evaluation reports across 19 IOs. Methodologically, the project builds on recent developments in text-as-data approaches, employing both quantitative and qualitative content analysis. If the data confirms the main hypothesis about the presence of political biases in IO evaluations, this would result in a strong claim that evaluation actually misses its goal in the policy-cycle; that is, critical reflection about what works and what not in policy implementation. Falsification of the main thesis, however, would reinforce the idea of evaluation as a ‘gold standard’ of institutional assessment. The project’s findings will therefore have significant theoretical and practical implications for our understanding of the role of evaluation in policy processes.