Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty
Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty
复制标题
DOI：
10.18653/v1/2020.nlpcss-1.13
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Katherine A. Keith;Christoph Teichmann;Brendan T. O'Connor;E. Meij
中科院分区：
文献类型：
--
作者：
Katherine A. Keith;Christoph Teichmann;Brendan T. O'Connor;E. Meij
方法和应用在科学中有着千丝万缕的联系，特别是在文本数据领域。在本文中，我们研究了这样一个文本即数据应用程序，这是一种既定的经济指数，可以根据新闻中关键词的出现来衡量经济政策的不确定性。该指数被证明与企业投资、就业和超额市场回报相关，对私营部门和学术界都产生了实质性影响。然而，当我们重新审视和扩展原作者的注释和文本测量时，我们发现了有趣的文本即数据方法论研究问题：（1）注释者的分歧是否反映了语言的歧义？ (2) 替代文本测量是否相互关联以及与外部预测有效性的测量是否相关？我们发现，对于该应用程序，（1）经济政策不确定性的一些注释者分歧可归因于语言的模糊性，（2）将测量从关键字匹配转换为监督机器学习分类器会导致相关性较低，这对指数的有效性产生了令人担忧的影响。
Methods and applications are inextricably linked in science, and in particular in the domain of text-as-data. In this paper, we examine one such text-as-data application, an established economic index that measures economic policy uncertainty from keyword occurrences in news. This index, which is shown to correlate with firm investment, employment, and excess market returns, has had substantive impact in both the private sector and academia. Yet, as we revisit and extend the original authors’ annotations and text measurements we find interesting text-as-data methodological research questions: (1) Are annotator disagreements a reflection of ambiguity in language? (2) Do alternative text measurements correlate with one another and with measures of external predictive validity? We find for this application (1) some annotator disagreements of economic policy uncertainty can be attributed to ambiguity in language, and (2) switching measurements from keyword-matching to supervised machine learning classifiers results in low correlation, a concerning implication for the validity of the index.