Collaborative Research: Updating the Militarized Dispute Data Through Crowdsourcing: MID5

战争相关项目的军事化国家间争端（MID）数据是国际冲突研究中最突出和最常用的数据收集。最新版本（MID4）于2014年发布，涵盖的时间段为1816-2010年。MID4项目利用自动文本分类过程，使识别相关新闻故事的过程更加高效。在该项目的过程中，pi确定工作流中的主要瓶颈是那些新闻文档的编码。为了解决这种低效率问题，PIs完成了一个试点项目，以确定是否可以使用众包技术来编写这些文档。在试点项目中，非专业工人获得小额报酬，阅读文件并回答一系列问题，这些问题的答案用于识别可能的军事化事件（构成MIDs的事件）的特征。将众包的回答与MID4项目训练有素的编码员的回答进行系统比较后发现，对于编码的新闻报道，众包编码的68%是完全准确的；更重要的是，人群对特定报告的反应高度一致与正确编码密切相关。这使pi能够检测哪些文档需要进一步的专家参与。因此，pi可以在接近实时的情况下以有限的财务成本生成大部分MID数据。这些程序应用于MID5项目，该项目将更新2011-2017年期间的MID5数据。MID5项目工作流程开始于从LexisNexis检索文档，并使用MID4中实现的软件和方法对文档进行分类。我们丢弃负机密文件，并继续从正机密文件中提取元数据，包括文件标题、发布报告的新闻机构、日期和文本中提到的任何演员。众工是通过亚马逊的土耳其机器人（Mechanical Turk）招聘来的，他们支付工资，阅读其中一份文件，并回答一系列简单、客观的问题。问卷是预定义的，但是会自动将一些提取的元数据插入到问卷中，以提高回答的质量。几个工作人员为每个文档完成一份调查问卷，这给pi留下了聚合问题：如何将多个工作人员的回答（可能涉及多个相关问题）组合成编写军事化事件所需的可用数据。在初步研究中，pi表明贝叶斯网络是实现这种聚合的最有效方法。最近，pi在混合深度受限玻尔兹曼机的半监督文本分类方面取得了进展，在该任务中优于以前的方法。

General Summary The Correlates of War Project's Militarized Interstate Dispute (MID) Data is the most prominent and heavily used data collection in the study of international conflict. The most recent version (MID4) was released in 2014 and brings the period covered to 1816-2010. The MID4 project utilized automated text classification procedures to make the process of identifying relevant news stories more efficient. Over the course of that project, the PIs determined the primary bottleneck in the workflow was the coding of those news documents. To address this inefficiency, The PIs completed a pilot project to determine whether crowdsourcing techniques could be used to code these documents. In the pilot, non-expert workers were paid small sums to read documents and to answer sets of questions, the answers to which were used to identify features of possible militarized incidents (the events that comprise MIDs). A systematic comparison of the crowdsourced responses with those of MID4 Project's trained coders revealed that the crowdsourced codings were completely accurate for 68 percent of the news reports coded; more importantly, high agreement among crowd responses on specific reports was strongly associated with correct coding. This enables the PIs to detect which documents require further expert involvement. As a result, the PIs can produce a majority of the MID data in near-realtime and at limited financial cost. These procedures are applied on the MID5 Project, which will update the MID data for the period 2011-2017.Technical Summary The MID5 project workflow begins with document retrieval from LexisNexis and document classification using the software and methods implemented in MID4. We discard the negatively classified documents, and proceed to extract metadata from the positively classified documents including the document title, the news agency that published the report, the date, and any actors mentioned in the text. Crowd workers are recruited through Amazon's Mechanical Turk and paid a wage to read one of these documents and answer a line of simple, objective questions about it. The questionnaire is predefined, but some extracted metadata is automatically inserted into the questionnaire to improve the quality of responses. Several workers complete a questionnaire for each document, leaving the PIs with problems of aggregation: how to combine multiple worker responses, possibly regarding multiple related questions, into usable data necessary to code the militarized incident. In the pilot study, the PIs show that Bayesian networks are the most effective way to achieve this aggregation. Recently, the PIs have made advances in semi-supervised text classification with hybrid, Deep Restricted Boltzmann Machines, which outperform previous methods in this task.