Coding Across the Disciplines: A Project-Based Workshop on Manual Text Annotation Techniques

该研讨会由政治学计划和科学与社会计划的政策，科学和工程组成部分的研究资助。 该研讨会将汇集社会和计算机科学家，讨论编码技术，以通过人工和机器注释的结合使用来重塑社会和政治现象的研究。 长期目标是培养一个持久的利益共同体，专注于发展新的注释研究伙伴关系，并以一种为一致性、可靠性、透明度和可复制性设定高科学标准的方式来实现。 编码文本语料库用于社会科学和计算科学的基础和应用研究。 然而，文本的手动注释“编码”通常以特别的、不一致的、不可复制的和不可靠的方式进行。 虽然来自不同学科的许多研究人员都可以从独立编码人员制作的可靠记录、公开可用、透明的大规模观察中受益，但目前很少有研究人员能够自信地说他们知道从哪里获取，甚至如何制作这样的注释文本语料库。 该讲习班的一个主要短期目标是在少数现有项目的努力基础上再接再厉，这些项目的特点是文本的手动注释。 一个相关的目标是促进一种特定的方法来手动编码，旨在为社会和政治问题的基础和应用研究构建算法。 该研讨会将是小型的，以项目为中心，并通过邀请。 所有研讨会的参与者将在一个长时间集中在一个项目的时间在一个花园式的设置在一起。 假设最多有六个项目，这两天将有两个小时的时间专门用于一个项目，以及一个总结会议，专门用于重要研讨会成果的协作总结。 所有参与的项目将被要求在研讨会之前通过研讨会wiki准备和传播一套特定的材料。 高质量的手动注释为社会科学家和计算科学家之间的跨学科研究提供了可能性。 这第二个机会的存在是因为计算科学的研究人员，特别是那些从事文本分类，信息检索，意见检测和自然语言处理（NLP）变体的研究人员，渴望在手动注释中实现难以捉摸的“黄金标准”。 具有足够大小和一致性的可靠编码文本语料库对于设计和训练NLP算法至关重要。

This workshop is funded by the Political Science Program and the Studies of Policy, Science and Engineering component of the Science and Society Program. The workshop will bring social and computer scientists together to discuss coding techniques in order to reshape the study of social and political phenomena via the combined use of manual and machine annotation. The longer-term goals are to foster a lasting community of interest focused around the development of new annotation research partnerships and to do so in a way that sets a high scientific standard for consistency, reliability, transparency, and replicability. Coded text corpora are used for basic and applied research in social and computational sciences. Yet the manual annotation of text "the coding" is often conducted in an ad hoc, inconsistent, non-replicable, and unreliable manner. While many researchers from a variety of disciplines stand to benefit from reliably recorded, publicly available, transparent, large-scale observations produced by independent coders, very few current researchers can say with confidence that they know where to acquire, or even how to produce, such annotated text corpora. A primary short-term objective of this workshop is to build on the efforts of a small number of existing projects that feature the manual annotation of text. A related goal is to foster a particular approach to manual coding that is designed to be useful for building algorithms for basic and applied research on social and political issues. The workshop will be small, project-centric, and by invitation only. All of the workshop participants will be together in a seminar-style setting focusing on one project at a time for an extended period. Assuming a maximum of six projects, both days will feature a 2-hour block of time devoted to a single project, as well as a wrap-up session devoted to the collaborative summarization of important workshop findings. All participating projects will be required to prepare and disseminate a specific set of materials in advance of the workshop via a workshop wiki. High quality manual annotation opens up the possibility for cross-disciplinary studies featuring collaboration between social and computational scientists. This second opportunity exists because researchers in the computational sciences, particularly those working in text classification, information retrieval, opinion detection, and variants of natural language processing (NLP), are hungry for the elusive "gold standard" in manual annotation. Reliably coded text corpora of sufficient size and consistency are essential to design and train NLP algorithms.