EAGER: CISE/IIS/RI/Program Element 7495: Crowdsourcing for NLP: Exploring Two Approaches

“众包”是利用“群体智慧”的理念，即结合非专家的大量判断，为复杂问题提供可靠的答案。在自然语言处理（NLP）领域，对句子进行注释以显示它们所表达的事件（以及句子的哪些部分表达了哪些参与者）是一项非常复杂的任务。例如，句子“Maria乘公共汽车从家到办公室”应该被认为是一个Ride_vehicle事件，其中“Maria”是Mover，“the bus”是Vehicle，“from home”是Source，“to her office”是Goal；NLP系统还应该能够识别句子中相同参与者的相同事件，“Maria从家到办公室坐公交车需要40分钟”，但目前大多数系统都不能。FrameNet （http://framenet.icsi.berkeley.edu）正在建立一个包含数百种事件类型（称为“语义框架”）的词汇数据库，以及每种事件在带注释的句子中的示例，可用于训练NLP系统。但是专家级的句子注释既慢又贵；这个项目正在测试众包是否可以加速这样的数据库的创建，特别是通过探索两种众包技术来看看哪一种更适合这些任务：(1)在线游戏，玩家竞争谁能快速准确地注释（类似于“冗长”游戏）；(2)一个系统，在这个系统中，人们可以获得少量的钱来完成这些任务，使用亚马逊的“机械土耳其人”（www.mturk.com）。如果成功，这些技术可以用来为新的NLP系统建立更好的数据库，真正理解“谁对谁做了什么”，从而改进问题回答和网络搜索。

"Crowdsourcing" is the idea of using the "wisdom of crowds", that is, combining large numbers of judgments by non-experts, to produce reliable answers to complex problems. In the field of natural language processing(NLP), annotating sentences to show what events they express (and which parts of the sentence express which participants) is such a complex task. For example, the sentence "Maria rides the bus from home to her office" should be recognized as a Ride_vehicle event, with "Maria" as Mover, "the bus" as the Vehicle, "from home" as the Source and "to her office" as the Goal; NLP systems should also be able to recognize the same event with the same participants in the sentence "Maria's bus ride from home to her office takes 40 minutes", but most current systems cannot.FrameNet (http://framenet.icsi.berkeley.edu) is building a lexical database of hundreds of event types (called "semantic frames") and examples of each in annotated sentences, which can be used to train NLP systems. But expert annotation of sentences is slow and expensive; this project is testing whether crowdsourcing can speed up the creation of such databases, specifically by exploring two crowdsourcing techniques to see which works better for these tasks: (1) online games, where players compete to see who can annotate rapidly and accurately (similar to the "Verbosity" game) and (2) a system in which people are paid small amounts of money to complete such tasks, using Amazon's "Mechanical Turk" (www.mturk.com). If successful, these techniques could be used to build better databases for new NLP systems that really understand "who did what to whom", thus improving question answering and web searching.