EAGER: CISE/IIS/RI/Program Element 7495: Crowdsourcing for NLP: Exploring Two Approaches

“众包”是利用“群体智慧”的想法，也就是说，将大量非专家的判断结合起来，为复杂问题提供可靠的答案。 在自然语言处理（NLP）领域，注释句子以显示它们表达了什么事件（以及句子的哪些部分表达了哪些参与者）是一项非常复杂的任务。 例如，句子“Maria rides the bus from home to her office”应该被识别为Ride_vehicle事件，其中“Maria”作为Mover，“the bus”作为Vehicle，“from home”作为Source，“to her office”作为Goal; NLP系统还应该能够识别句子中具有相同参与者的相同事件“Maria从家到她的办公室的公共汽车车程需要40分钟”，FrameNet（http：framenet.icsi.berkeley.edu）正在建立一个包含数百种事件类型（称为“语义框架”）的词汇数据库，以及每种事件类型在注释句子中的示例，这些示例可用于训练NLP系统。 但是，专家对句子的注释是缓慢和昂贵的;这个项目正在测试众包是否可以加快创建这样的数据库，特别是通过探索两种众包技术，看看哪种更适合这些任务：（1）网络游戏，玩家们要比赛看谁能快速准确地注释（类似于“冗长”游戏）和（2）一个系统，其中人们支付少量的钱来完成这些任务，使用亚马逊的“土耳其机械”（www.mturk.com）。 如果成功的话，这些技术可以用来为新的自然语言处理系统建立更好的数据库，真正理解“谁对谁做了什么”，从而改善问题回答和网络搜索。

"Crowdsourcing" is the idea of using the "wisdom of crowds", that is, combining large numbers of judgments by non-experts, to produce reliable answers to complex problems. In the field of natural language processing(NLP), annotating sentences to show what events they express (and which parts of the sentence express which participants) is such a complex task. For example, the sentence "Maria rides the bus from home to her office" should be recognized as a Ride_vehicle event, with "Maria" as Mover, "the bus" as the Vehicle, "from home" as the Source and "to her office" as the Goal; NLP systems should also be able to recognize the same event with the same participants in the sentence "Maria's bus ride from home to her office takes 40 minutes", but most current systems cannot.FrameNet (http://framenet.icsi.berkeley.edu) is building a lexical database of hundreds of event types (called "semantic frames") and examples of each in annotated sentences, which can be used to train NLP systems. But expert annotation of sentences is slow and expensive; this project is testing whether crowdsourcing can speed up the creation of such databases, specifically by exploring two crowdsourcing techniques to see which works better for these tasks: (1) online games, where players compete to see who can annotate rapidly and accurately (similar to the "Verbosity" game) and (2) a system in which people are paid small amounts of money to complete such tasks, using Amazon's "Mechanical Turk" (www.mturk.com). If successful, these techniques could be used to build better databases for new NLP systems that really understand "who did what to whom", thus improving question answering and web searching.