CAREER: High-Agreement Crowdsourcing for Difficult Language-Understanding Tasks

职业：针对困难的语言理解任务的高度一致的众包

• 批准号: 2046556
• 负责人: Samuel Bowman
• 金额: $ 55万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2046556&HistoricalAwards=false
• 关键词: CAREER; Agreement; Crowdsourcing; Difficult; Language

When engineers build modern artificial intelligence (AI) systems for language problems like question answering, they use datasets of examples to teach the systems how to solve the problem, rather than programming the systems directly. These datasets of examples are often collected through crowd work, where a large population of non-specialists are hired to come up with example answers to questions, example summaries of documents, or the like. Having a diverse group of people provide data for pay is meant to make it possible to build specialized language technology systems quickly, and to ensure that they can cover a wide range of styles of language, but this has not always worked well in practice: Crowd work is often set up in a way that forces participants to work quickly and sloppily, and produces data that’s ineffective at teaching machines to do what we want. This award supports research that aims to fix this, by developing and evaluating best practices for crowd worker training, feedback, and bonus pay to help crowd-worker dataset creators develop professional skills and produce better data that will lead to truly effective language technologies. The project award will also support parallel efforts at training new scientists and engineers, including programming targeting advanced technical students and outreach events targeting newcomers to the field.Technically, the project will establish a scientifically-grounded set of practices for crowdsourced data collection for natural language understanding tasks like reading comprehension question answering, coreference resolution, and natural language inference, with a focus on methods that can ensure that the resulting data is diverse, challenging, and high-quality in the face of obstacles posed by subjectivity and legitimate annotator disagreements. The main experiments to isolate the effect of several novel techniques for data collection, covering the training, feedback, and incentive structures used in crowdsourced data collection. A complementary thread will evaluate and refine task designs with the goal of identifying the task formulations that best isolate and reinforce model abilities to understand and reason with texts, informed by large experimental surveys of existing tasks. The accompanying education program will scale up processes for research mentorship to reach a larger fraction of the diverse and qualified undergraduate and graduate student population at New York University, both through seminars and taught research methods courses. The accompanying outreach plan will support the development of a recurring workshop series for early-year undergraduates tentatively interested in careers in AI and language technology, recruiting especially from groups underrepresented in computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当工程师为语言问题（如问答）构建现代人工智能（AI）系统时，他们使用示例数据集来教系统如何解决问题，而不是直接对系统进行编程。这些示例的数据集通常通过群体工作来收集，其中雇用大量非专业人员来提出问题的示例答案、文档的示例摘要等。让不同的人群提供薪酬数据是为了能够快速构建专业的语言技术系统，并确保它们能够覆盖广泛的语言风格，但这在实践中并不总是有效：人群工作的设置方式往往迫使参与者快速而草率地工作，并产生无法教会机器做我们想要做的事情的数据。该奖项支持旨在解决这一问题的研究，通过开发和评估人群工作者培训，反馈和奖金支付的最佳实践，帮助人群工作者数据集创建者发展专业技能并产生更好的数据，从而实现真正有效的语言技术。该项目奖还将支持培训新科学家和工程师的平行努力，包括针对高级技术学生的编程和针对该领域新人的外展活动。从技术上讲，该项目将建立一套科学的实践，用于自然语言理解任务的众包数据收集，如阅读理解问题回答，共指消解和自然语言推理，重点是可以确保所产生的数据是多样化的，具有挑战性的，以及面对主观性和合法注释者分歧所造成的障碍的高质量的方法。主要的实验，以隔离数据收集的几种新技术的效果，包括培训，反馈和激励结构中使用的众包数据收集。一个补充的线程将评估和完善任务设计，目标是确定任务的制定，最好的隔离和加强模型的能力，理解和推理的文本，通过现有任务的大型实验调查。伴随的教育计划将扩大研究导师的过程，以达到更大比例的多元化和合格的本科生和研究生人口在纽约大学，无论是通过研讨会和教授的研究方法课程。伴随的外展计划将支持为对人工智能和语言技术职业暂时感兴趣的早期本科生开发一个经常性的研讨会系列，特别是从计算领域代表性不足的群体中招募。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail

• DOI: 10.18653/v1/2022.acl-long.516
• 发表时间: 2021-10
• 作者: Sam Bowman

Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers

• DOI: 10.18653/v1/2021.blackboxnlp-1.42
• 发表时间: 2021-09
• 作者: Jason Phang;Haokun Liu;Samuel R. Bowman

Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions

单轮辩论无助于人类回答困难的阅读理解问题

• DOI: 10.18653/v1/2022.lnls-1.3
• 发表时间: 2022
• 期刊: Proceedings of the First Workshop on Learning with Natural Language Supervision
• 作者: Parrish, Alicia;Trivedi, Harsh;Perez, Ethan;Chen, Angelica;Nangia, Nikita;Phang, Jason;Bowman, Samuel
• 通讯作者: Bowman, Samuel

BBQ: A hand-built bias benchmark for question answering

• DOI: 10.18653/v1/2022.findings-acl.165
• 发表时间: 2021-10
• 作者: Alicia Parrish;Angelica Chen;Nikita Nangia;Vishakh Padmakumar;Jason Phang;Jana Thompson;Phu Mon Htut;Sam Bowman

Does Putting a Linguist in the Loop Improve NLU Data Collection?

让语言学家参与其中是否可以改善 NLU 数据收集？

• DOI: 10.18653/v1/2021.findings-emnlp.421
• 发表时间: 2021
• 期刊: Findings of the Association for Computational Linguistics: EMNLP 2021
• 作者: Parrish, Alicia;Huang, William;Agha, Omar;Lee, Soo-Hwan;Nangia, Nikita;Warstadt, Alexia;Aggarwal, Karmanya;Allaway, Emily;Linzen, Tal;Bowman, Samuel R.
• 通讯作者: Bowman, Samuel R.