III: Small: Labeling Massive Data from Noisy, Incomplete and Crowdsourced Annotations

随着深度学习的繁荣，对可靠标记数据的需求空前高涨。标签获取是一项非常重要的任务-数据标签是乏味的，劳动密集型的，并且容易出错。众包技术集成了来自多个注释器的注释以提高准确性，这对于标记大规模数据至关重要。然而，现有的众包技术面临着迫切的挑战，如繁重的工作量的注释，计算成本高，缺乏强有力的理论保证。 该项目将开发一系列分析和计算工具，用于从嘈杂的，不完整的和众包的注释中准确标记大量数据集，并提供可证明的保证。利用先进的非负矩阵分解理论，该项目将提供在临界条件下高效和有效的解决方案。预计这些成果将对目前渴望标签的人工智能行业和数据注释劳动力产生广泛而实质性的积极影响。例如，设计用于处理结构化数据的算法（例如，语音）将极大地有益于及时的应用，例如，智能助手，如Alexa和Siri。在大部分不完整的数据下可靠工作的能力将有助于设计新的数据分发方案，从而大大减少注释器的工作量。该项目还将为本科生提供许多培训机会，重点是吸引那些代表性不足的群体。在理论和方法方面，众包数据标签的许多方面（例如，样本复杂性、噪声鲁棒性和潜在统计模型的可识别性）仍然知之甚少。该项目将提供一套理论和计算工具，推进这些方面。具体而言，第一个推力将建立一个耦合的非负矩阵分解（CNMF）框架，桥梁的经典Dawid-Skene模型的众包和先进的非负因素分析理论。这将为关键条件下的众包建立坚实的理论基础，并导致理论支持的算法，以实现大幅改善的样本复杂性和噪声/不完整数据的鲁棒性。第二个推力利用领域相关知识，例如，数据结构和注释器依赖性，以提出用于增强性能的情境感知众包技术。第三推力设计随机优化策略，提供可扩展的CNMF框架的实现，并评估所提出的方法在各种现实世界的应用。该项目开发的分析和计算工具将为众包数据标签的长期挑战提供强有力的可证明保证和令人耳目一新的算法解决方案。此外，CNMF理论和算法是计算线性代数令人兴奋的新方向，其影响可以远远超出这个项目。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Alongside the prosperity of deep learning, the demand for reliably labeled data is unprecedentedly high. Label acquisition is a highly nontrivial task---data labeling is tedious, labor-intensive, and prone to mistakes. Crowdsourcing techniques that integrate annotations from multiple annotators to improve accuracy have been essential for labeling large-scale data. However, existing crowdsourcing techniques face pressing challenges such as heavy workload of annotators, high computational cost, and a lack of strong theoretical guarantees.  This project will develop a series of analytical and computational tools for accurately labeling massive datasets from noisy, incomplete, and crowdsourced annotations---with provable guarantees. Leveraging advanced nonnegative matrix factorization theory, this project will offer solutions that are efficient and effective under critical conditions. The outcomes are expected to have broad and substantial positive impacts on the currently label-hungry artificial intelligence industry and the data annotation workforce. For example, the algorithms designed for handling structured data (e.g., speech) will largely benefit timely applications, e.g., intelligent assistants such as Alexa and Siri. The ability of reliably working under largely incomplete data will help design new data dispatch schemes leading to significantly reduced annotator workload. The project will also offer many training opportunities for undergraduate students, with an emphasis on engaging those from underrepresented groups.In terms of theory and methods, many aspects of crowdsourced data labeling (e.g., sample complexity, noise robustness, and identifiability of the underlying statistical model) are still poorly understood. This project will provide a suite of theoretical and computational tools that advance these aspects. To be specific, the first thrust will build up a coupled nonnegative matrix factorization (CNMF) framework that bridges the classic Dawid-Skene model for crowdsourcing and advanced nonnegative factor analysis theories. This will establish firm theoretical foundations for crowdsourcing under critical conditions, and lead to theory-backed algorithms to attain substantially improved sample complexity and noise/incomplete data robustness. The second thrust exploits domain-dependent knowledge, e.g., data structure and annotator dependence, to come up with situation-aware crowdsourcing techniques for enhanced performance. The third thrust designs stochastic optimization strategies to provide scalable implementations for the CNMF framework, and evaluates the proposed methods over a variety of real-world applications. The analytical and computational tools developed in this project will provide strong provable guarantees and refreshing algorithmic solutions for long-standing challenges in crowdsourced data labeling. In addition, the CNMF theory and algorithms are exciting new directions for computational linear algebra, whose impacts can go well beyond this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.