III: Small: Collaborative Research: Algorithms, systems, and theories for exploiting data dependencies in crowdsourcing

大量的数据可用于编码许多领域的知识，如生物医学研究、在线商务、开放政府、教育和公共卫生。机器学习是从数据中发现新知识并帮助个人和组织做出明智决策的强大工具。然而，机器学习需要由人类注释的知识来引导，这些知识获取起来可能很昂贵，而且还包含人为错误。研究团队通过新的方法发现并利用数据中的依赖关系，以显著降低为机器学习提供关键知识时的成本和噪音。包括算法、系统和理论在内的研究成果具有足够的通用性，使许多适用于机器学习的领域受益。通过开展基础研究，该团队将为全国的STEM劳动力培训本科生和毕业生。研究人员将合作开发算法、系统和理论，以降低标注依赖数据时的成本和噪声，称为“结构化标注”，为机器学习提供监督知识。虽然依赖关系可能会使数据注释成本高昂且容易出错，但研究人员将这些依赖关系视为对选择性和准确注释有用的归纳偏向。特别是，研究团队提出了一个人在回路中的系统，以帮助构建适当的概率图形模型来编码依赖关系。该项目团队将上下文和多武装强盗与可伸缩的图形推理算法相结合，以降低标签成本。基于图形强盗，该团队在重复查询相同数据点的标签时解决预算分配问题，以实现健壮性。通过嘈杂的人工注释，团队制定了优化问题和算法，以联合推断注释员的能力和数据的基本事实标签。从理论上看，该项目将以更真实的噪声分布促进众包环境中的主动学习，并将分析结构化注释中的遗憾。该项目将产生数据集、算法和试验台系统，不仅使核心机器学习研究社区受益，还将使许多使用机器学习的领域受益。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Data are abundantly available to encode knowledge in many domains, such as biomedical research, online commerce, open government, education, and public health. Machine learning is a powerful tool to discover novel knowledge from data and to help individuals and organizations make informed decisions. However, machine learning needs to be bootstrapped by human-annotated knowledge, which can be expensive to obtain and also contain human errors. The team of researchers discovers and exploits the dependencies in the data, via novel methodologies to significantly reduce the cost and noises when providing critical knowledge for machine learning. The research outputs, including algorithms, systems, and theories, are sufficiently generic to benefit many domains where machine learning is applicable. By conducting the fundamental research, the team will train undergraduates and graduates for the STEM workforce in the nation.The researchers will collaborate to develop algorithms, systems, and theories for reducing costs and noises when annotating dependent data, termed as “structured annotations”, to provide supervision knowledge for machine learning. While the dependencies can make data annotations costly and error-prone, the researchers view the dependencies as a useful inductive bias for selective and accurate annotations. In particular, the research team proposes a human-in-the-loop system to aid the construction of proper probabilistic graphical models to encode the dependencies. The project team combines contextual and multi-armed bandits with scalable graph inference algorithms to reduce labeling costs. Based on the graphical bandits, the team addresses the budget allocation when querying labels of the same data point repetitively for robustness. With noisy human annotations, the team formulates optimization problems and algorithms to jointly infer the annotator competences and the ground truth labels of the data. From the theoretical perspective, the project will advance the active learning in crowdsourcing settings with more realistic noise distributions and will analyze the regrets in structured annotations. The project will result in datasets, algorithms, and a testbed system that benefit not only the core machine learning research community but also many domains that use machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.