III: Small: Collaborative Research: Algorithms, systems, and theories for exploiting data dependencies in crowdsourcing

在许多领域，如生物医学研究、在线商务、开放式政府、教育和公共卫生，都有大量的数据可用于编码知识。机器学习是从数据中发现新知识并帮助个人和组织做出明智决策的强大工具。然而，机器学习需要由人类注释的知识引导，这可能是昂贵的获得，也包含人为错误。研究团队发现并利用数据中的依赖关系，通过新颖的方法，在为机器学习提供关键知识时显着降低成本和噪音。研究成果，包括算法、系统和理论，都是足够通用的，可以使机器学习适用的许多领域受益。通过进行基础研究，该团队将为全国的STEM劳动力培养本科生和研究生。研究人员将合作开发算法，系统和理论，以减少注释依赖数据时的成本和噪音，称为“结构化注释”，为机器学习提供监督知识。虽然依赖关系可能会使数据注释成本高昂且容易出错，但研究人员认为依赖关系是选择性和准确注释的有用归纳偏差。特别是，研究小组提出了一个人在回路系统，以帮助构建适当的概率图形模型来编码依赖关系。该项目团队将上下文和多臂强盗与可扩展的图推理算法相结合，以降低标记成本。基于图形强盗，该团队在重复查询同一数据点的标签时解决了预算分配问题，以确保鲁棒性。通过嘈杂的人类注释，该团队制定了优化问题和算法，以共同推断注释者的能力和数据的地面真实标签。从理论的角度来看，该项目将在具有更真实的噪声分布的众包环境中推进主动学习，并将分析结构化注释中的遗憾。该项目将产生数据集、算法和测试平台系统，不仅有利于核心机器学习研究社区，也有利于许多使用机器学习的领域。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data are abundantly available to encode knowledge in many domains, such as biomedical research, online commerce, open government, education, and public health. Machine learning is a powerful tool to discover novel knowledge from data and to help individuals and organizations make informed decisions. However, machine learning needs to be bootstrapped by human-annotated knowledge, which can be expensive to obtain and also contain human errors. The team of researchers discovers and exploits the dependencies in the data, via novel methodologies to significantly reduce the cost and noises when providing critical knowledge for machine learning. The research outputs, including algorithms, systems, and theories, are sufficiently generic to benefit many domains where machine learning is applicable. By conducting the fundamental research, the team will train undergraduates and graduates for the STEM workforce in the nation.The researchers will collaborate to develop algorithms, systems, and theories for reducing costs and noises when annotating dependent data, termed as “structured annotations”, to provide supervision knowledge for machine learning. While the dependencies can make data annotations costly and error-prone, the researchers view the dependencies as a useful inductive bias for selective and accurate annotations. In particular, the research team proposes a human-in-the-loop system to aid the construction of proper probabilistic graphical models to encode the dependencies. The project team combines contextual and multi-armed bandits with scalable graph inference algorithms to reduce labeling costs. Based on the graphical bandits, the team addresses the budget allocation when querying labels of the same data point repetitively for robustness. With noisy human annotations, the team formulates optimization problems and algorithms to jointly infer the annotator competences and the ground truth labels of the data. From the theoretical perspective, the project will advance the active learning in crowdsourcing settings with more realistic noise distributions and will analyze the regrets in structured annotations. The project will result in datasets, algorithms, and a testbed system that benefit not only the core machine learning research community but also many domains that use machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.