III: Small: Collaborative Research: Algorithms, systems, and theories for exploiting data dependencies in crowdsourcing

在许多领域，如生物医学研究、在线商务、开放政府、教育和公共卫生，都有大量数据可用于对知识进行编码。机器学习是一种强大的工具，可以从数据中发现新知识，帮助个人和组织做出明智的决策。然而，机器学习需要由人类注释的知识来引导，这些知识的获取成本很高，而且还包含人为错误。研究团队通过新颖的方法发现并利用数据中的依赖关系，在为机器学习提供关键知识时显著降低成本和噪音。研究成果，包括算法、系统和理论，都具有足够的通用性，可以使机器学习应用的许多领域受益。通过进行基础研究，该团队将为全国的STEM劳动力培养本科生和毕业生。研究人员将合作开发算法、系统和理论，以降低注释依赖数据时的成本和噪音，称为“结构化注释”，为机器学习提供监督知识。虽然依赖关系会使数据注释成本高昂且容易出错，但研究人员认为依赖关系是一种有用的归纳偏差，可用于选择性和准确的注释。特别是，研究小组提出了一个人在环系统，以帮助构建适当的概率图形模型来编码依赖关系。项目团队将上下文和多臂强盗与可扩展的图推理算法相结合，以降低标记成本。基于图形强盗，该团队在重复查询相同数据点的标签时解决了预算分配问题，以提高鲁棒性。通过嘈杂的人工注释，团队制定了优化问题和算法，以共同推断注释者的能力和数据的基础真值标签。从理论角度来看，该项目将推进众包环境下的主动学习，使其具有更真实的噪声分布，并分析结构化注释中的遗憾。该项目将产生数据集、算法和测试平台系统，不仅有利于核心机器学习研究社区，也有利于许多使用机器学习的领域。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data are abundantly available to encode knowledge in many domains, such as biomedical research, online commerce, open government, education, and public health. Machine learning is a powerful tool to discover novel knowledge from data and to help individuals and organizations make informed decisions. However, machine learning needs to be bootstrapped by human-annotated knowledge, which can be expensive to obtain and also contain human errors. The team of researchers discovers and exploits the dependencies in the data, via novel methodologies to significantly reduce the cost and noises when providing critical knowledge for machine learning. The research outputs, including algorithms, systems, and theories, are sufficiently generic to benefit many domains where machine learning is applicable. By conducting the fundamental research, the team will train undergraduates and graduates for the STEM workforce in the nation.The researchers will collaborate to develop algorithms, systems, and theories for reducing costs and noises when annotating dependent data, termed as “structured annotations”, to provide supervision knowledge for machine learning. While the dependencies can make data annotations costly and error-prone, the researchers view the dependencies as a useful inductive bias for selective and accurate annotations. In particular, the research team proposes a human-in-the-loop system to aid the construction of proper probabilistic graphical models to encode the dependencies. The project team combines contextual and multi-armed bandits with scalable graph inference algorithms to reduce labeling costs. Based on the graphical bandits, the team addresses the budget allocation when querying labels of the same data point repetitively for robustness. With noisy human annotations, the team formulates optimization problems and algorithms to jointly infer the annotator competences and the ground truth labels of the data. From the theoretical perspective, the project will advance the active learning in crowdsourcing settings with more realistic noise distributions and will analyze the regrets in structured annotations. The project will result in datasets, algorithms, and a testbed system that benefit not only the core machine learning research community but also many domains that use machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.