CCF: CIF: Small: Interactive Learning from Noisy, Heterogeneous Feedback

CCF：CIF：小型：从嘈杂、异构的反馈中进行交互式学习

• 批准号: 1719133
• 负责人: Kamalika Chaudhuri
• 金额: $ 50万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-07-15 至 2021-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1719133&HistoricalAwards=false
• 关键词: CCF; CIF; Small; Interactive; Learning

The goal of this project is to develop interactive learning frameworks and methods that can learn predictors based on complex, imperfect feedback adaptively solicited in an on-line fashion from human annotators. Such predictors can significantly benefit the practice of machine learning by making it more accessible in domains where annotations are expensive. Currently, beyond a handful of heuristic studies, the only well-understood interactive learning setting is active binary classification, where a single annotator interactively provides labels to a learning algorithm. The main challenge in exploiting richer feedback is that human responses are inherently inconsistent and imperfect. This project will overcome this challenge by assuming that the responses come from unknown probability distributions with some mild yet realistic properties, which will be exploited to provide methods that can learn reliably from complex feedback.Specifically, this project will introduce a general framework for interactive learning from imperfect, complex feedback, and develop methods for three common cases: (1) Active Learning with Abstention Feedback, where annotators can either provide a label or declare I Don't Know (2) Active Learning for Multiclass Classification, where the goal is to learn a classifier for a large number of classes and (3) Active Learning with Feedback from Multiple Annotators, where the goal is to combine feedback from many labelers with varying amounts of expertise subject to a budget. These problems will be approached through two main tools -- adaptive hypothesis testing and surrogate loss minimization. Combining these approaches will lead to principled algorithms for building accurate machine learning predictors with low annotation cost, which in turn, will benefit the practice of machine learning in domains where annotated data is expensive.

这个项目的目标是开发交互式学习框架和方法，可以学习预测的基础上，复杂的，不完善的反馈自适应地征求在一个在线的方式从人类注释。这样的预测器可以使机器学习在注释昂贵的领域中更容易获得，从而大大有益于机器学习的实践。目前，除了少数启发式研究之外，唯一一个很好理解的交互式学习设置是主动二进制分类，其中单个注释器交互式地为学习算法提供标签。利用更丰富的反馈的主要挑战是，人类的反应本质上是不一致和不完美的。该项目将通过假设响应来自具有一些温和但现实属性的未知概率分布来克服这一挑战，这将被利用来提供可以从复杂反馈中可靠学习的方法。具体来说，该项目将介绍一个从不完美的复杂反馈中进行交互式学习的一般框架，并为三种常见情况开发方法：（1）主动学习与弃权反馈，其中注释器可以提供标签或声明我不知道（2）多类分类的主动学习，其中目标是学习大量类的分类器和（3）主动学习与来自多个注释器的反馈，其中目标是将来自许多贴标机的反馈与受制于预算的不同量的专业知识结合联合收割机。这些问题将通过两个主要工具-自适应假设检验和替代损失最小化。结合这些方法将产生用于构建具有低注释成本的准确机器学习预测器的原则性算法，这反过来将有利于在注释数据昂贵的领域中进行机器学习。

项目成果

Active Learning for Classification With Abstention

主动学习弃权分类

• DOI: 10.1109/jsait.2021.3081433
• 发表时间: 2021
• 期刊: IEEE Journal on Selected Areas in Information Theory
• 作者: Shekhar, Shubhanshu;Ghavamzadeh, Mohammad;Javidi, Tara
• 通讯作者: Javidi, Tara

Active Learning with Logged Data

• 发表时间: 2018-02
• 期刊: ArXiv
• 作者: Songbai Yan;Kamalika Chaudhuri;T. Javidi

Multiscale Gaussian Process Level Set Estimation

多尺度高斯过程水平集估计

• 发表时间: 2019
• 期刊: Proceedings of Machine Learning Research
• 作者: Shekhar, Subhanshu;Javidi, Tara
• 通讯作者: Javidi, Tara

The Label Complexity of Active Learning from Observational Data

• 发表时间: 2019-05
• 期刊: ArXiv
• 作者: Songbai Yan;Kamalika Chaudhuri;T. Javidi

Exploring Connections Between Active Learning and Model Extraction

• 发表时间: 2018-11
• 作者: Varun Chandrasekaran;Kamalika Chaudhuri;Irene Giacomelli;S. Jha;Songbai Yan