CCF: CIF: Small: Interactive Learning from Noisy, Heterogeneous Feedback

该项目的目标是开发交互式学习框架和方法，这些框架和方法可以根据在线方式自适应地从人类注释者那里征求的复杂、不完美的反馈来学习预测器。这样的预测器可以使机器学习在注释昂贵的领域中更容易获得，从而大大有利于机器学习的实践。目前，除了少数启发式研究之外，唯一被充分理解的交互式学习设置是主动二元分类，其中单个注释器交互式地为学习算法提供标签。利用更丰富的反馈的主要挑战是，人类的反应本质上是不一致和不完美的。该项目将克服这一挑战，假设响应来自未知的概率分布，具有一些温和但现实的属性，这将被用于提供可以从复杂反馈中可靠地学习的方法。具体来说，本项目将引入一个从不完美、复杂的反馈中进行互动学习的一般框架，并针对三种常见情况开发方法：(1)带有弃权反馈的主动学习，注释者可以提供标签或声明“我不知道”(2)针对多类分类的主动学习，其目标是为大量类学习分类器；(3)带有多个注释者反馈的主动学习，其目标是将来自具有不同专业知识的许多标注者的反馈结合在一起，并受到预算的限制。这些问题将通过两个主要工具来解决——适应性假设检验和代理损失最小化。结合这些方法将导致有原则的算法，以低注释成本构建准确的机器学习预测器，这反过来将有利于机器学习在注释数据昂贵的领域的实践。

The goal of this project is to develop interactive learning frameworks and methods that can learn predictors based on complex, imperfect feedback adaptively solicited in an on-line fashion from human annotators. Such predictors can significantly benefit the practice of machine learning by making it more accessible in domains where annotations are expensive. Currently, beyond a handful of heuristic studies, the only well-understood interactive learning setting is active binary classification, where a single annotator interactively provides labels to a learning algorithm. The main challenge in exploiting richer feedback is that human responses are inherently inconsistent and imperfect. This project will overcome this challenge by assuming that the responses come from unknown probability distributions with some mild yet realistic properties, which will be exploited to provide methods that can learn reliably from complex feedback.Specifically, this project will introduce a general framework for interactive learning from imperfect, complex feedback, and develop methods for three common cases: (1) Active Learning with Abstention Feedback, where annotators can either provide a label or declare I Don't Know (2) Active Learning for Multiclass Classification, where the goal is to learn a classifier for a large number of classes and (3) Active Learning with Feedback from Multiple Annotators, where the goal is to combine feedback from many labelers with varying amounts of expertise subject to a budget. These problems will be approached through two main tools -- adaptive hypothesis testing and surrogate loss minimization. Combining these approaches will lead to principled algorithms for building accurate machine learning predictors with low annotation cost, which in turn, will benefit the practice of machine learning in domains where annotated data is expensive.