Yielding and Exploiting Confidence in Deep Learning

点击翻译按钮获取中文摘要

The scale of research and application of deep learning continues to accelerate. In our work and our daily lives, we see more dependence on these systems for making predictions. In some cases, they are automated, and in others, they involve humans "in-the-loop". Unfortunately, these systems can fail silently. Examples of failure include erroneous yet highly confident predictions, as well as susceptibility to adversarial attacks and so-called fooling images. One way to build more fault-tolerant machine learning systems is to calibrate their confidence or uncertainty measures for interpretation by humans or other systems. Such measures are useful, for example, to refer data points to a human expert or another system for further processing. Proposing different techniques for estimating confidence or uncertainty is an active area of research. Most approaches aim to either estimate or re-calibrate a confidence measure after training; however, there are many uses for confidence throughout learning. Examples include choosing additional data points for annotation (active learning), selecting which examples to visit next (self-paced learning), and learning with a reject option. I propose a research program that will focus on novel and efficient ways of expressing, quantifying, and channeling confidence, both in training and deploying machine learning systems. The long-term goal is toward systems with more responsibility in their autonomy and improved collaboration with humans. It will span three activities. Modulating the learning experience We will compare and develop different confidence measures to: 1) enable a system to optimally ask for "hints" from an oracle; 2) modulate the type and amount of data augmentation applied to an input; and 3) choose among auxiliary tasks. Our goal is to learn faster with less human-labeled examples. Confidence and conditional computation We will investigate dynamic network architectures whose processing depends on the input through confidence estimates. For example, during training, a simple example could require very little computation, a moderate example could require more computation, and a difficult example could be ignored and revisited at a future epoch. Our goal is to optimize computation during training and deployment. Evaluation and cross-discipline collaboration In reporting performance, typical machine learning benchmarks do not consider that some tasks are more costly than others. Moreover, confidence estimates are rarely evaluated directly: they use proxies like out-of-distribution or adversarial example detection. We will work with neuroscientists to assess methods for quantifying confidence in human decision making and gain a better understanding of its role in human learning. We will develop a standard benchmark for the use of confidence in machine learning. This research program will develop more robust, efficient, and transparent deep learning models designed to improve life.