RI: Medium: Quantifying and utilizing confidence in machine learning

该项目定义了预测置信度的有意义的概念，设计了计算这些概念的程序，并将这些程序应用于核心机器学习任务，如主动学习，众包学习和跟踪。在许多应用中，分类器与每个预测一起输出预测实际上是正确的置信度评级是有帮助的。现有的文献要么提供了各种特设的方式来计算这样的评级，通常缺乏严格的数学基础，或提供数学上一致的方法（在贝叶斯框架）计算置信度评级下非常强的假设，不太可能在实践中举行。该研究团队研究了计算置信度的方法，这些方法在数学上是严格的，同时对数据生成的方式做最小的假设，并使用这些方法进一步开发核心机器学习任务的解决方案。定义和计算数学上合理的置信度是机器学习，模式识别和人工智能不确定性的核心。置信度预测，主动学习和跟踪是机器学习和统计的基本任务，在大规模问题中反复出现;该项目将为这些问题制定严格的解决方案。在这项工作中开发的算法进行了测试，并用于自动摄像师项目，在UCSD计算机科学系的互动，视听装置。互动式自动摄影师系统是一种教育工具，可以由不同技能水平的学生团队向许多不同的方向扩展。

This project defines meaningful notions of confidence in prediction, designs procedures for computing such notions, and applies these procedures to core machine learning tasks such as active learning, crowd-sourced learning, and tracking. In many applications it is helpful to have classifiers that output, together with each prediction, a rating of the confidence that the prediction is in fact correct. Existing literature either provides various ad-hoc ways for computing such ratings which typically lack a rigorous mathematical footing, or provides mathematically consistent methods (in the Bayesian framework) for computing confidence ratings under very strong assumptions that are unlikely to hold in practice. The research team investigates methods of computing measures of confidence that are mathematically rigorous while making minimal assumptions on the way data is generated, and use these measures to further develop solutions to core machine learning tasks.Defining and computing mathematically sound measures of confidence lies at the heart of machine learning, pattern recognition and uncertainty in AI. Confidence-rated prediction, active learning, and tracking are fundamental tasks of machine learning and statistics that arise repeatedly in large-scale problems; this project will develop rigorous solutions to these problems. The algorithms developed in this work are tested and used in the Automatic Cameraman project, an interactive, audio-visual installation in the UCSD Computer Science department. The interactive Automatic Cameraman system are used an educational tool to be extended in many different directions, by teams of students at a variety of skill levels.