RI: Small: Robustness and Confidence in Machine-Learned Systems

机器学习在从医学到移动数据收集平台等领域的应用前景广阔。然而，在一个不断变化的世界里，随着数据来自更多不同的来源，人们怎么能相信机器学习的系统不只是符合他们观察到的一些奇怪的特性呢？这个项目开发了机器学习的方法，这样这样的系统就不会脆弱，不会对收集的数据中的微小变化敏感，也不会在罕见的种群中犯下关键错误。随着数据分析在科学、工业和医疗保健中的重要性与日俱增，稳健性、安全性和校准的原则性和实用性方法产生了立竿见影的广泛影响。该项目的一个主要目标是为决策者提供来自机器学习模型的可信预测。第二个目标是教学上的：随着机器学习的迅速崛起，我们错过了教育学生、研究人员和工程师使他们能够实际构建可信系统的机会；该项目旨在围绕这些挑战制定课程。该项目开发稳健的学习过程，努力建立可信的机器学习。三个混凝土突起支撑着这项工作。第一个建立在研究人员在分布稳健性方面的工作基础上，分布稳健性对模型进行拟合，以最大限度地提高接近可用数据的种群的性能。第二是创造性地和正确地使用数据；这需要使用数据来定义稳健性，理解方法敏感性，使用未标记(廉价)的数据来建立更健壮的表示，并构建基于数据的正则化。第三个目标是信心和校准，建立提供无假设有效预测的模型。在这种情况下，目标是寻找具有校准置信度的预测者，建立在保角预测的基础上，这是现代学习方法强调不提供的。更广泛地说，分布变化挑战了统计机器学习方法，该项目旨在开发新的验证和测试方法，以了解这种变化，识别方法对基础数据变化敏感的情况，并即使在不断变化的环境中也允许对预测保持有效的信心。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The application of machine learning, in fields from medicine to mobile data gathering platforms, has substantial promise. Yet as data comes from a greater variety of sources in an ever-shifting world, how can one trust that machine-learned systems have not simply fit some strange idiosyncrasies they observe? This project develops methods for machine learning so that such systems are not brittle, sensitive to tiny changes in collected data, or likely to make critical mistakes on rare populations. With the growing importance of data analysis in science, industry, and healthcare, principled and practical approaches to robustness, safety, and calibration have immediate and wide-ranging effects. A major goal of the project is to provide decision makers with trustworthy predictions from machine-learned models. A second goal is pedagogical: with the meteoric rise of machine learning, there is a missed opportunity to educate students, researchers, and engineers to give them the ability to actually build trustworthy systems; this project aims toward a curriculum around such challenges.This project develops robust learning procedures in effort to build trustable machine learning. Three concrete thrusts underpin the work. The first builds off of the investigator's work in distributional robustness, which fits models to maximize performance on populations near enough to available data. The second is to use data creatively and correctly; this entails using the data to define robustness, understand method sensitivities, use unlabeled (cheap) data to build more robust representations, and construct data-based regularization. The third targets confidence and calibration, building models that provide assumption-free valid predictions. In this case, the aim is to seek predictors with calibrated confidence, building out of conformal prediction, which modern learning methods emphatically do not provide. More generally, distributional shifts challenge statistical machine learning methods, and the project aims for new validation and testing methodologies to understand such shifts, identify situations where methods are sensitive to changes in underlying data, and to allow valid confidence in predictions even in changing environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.