RI: Medium: Foundations of Self-Supervised Learning Through the Lens of Probabilistic Generative Models

RI：媒介：通过概率生成模型的视角进行自我监督学习的基础

• 批准号: 2211907
• 负责人: Pradeep Ravikumar
• 金额: $ 112.79万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211907&HistoricalAwards=false
• 关键词: RI; Medium; Foundations; Self; Supervised

Supervised learning of modern machine learning models requires very large high-quality labeled datasets. Labeling data requires very expensive human annotations, which is often too expensive for under-resourced end-users of machine learning. Unsupervised learning of machine learning models from unlabeled data has the promise to vastly increase the accessibility and inclusivity of modern machine learning. An emerging paradigm for such unsupervised learning is self-supervised learning (SSL), wherein a machine learning model is trained on tasks for which labels can be automatically generated. This approach is at the core of high-performing language and image machine learning models like BERT and DALL-E. However, despite its promise on many benchmarks across diverse domains, a lot of current methodology for developing SSL methods is opaque and heuristic, and evaluation relies on ad-hoc choices of performance metrics. The goal of this project is to build scientific and mathematical foundations of SSL, and consequently also improve its practice. In some of the earliest work in this area, SSL was used to speed up tasks involving the learning of probabilistic models. Progressively, via a series of approximations for scalability, the outputs of SSL could no longer be rigorously tied to probabilistic model parameters, and the goal shifted to learning features that are "useful" for downstream tasks, that is representation learning. "Useful" however can often be mathematically difficult to pin down, so it is frequently not clear (even empirically, much less theoretically) what these methods learn about the data. At present, designing a well-performing SSL method entails trying many combinations of tasks and model architectures, until a particular one gives good results on the downstream tasks. This has two downsides: (i) it requires a substantial amount of trial-and-error; (ii) on a scientific level, it doesn't yield any understanding of what makes a particular task/architecture suitable, and what the features learned capture about the data distribution. This project will repair the severed tie between probabilistic models and feature learning via self-supervised models by analyzing the aspects of a deep generative model that can be recovered via self-supervised learning. Moreover, through this lens, we propose to understand the relative advantages---both statistical and algorithmic---of self-supervised learning methods over other methods for learning probabilistic models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代机器学习模型的监督学习需要非常大的高质量标签数据集。标记数据需要非常昂贵的人工注释，对于资源不足的机器学习终端用户来说，这往往过于昂贵。从未标记的数据中进行机器学习模型的无监督学习有望极大地提高现代机器学习的可及性和包容性。这种无监督学习的一个新兴范例是自我监督学习(SSL)，其中机器学习模型针对可以自动生成标签的任务进行训练。这种方法是BERT和DALL-E等高性能语言和图像机器学习模型的核心。然而，尽管它承诺在不同领域的许多基准测试中使用，但当前许多用于开发SSL方法的方法是不透明的和启发式的，并且评估依赖于对性能度量的临时选择。该项目的目标是建立科学和数学基础，并因此改进其实践。在这一领域最早的一些工作中，使用了SSL来加速涉及概率模型学习的任务。逐步地，通过一系列可伸缩性的近似，SSL的输出不再严格地绑定到概率模型参数，目标转移到对下游任务“有用”的学习特征，即表示学习。然而，“有用的”往往很难在数学上确定，因此通常不清楚(即使是从经验上，更不用说从理论上)这些方法从数据中学到了什么。目前，设计性能良好的SSL方法需要尝试许多任务和模型体系结构的组合，直到某个特定的组合在下游任务上获得良好的结果。这有两个缺点：(I)它需要大量的试错；(Ii)在科学层面上，它不会产生任何关于特定任务/体系结构适合的理解，以及了解到的关于数据分布的特性。这个项目将通过分析可以通过自我监督学习恢复的深度生成模型的各个方面，修复概率模型和通过自我监督模型进行特征学习之间的断绝联系。此外，通过这个镜头，我们建议理解自我监督学习方法相对于其他学习概率模型的方法的相对优势-统计和算法。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Identifiability of deep generative models without auxiliary information

• 发表时间: 2022-06
• 作者: Bohdan Kivva;Goutham Rajendran;Pradeep Ravikumar;Bryon Aragam

Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments

• 发表时间: 2021-06
• 期刊: ArXiv
• 作者: Yining Chen;Elan Rosenfeld;Mark Sellke;Tengyu Ma;Andrej Risteski

Masked Prediction: A Parameter Identifiability View

• 发表时间: 2022
• 作者: Bingbin Liu;Daniel J. Hsu;Pradeep Ravikumar;Andrej Risteski

Continual learning: a feature extraction formalization, an efficient algorithm, and barriers

• 作者: Binghui Peng;Andrej Risteski

Concept Gradient: Concept-based Interpretation Without Linear Assumption

• DOI: 10.48550/arxiv.2208.14966
• 发表时间: 2022-08
• 期刊: ArXiv
• 作者: Andrew Bai;Chih-Kuan Yeh;Pradeep Ravikumar;Neil Y. C. Lin;Cho-Jui Hsieh