RI: Medium: Foundations of Self-Supervised Learning Through the Lens of Probabilistic Generative Models

现代机器学习模型的监督学习需要非常大的高质量标记数据集。标记数据需要非常昂贵的人工注释，对于资源不足的机器学习最终用户来说，这通常过于昂贵。从未标记数据中对机器学习模型进行无监督学习有望大大提高现代机器学习的可访问性和包容性。这种无监督学习的一个新兴范例是自监督学习（SSL），其中机器学习模型在可以自动生成标签的任务上进行训练。这种方法是BERT和DALL-E等高性能语言和图像机器学习模型的核心。然而，尽管它在不同领域的许多基准测试中都有承诺，但目前开发SSL方法的许多方法都是不透明和启发式的，并且评估依赖于性能指标的特别选择。这个项目的目标是建立SSL的科学和数学基础，从而改进它的实践。在该领域的一些早期工作中，SSL被用于加速涉及概率模型学习的任务。渐渐地，通过一系列可伸缩性的近似，SSL的输出不再严格地与概率模型参数联系在一起，并且目标转移到学习对下游任务“有用”的特征，即表示学习。然而，“有用”通常在数学上难以确定，因此通常不清楚（甚至从经验上，更不用说从理论上）这些方法从数据中了解到什么。目前，设计一个性能良好的SSL方法需要尝试许多任务和模型体系结构的组合，直到特定的一个在下游任务上提供良好的结果。这有两个缺点：(i)它需要大量的试错；（ii）在科学层面上，它并没有产生任何关于什么使特定任务/架构适合的理解，以及学习到的关于数据分布的特征捕获。该项目将通过分析可以通过自监督学习恢复的深度生成模型的各个方面，修复概率模型与通过自监督模型进行特征学习之间的断裂联系。此外，通过这个视角，我们建议理解自监督学习方法相对于其他概率模型学习方法的相对优势——统计和算法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Supervised learning of modern machine learning models requires very large high-quality labeled datasets. Labeling data requires very expensive human annotations, which is often too expensive for under-resourced end-users of machine learning. Unsupervised learning of machine learning models from unlabeled data has the promise to vastly increase the accessibility and inclusivity of modern machine learning. An emerging paradigm for such unsupervised learning is self-supervised learning (SSL), wherein a machine learning model is trained on tasks for which labels can be automatically generated. This approach is at the core of high-performing language and image machine learning models like BERT and DALL-E. However, despite its promise on many benchmarks across diverse domains, a lot of current methodology for developing SSL methods is opaque and heuristic, and evaluation relies on ad-hoc choices of performance metrics. The goal of this project is to build scientific and mathematical foundations of SSL, and consequently also improve its practice. In some of the earliest work in this area, SSL was used to speed up tasks involving the learning of probabilistic models. Progressively, via a series of approximations for scalability, the outputs of SSL could no longer be rigorously tied to probabilistic model parameters, and the goal shifted to learning features that are "useful" for downstream tasks, that is representation learning. "Useful" however can often be mathematically difficult to pin down, so it is frequently not clear (even empirically, much less theoretically) what these methods learn about the data. At present, designing a well-performing SSL method entails trying many combinations of tasks and model architectures, until a particular one gives good results on the downstream tasks. This has two downsides: (i) it requires a substantial amount of trial-and-error; (ii) on a scientific level, it doesn't yield any understanding of what makes a particular task/architecture suitable, and what the features learned capture about the data distribution. This project will repair the severed tie between probabilistic models and feature learning via self-supervised models by analyzing the aspects of a deep generative model that can be recovered via self-supervised learning. Moreover, through this lens, we propose to understand the relative advantages---both statistical and algorithmic---of self-supervised learning methods over other methods for learning probabilistic models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.