RI: Medium: Foundations of Self-Supervised Learning Through the Lens of Probabilistic Generative Models

RI:媒介:通过概率生成模型的视角进行自我监督学习的基础

基本信息

  • 批准号:
    2211907
  • 负责人:
  • 金额:
    $ 112.79万
  • 依托单位:
  • 依托单位国家:
    美国
  • 项目类别:
    Standard Grant
  • 财政年份:
    2022
  • 资助国家:
    美国
  • 起止时间:
    2022-10-01 至 2026-09-30
  • 项目状态:
    未结题

项目摘要

Supervised learning of modern machine learning models requires very large high-quality labeled datasets. Labeling data requires very expensive human annotations, which is often too expensive for under-resourced end-users of machine learning. Unsupervised learning of machine learning models from unlabeled data has the promise to vastly increase the accessibility and inclusivity of modern machine learning. An emerging paradigm for such unsupervised learning is self-supervised learning (SSL), wherein a machine learning model is trained on tasks for which labels can be automatically generated. This approach is at the core of high-performing language and image machine learning models like BERT and DALL-E. However, despite its promise on many benchmarks across diverse domains, a lot of current methodology for developing SSL methods is opaque and heuristic, and evaluation relies on ad-hoc choices of performance metrics. The goal of this project is to build scientific and mathematical foundations of SSL, and consequently also improve its practice. In some of the earliest work in this area, SSL was used to speed up tasks involving the learning of probabilistic models. Progressively, via a series of approximations for scalability, the outputs of SSL could no longer be rigorously tied to probabilistic model parameters, and the goal shifted to learning features that are "useful" for downstream tasks, that is representation learning. "Useful" however can often be mathematically difficult to pin down, so it is frequently not clear (even empirically, much less theoretically) what these methods learn about the data. At present, designing a well-performing SSL method entails trying many combinations of tasks and model architectures, until a particular one gives good results on the downstream tasks. This has two downsides: (i) it requires a substantial amount of trial-and-error; (ii) on a scientific level, it doesn't yield any understanding of what makes a particular task/architecture suitable, and what the features learned capture about the data distribution. This project will repair the severed tie between probabilistic models and feature learning via self-supervised models by analyzing the aspects of a deep generative model that can be recovered via self-supervised learning. Moreover, through this lens, we propose to understand the relative advantages---both statistical and algorithmic---of self-supervised learning methods over other methods for learning probabilistic models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
现代机器学习模型的监督学习需要非常大的高质量标签数据集。标记数据需要非常昂贵的人工注释,对于资源不足的机器学习终端用户来说,这往往过于昂贵。从未标记的数据中进行机器学习模型的无监督学习有望极大地提高现代机器学习的可及性和包容性。这种无监督学习的一个新兴范例是自我监督学习(SSL),其中机器学习模型针对可以自动生成标签的任务进行训练。这种方法是BERT和DALL-E等高性能语言和图像机器学习模型的核心。然而,尽管它承诺在不同领域的许多基准测试中使用,但当前许多用于开发SSL方法的方法是不透明的和启发式的,并且评估依赖于对性能度量的临时选择。该项目的目标是建立科学和数学基础,并因此改进其实践。在这一领域最早的一些工作中,使用了SSL来加速涉及概率模型学习的任务。逐步地,通过一系列可伸缩性的近似,SSL的输出不再严格地绑定到概率模型参数,目标转移到对下游任务“有用”的学习特征,即表示学习。然而,“有用的”往往很难在数学上确定,因此通常不清楚(即使是从经验上,更不用说从理论上)这些方法从数据中学到了什么。目前,设计性能良好的SSL方法需要尝试许多任务和模型体系结构的组合,直到某个特定的组合在下游任务上获得良好的结果。这有两个缺点:(I)它需要大量的试错;(Ii)在科学层面上,它不会产生任何关于特定任务/体系结构适合的理解,以及了解到的关于数据分布的特性。这个项目将通过分析可以通过自我监督学习恢复的深度生成模型的各个方面,修复概率模型和通过自我监督模型进行特征学习之间的断绝联系。此外,通过这个镜头,我们建议理解自我监督学习方法相对于其他学习概率模型的方法的相对优势-统计和算法。这一奖项反映了NSF的法定使命,并通过使用基金会的智力优势和更广泛的影响审查标准进行评估,被认为值得支持。

项目成果

期刊论文数量(7)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
Identifiability of deep generative models without auxiliary information
  • DOI:
  • 发表时间:
    2022-06
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Bohdan Kivva;Goutham Rajendran;Pradeep Ravikumar;Bryon Aragam
  • 通讯作者:
    Bohdan Kivva;Goutham Rajendran;Pradeep Ravikumar;Bryon Aragam
Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments
  • DOI:
  • 发表时间:
    2021-06
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Yining Chen;Elan Rosenfeld;Mark Sellke;Tengyu Ma;Andrej Risteski
  • 通讯作者:
    Yining Chen;Elan Rosenfeld;Mark Sellke;Tengyu Ma;Andrej Risteski
Masked Prediction: A Parameter Identifiability View
  • DOI:
  • 发表时间:
    2022
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Bingbin Liu;Daniel J. Hsu;Pradeep Ravikumar;Andrej Risteski
  • 通讯作者:
    Bingbin Liu;Daniel J. Hsu;Pradeep Ravikumar;Andrej Risteski
Continual learning: a feature extraction formalization, an efficient algorithm, and barriers
  • DOI:
  • 发表时间:
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Binghui Peng;Andrej Risteski
  • 通讯作者:
    Binghui Peng;Andrej Risteski
Concept Gradient: Concept-based Interpretation Without Linear Assumption
  • DOI:
    10.48550/arxiv.2208.14966
  • 发表时间:
    2022-08
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Andrew Bai;Chih-Kuan Yeh;Pradeep Ravikumar;Neil Y. C. Lin;Cho-Jui Hsieh
  • 通讯作者:
    Andrew Bai;Chih-Kuan Yeh;Pradeep Ravikumar;Neil Y. C. Lin;Cho-Jui Hsieh
{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ monograph.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ sciAawards.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ conferencePapers.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ patent.updateTime }}

Pradeep Ravikumar其他文献

Ordinal Graphical Models: A Tale of Two Approaches
序数图形模型:两种方法的故事
  • DOI:
    10.5555/3305890.3306018
  • 发表时间:
    2017
  • 期刊:
  • 影响因子:
    0
  • 作者:
    A. Suggala;Eunho Yang;Pradeep Ravikumar
  • 通讯作者:
    Pradeep Ravikumar
XMRF: an R package to fit Markov Networks to high-throughput genetics data
XMRF:一个 R 包,用于使马尔可夫网络适应高通量遗传学数据
  • DOI:
  • 发表时间:
    2015
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Ying;Genevera I. Allen;Yulia Baker;Eunho Yang;Pradeep Ravikumar;Zhandong Liu
  • 通讯作者:
    Zhandong Liu
Nonparametric sparse hierarchical models describe V1 fMRI responses to natural images
非参数稀疏分层模型描述 V1 fMRI 对自然图像的响应
  • DOI:
  • 发表时间:
    2008
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Pradeep Ravikumar;Vincent Q. Vu;Bin Yu;Thomas Naselaris;Kendrick Norris Kay;J. Gallant
  • 通讯作者:
    J. Gallant
Deep Density Destructors
深度密度破坏函数
Predicting Growth Conditions from Internal Metabolic Fluxes in an In-Silico Model of E. coli
根据大肠杆菌的计算机模型中的内部代谢通量预测生长条件
  • DOI:
  • 发表时间:
    2014
  • 期刊:
  • 影响因子:
    0
  • 作者:
    V. Sridhara;A. Meyer;Piyush Rai;Jeffrey E. Barrick;Pradeep Ravikumar;D. Segrè;C. Wilke
  • 通讯作者:
    C. Wilke

Pradeep Ravikumar的其他文献

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

{{ truncateString('Pradeep Ravikumar', 18)}}的其他基金

Collaborative Research: RI: Medium: A Rigorous, General Framework for Tractable Learning of Large-Scale DAGs from Data
协作研究:RI:Medium:从数据中轻松学习大规模 DAG 的严格通用框架
  • 批准号:
    1955532
  • 财政年份:
    2020
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
RI: Small: Non-parametric Machine Learning in the Age of Deep and High-Dimensional Models
RI:小:深度和高维模型时代的非参数机器学习
  • 批准号:
    1909816
  • 财政年份:
    2019
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Standard Grant
Collaborative Research: Physics-Based Machine Learning for Sub-Seasonal Climate Forecasting
合作研究:基于物理的机器学习用于次季节气候预测
  • 批准号:
    1934584
  • 财政年份:
    2019
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
CAREER: A New Neat Framework for Statistical Machine Learning
职业:统计机器学习的新简洁框架
  • 批准号:
    1661755
  • 财政年份:
    2016
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
BIGDATA: F: DKA: Collaborative Research: High-Dimensional Statistical Machine Learning for Spatio-Temporal Climate Data
BIGDATA:F:DKA:协作研究:时空气候数据的高维统计机器学习
  • 批准号:
    1664720
  • 财政年份:
    2016
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Standard Grant
Collaborative Research: Statistical Methods for Integrated Analysis of High-Throughput Biomedical Data
合作研究:高通量生物医学数据综合分析的统计方法
  • 批准号:
    1661802
  • 财政年份:
    2016
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
BIGDATA: F: DKA: Collaborative Research: High-Dimensional Statistical Machine Learning for Spatio-Temporal Climate Data
BIGDATA:F:DKA:协作研究:时空气候数据的高维统计机器学习
  • 批准号:
    1447574
  • 财政年份:
    2014
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Standard Grant
Collaborative Research: Statistical Methods for Integrated Analysis of High-Throughput Biomedical Data
合作研究:高通量生物医学数据综合分析的统计方法
  • 批准号:
    1264033
  • 财政年份:
    2013
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
RI: Small: Collaborative Research: Statistical ranking theory without a canonical loss
RI:小:协作研究:没有典型损失的统计排名理论
  • 批准号:
    1320894
  • 财政年份:
    2013
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Standard Grant
CAREER: A New Neat Framework for Statistical Machine Learning
职业:统计机器学习的新简洁框架
  • 批准号:
    1149803
  • 财政年份:
    2012
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant

相似海外基金

Collaborative Research: AF: Medium: Foundations of Oblivious Reconfigurable Networks
合作研究:AF:媒介:遗忘可重构网络的基础
  • 批准号:
    2402851
  • 财政年份:
    2024
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
Collaborative Research: AF: Medium: Foundations of Oblivious Reconfigurable Networks
合作研究:AF:媒介:遗忘可重构网络的基础
  • 批准号:
    2402852
  • 财政年份:
    2024
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
Collaborative Research: AF: Medium: Foundations of Anonymous Communication in Large-Scale Networks
合作研究:AF:媒介:大规模网络中匿名通信的基础
  • 批准号:
    2312241
  • 财政年份:
    2023
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Distributionally Robust Policy Learning
合作研究:CIF:媒介:分布式稳健政策学习的统计和算法基础
  • 批准号:
    2312205
  • 财政年份:
    2023
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
Collaborative Research: AF: SaTC: Medium: Theoretical Foundations of Lattice-Based Cryptography
合作研究:AF:SaTC:媒介:基于格的密码学的理论基础
  • 批准号:
    2312296
  • 财政年份:
    2023
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
Collaborative Research: AF: Medium: Foundations of Anonymous Communication in Large-Scale Networks
合作研究:AF:媒介:大规模网络中匿名通信的基础
  • 批准号:
    2312242
  • 财政年份:
    2023
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
NeTS: Medium: Foundations and Applications of Modular Verification of Networks
NeTS:媒介:网络模块化验证的基础和应用
  • 批准号:
    2312539
  • 财政年份:
    2023
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
RI: Medium: Foundations of Recourse Verification in Machine Learning
RI:媒介:机器学习资源验证的基础
  • 批准号:
    2313105
  • 财政年份:
    2023
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Standard Grant
Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Distributionally Robust Policy Learning
合作研究:CIF:媒介:分布式稳健政策学习的统计和算法基础
  • 批准号:
    2312204
  • 财政年份:
    2023
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
Collaborative Research: AF: Medium: Foundations of Anonymous Communication in Large-Scale Networks
合作研究:AF:媒介:大规模网络中匿名通信的基础
  • 批准号:
    2312243
  • 财政年份:
    2023
  • 资助金额:
    $ 112.79万
  • 项目类别:
    Continuing Grant
{{ showInfoDetail.title }}

作者:{{ showInfoDetail.author }}

知道了