Collaborative Research: RI: Small: Theoretical Foundations: TheAdvantage of Deep Learning over Traditional Shallow Learning Methods

合作研究：RI：小型：理论基础：深度学习相对于传统浅层学习方法的优势

• 批准号: 2008559
• 负责人: Yingyu Liang
• 金额: $ 15.92万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008559&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Small; Theoretical

Machine learning has been a primary driving force behind many current intelligent decision-making systems. Recently it shows a paradigm shift with increasing reliance on deep learning approaches, which have achieved unprecedented performance in various applications such as image processing, speech recognition, language translation, and game playing. Besides the empirical success, provable guarantees and insights into the principles behind the success have also become sought-after goals. However, the lack of adequate understanding is still limiting our capacity to fully exploit the potential of deep learning. This project aims to lay the foundations for supporting the practical trends, by understanding the advantages of deep learning over traditional learning methods, which is crucial for revealing key factors behind the practical success. The project will provide frameworks for proving performance guarantees and advantages of deep learning over traditional learning methods and enable the development of new deep learning methods that are more efficient and accessible. This project will develop a thorough and systematic approach for understanding the superior empirical performance of deep learning over traditional learning methods and use the obtained insights to design new learning methods. It will develop new theoretical models of properties on the labeling function of the data and the structure of the input leading to the practical success, and also provides frameworks for proving performance guarantees and advantages over shallow learning. It will also design new learning methods that explicitly exploit those properties and thus can be more efficient and accessible. This direction is still largely unexplored, despite significant recent research activities. The planned theoretical and algorithmic solutions are possible through an interdisciplinary mix of tools from machine learning, statistics, and optimization. The proposed program is grounded in the investigators' prior work that includes both theoretical results and empirical validation. If successful, the proposed research can be transformational for modern intelligent systems by laying the foundations for further development. It will also help to solve new theoretical problems from practice that are not adequately addressed by current theory and will have lasting impacts on machine learning and optimization.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习一直是当前许多智能决策系统背后的主要驱动力。最近，随着对深度学习方法的依赖越来越大，它显示出一种范式转变，这些方法在图像处理、语音识别、语言翻译和游戏等各种应用中取得了前所未有的性能。除了经验上的成功，可证明的保证和对成功背后原则的洞察也成为人们追求的目标。然而，缺乏足够的理解仍然限制了我们充分利用深度学习潜力的能力。该项目旨在通过了解深度学习相对于传统学习方法的优势，为支持实践趋势奠定基础，这对于揭示实践成功背后的关键因素至关重要。该项目将提供框架，以证明深度学习相对于传统学习方法的性能保证和优势，并能够开发更有效和更易于访问的新深度学习方法。 该项目将开发一种全面和系统的方法，用于理解深度学习相对于传统学习方法的上级经验性能，并使用所获得的见解来设计新的学习方法。它将开发新的理论模型的属性上的标签功能的数据和结构的输入导致实际的成功，并提供了框架，证明性能保证和优势比浅学习。它还将设计新的学习方法，明确利用这些属性，从而可以更有效和更容易获得。这一方向在很大程度上仍然是未探索的，尽管最近的研究活动显着。计划的理论和算法解决方案可以通过机器学习，统计和优化的跨学科工具组合来实现。所提出的方案是基于调查人员的先前工作，包括理论结果和实证验证。如果成功，拟议的研究可以通过为进一步发展奠定基础来改造现代智能系统。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响评审标准进行评估，被认为值得支持。

项目成果

Attentive Walk-Aggregating Graph Neural Networks

• 发表时间: 2021-10
• 期刊: Trans. Mach. Learn. Res.
• 作者: M. F. Demirel;Shengchao Liu;Siddhant Garg;Zhenmei Shi;Yingyu Liang

Stratified Adversarial Robustness with Rejection

• DOI: 10.48550/arxiv.2305.01139
• 发表时间: 2023-05
• 期刊: ArXiv
• 作者: Jiefeng Chen;Jayaram Raghuram;Jihye Choi;Xi Wu;Yingyu Liang;S. Jha

The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning

• DOI: 10.48550/arxiv.2303.00106
• 发表时间: 2023-02
• 期刊: ArXiv
• 作者: Zhenmei Shi;Jiefeng Chen;Kunyang Li;Jayaram Raghuram;Xi Wu;Yingyu Liang;S. Jha

When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysis

• DOI: 10.48550/arxiv.2308.05017
• 发表时间: 2023-08
• 期刊: ArXiv
• 作者: Yiyou Sun;Zhenmei Shi;Yingyu Liang;Yixuan Li

Towards Evaluating the Robustness of Neural Networks Learned by Transduction

• 发表时间: 2021-10
• 期刊: ArXiv
• 作者: Jiefeng Chen;Xi Wu;Yang Guo;Yingyu Liang;S. Jha