"No Free Lunch" in Neural Architectures? A Joint Analysis of Expressivity, Convergence, and Generalization
"No Free Lunch" in Neural Architectures? A Joint Analysis of Expressivity, Convergence, and Generalization
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Wuyang Chen;Wei Huang;Zhangyang Wang
中科院分区：
文献类型：
--
作者：
Wuyang Chen;Wei Huang;Zhangyang Wang
深度学习和自动机器学习(AutoML)的繁荣在很大程度上植根于新型神经网络的发展--但在架构空间中，是什么定义了网络的“善”呢？测试精度是AutoML的黄金标准，它与三个方面密切相关：(1)表现力(网络对训练数据逼近的复杂函数)；(2)收敛(网络在梯度下降下达到低训练误差的速度)；(3)泛化(是否能将训练网络从训练数据推广到具有低测试误差的未见样本)。然而，以前的大多数理论论文都集中在固定模型结构上，而在很大程度上忽略了实际中使用的复杂网络。为了促进AutoML对体系结构设计的解释和理解，我们的目标是连接一个更大的图景：体系结构如何共同影响其表现力、汇聚性和泛化？我们从一个体系结构空间论证了网络中“没有免费午餐”的行为：给定参数数量的固定预算，不存在一个在所有三个方面都是最优的单一体系结构。换句话说，分别优化表现力、收敛和泛化将在体系结构空间中实现不同的网络。我们的分析解释了AutoML中广泛的观察结果。在流行基准上的实验证实了我们的分析。代码可从以下网址获得：https://github.com/chenwydj/no_free_lunch_architectures。
The prosperity of deep learning and automated machine learning (AutoML) is largely rooted in the development of novel neural networks – but what defines the “goodness” of networks in an architecture space? Test accuracy, a golden standard in AutoML, is closely related to three aspects: (1) expressivity ( how complicated functions a network can approximate over the training data); (2) convergence ( how fast the network can reach low training error under gradient descent); (3) generalization (whether a trained network can be generalized from the training data to unseen samples with low test error ). However, most previous theory papers focus on fixed model structures, largely ignoring sophisticated networks used in practice. To facilitate the interpretation and understanding of the architecture design by AutoML, we target connecting a bigger picture: how does the architecture jointly impact its expressivity, convergence, and generalization? We demonstrate the “no free lunch” behavior in networks from an architecture space: given a fixed budget on the number of parameters, there does not exist a single architecture that is optimal in all three aspects. In other words, separately optimizing expressivity, convergence, and generalization will achieve different networks in the architecture space. Our analysis explains a wide range of observations in AutoML. Experiments on popular benchmarks confirm our analysis. Code is available at: https://github.com/chenwydj/no_free_lunch_architectures .