Generalization bounds for sparse random feature expansions
Generalization bounds for sparse random feature expansions
复制标题
DOI：
10.1016/j.acha.2022.08.003
复制
发表时间：
2021-03
影响因子：
2.5
通讯作者：
Abolfazl Hashemi;Hayden Schaeffer;Robert Shi;U. Topcu;Giang Tran;Rachel A. Ward
中科院分区：
文献类型：
--
作者：
Abolfazl Hashemi;Hayden Schaeffer;Robert Shi;U. Topcu;Giang Tran;Rachel A. Ward
随机特征方法已经在各种机器学习任务中取得了成功，易于计算，并且具有理论精度范围。它们作为标准神经网络的替代方法，因为它们可以表示类似的函数空间，而无需昂贵的训练阶段。然而，为了准确性，随机特征方法需要比可训练参数更多的测量，限制了它们在数据稀缺应用中的使用。我们引入稀疏随机特征扩展来获得简约的随机特征模型。我们利用压缩感知的思想来生成随机特征扩展，即使在数据稀缺的情况下也有理论保证。我们提供的泛化界的功能在一定的类取决于样本的数量和分布的功能。通过引入稀疏特征，即具有随机稀疏权重的特征，我们为低阶函数提供了改进的边界。我们证明了我们的方法在几个科学机器学习任务中优于浅层网络。
Random feature methods have been successful in various machine learning tasks, are easy to compute, and come with theoretical accuracy bounds. They serve as an alternative approach to standard neural networks since they can represent similar function spaces without a costly training phase. However, for accuracy, random feature methods require more measurements than trainable parameters, limiting their use for data-scarce applications. We introduce the sparse random feature expansion to obtain parsimonious random feature models. We leverage ideas from compressive sensing to generate random feature expansions with theoretical guarantees even in the data-scarce setting. We provide generalization bounds for functions in a certain class depending on the number of samples and the distribution of features. By introducing sparse features, i.e. features with random sparse weights, we provide improved bounds for low order functions. We show that our method outperforms shallow networks in several scientific machine learning tasks.