Theory and Practice of Natural Computing - 7th International Conference, TPNC 2018, Dublin, Ireland, December 12-14, 2018, Proceedings
Theory and Practice of Natural Computing - 7th International Conference, TPNC 2018, Dublin, Ireland, December 12-14, 2018, Proceedings
复制标题
自然计算的理论与实践 - 第七届国际会议，TPNC 2018，爱尔兰都柏林，2018 年 12 月 12-14 日，会议记录
DOI：
10.1007/978-3-030-04070-3_30
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Kabán A
中科院分区：
文献类型：
--
作者：
Kabán A
我们感兴趣的是经典的具有S型激活函数的两层前馈神经网络的理论保证，该网络的输入通过随机投影线性压缩。随着现代数据集维度的迅速增加，以及压缩感知中新型数据采集设备的发展，正确理解压缩感知所能获得的保证具有重要的现实意义。我们从分析以前的工作开始，这些工作试图推导出目标维度的下界，以确保随机投影下的输出具有较低的失真，我们发现与经验观察到的行为不一致。然后，我们给出了目标维度的一个新的下界，与以前的工作不同，它不依赖于隐含神经元的数量，而只依赖于第一层权重的Frobenius范数，此外，它还适用于更大类别的随机投影。数值实验与我们的发现一致。此外，我们能够根据误差和最优网络在原始未压缩类中的期望失真来限定压缩网络的泛化误差。这些结果意味着，只要原始学习问题有足够的正则性，就可以从随机压缩数据中学习具有任意多个隐含单元的网络，我们的分析严格量化了这一点。
We are interested in theoretical guarantees for classic 2-layer feed-forward neural networks with sigmoidal activation functions, having inputs linearly compressed by random projection. Due to the speedy increase of the dimensionality of modern data sets, and the development of novel data acquisition devices in compressed sensing, a proper understanding of are the guarantees obtainable is of much practical importance. We start by analysing previous work that attempted to derive a lower bound on the target dimension to ensure low distortion of the outputs under random projection, we find a disagreement with empirically observed behaviour. We then give a new lower bound on the target dimension that, in contrast with previous work, does not depend on the number of hidden neurons, but only depends on the Frobenius norm of the first layer weights, and in addition it holds for a much larger class of random projections. Numerical experiments agree with our finding. Furthermore, we are able to bound the generalisation error of the compressive network in terms of the error and the expected distortion of the optimal network in the original uncompressed class. These results mean that one can provably learn networks with arbitrarily large number of hidden units from randomly compressed data, as long as there is sufficient regularity in the original learning problem, which our analysis rigorously quantifies.