Matrix inference and estimation in multi-layer models*
Matrix inference and estimation in multi-layer models*
复制标题
多层模型中的矩阵推理和估计*
DOI：
10.1088/1742-5468/ac3a75
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Fletcher, Alyson K
中科院分区：
文献类型：
--
作者：
Pandit, Parthe;Sahraee-Ardakan, Mojtaba;Rangan, Sundeep;Schniter, Philip;Fletcher, Alyson K
我们考虑了随机多层神经网络(NN)的输入变量和隐变量的估计问题。每一层中的隐藏变量被表示为具有沿行和列的统计交互的矩阵。这个问题适用于矩阵补偿、通过深度生成先验模型进行信号恢复、多任务和混合回归以及学习某些类型的两层神经网络。对于这个矩阵推理问题，我们扩展了一种新的算法--多层向量近似消息传递。结果表明，当未知量的维度N×d增长为N→∞且d固定时，所提出的多层矩阵向量近似消息传递算法的性能可以在一定的随机大系统极限下准确地预测.在两层神经网络学习问题中，这种缩放对应于输入特征和训练样本的数量增长到无穷大，而隐藏节点的数量保持不变的情况。通过分析可以对学习的参数和测试误差进行精确的预测。
We consider the problem of estimating the input and hidden variables of a stochastic multi-layer neural network (NN) from an observation of the output. The hidden variables in each layer are represented as matrices with statistical interactions along both rows as well as columns. This problem applies to matrix imputation, signal recovery via deep generative prior models, multi-task and mixed regression, and learning certain classes of two-layer NNs. We extend a recently-developed algorithm—multi-layer vector approximate message passing, for this matrix-valued inference problem. It is shown that the performance of the proposed multi-layer matrix vector approximate message passing algorithm can be exactly predicted in a certain random large-system limit, where the dimensions N× d of the unknown quantities grow as N→∞ with d fixed. In the two-layer neural-network learning problem, this scaling corresponds to the case where the number of input features as well as training samples grow to infinity but the number of hidden nodes stays fixed. The analysis enables a precise prediction of the parameter and test error of the learning.