Fixed-Weight Difference Target Propagation
Fixed-Weight Difference Target Propagation
复制标题
DOI：
10.48550/arxiv.2212.10352
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Tatsukichi Shibuya;Nakamasa Inoue;Rei Kawakami;Ikuro Sato
中科院分区：
文献类型：
--
作者：
Tatsukichi Shibuya;Nakamasa Inoue;Rei Kawakami;Ikuro Sato
靶标传播（TP）是一种比误差反射（BP）更合理的算法，用于训练深层网络，而提高TP的实用性是一个空旷的问题。 TP方法要求FeedForward和反馈网络形成层层自动编码器，以传播输出层生成的目标值。但是，这会导致某些缺点。例如，需要仔细的高参数调整才能同步进料和反馈训练，并且通常需要比进料路径的频繁更新反馈路径。了解馈电和反馈网络足以使TP方法能够培训，但是让这些层的自动编码器是TP工作的必要条件？我们通过提出固定重量差异目标传播（FW-DTP）来回答这个问题，该目标在训练过程中保持反馈权重的恒定。我们确认，这种简单的方法自然可以解决TP的上述问题，仍然可以为给定任务提供信息的目标值。实际上，FW-DTP始终在四个分类数据集上的基线（DTP）始终达到更高的测试性能。我们还提出了一种新颖的传播体系结构，该结构解释了DTP的反馈函数的确切形式，以分析FW-DTP。我们的代码可在https://github.com/tatsukichishibuya/fixed-weight-difference-target-propagation上找到。
Target Propagation (TP) is a biologically more plausible algorithm than the error backpropagation (BP) to train deep networks, and improving practicality of TP is an open issue. TP methods require the feedforward and feedback networks to form layer-wise autoencoders for propagating the target values generated at the output layer. However, this causes certain drawbacks; e.g., careful hyperparameter tuning is required to synchronize the feedforward and feedback training, and frequent updates of the feedback path are usually required than that of the feedforward path. Learning of the feedforward and feedback networks is sufficient to make TP methods capable of training, but is having these layer-wise autoencoders a necessary condition for TP to work? We answer this question by presenting Fixed-Weight Difference Target Propagation (FW-DTP) that keeps the feedback weights constant during training. We confirmed that this simple method, which naturally resolves the abovementioned problems of TP, can still deliver informative target values to hidden layers for a given task; indeed, FW-DTP consistently achieves higher test performance than a baseline, the Difference Target Propagation (DTP), on four classification datasets. We also present a novel propagation architecture that explains the exact form of the feedback function of DTP to analyze FW-DTP. Our code is available at https://github.com/TatsukichiShibuya/Fixed-Weight-Difference-Target-Propagation.