Learning and Memorization
Learning and Memorization
复制标题
学习与记忆
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
S. Chatterjee
中科院分区：
文献类型：
--
作者：
S. Chatterjee
在机器学习研究界，人们普遍认为记忆和泛化之间存在紧张关系。在这项工作中，我们检查到什么程度的紧张存在，通过探索是否有可能通过记忆单独概括。虽然直接使用查找表进行记忆显然不会泛化，但我们发现，以支持有限的查找表网络的形式引入深度会导致泛化，这种泛化明显高于机会，并且更接近于标准学习算法在MNIST和CIFAR-10中获得的几个任务。此外，我们通过一系列的实证结果表明，我们的方法允许记忆和泛化之间的平滑权衡，并表现出神经网络的一些最显着的特征：深度提高性能;随机数据可以记忆，但在真实的数据上有泛化;记忆随机数据在某种意义上比记忆真实的数据更难。该算法的极端简单性和与泛化理论的潜在联系为未来的研究指明了几个有趣的方向。
In the machine learning research community, it is generally believed that there is a tension be-tween memorization and generalization. In this work, we examine to what extent this tension exists, by exploring if it is possible to generalize by memorizing alone. Although direct memo-rization with a lookup table obviously does not generalize, we find that introducing depth in the form of a network of support-limited lookup tables leads to generalization that is significantly above chance and closer to those obtained by standard learning algorithms on several tasks derived from MNIST and CIFAR -10. Furthermore, we demonstrate through a series of empirical results that our approach allows for a smooth tradeoff between memorization and generalization and exhibits some of the most salient characteristics of neural networks: depth improves performance; random data can be memorized and yet there is generalization on real data; and memorizing random data is harder in a certain sense than mem-orizing real data. The extreme simplicity of the algorithm and potential connections with generalization theory point to several interesting directions for future research.