Autonomous and Efficiently Scalable Deep Learning

人类和动物的神经回路通过不同的信息处理阶段从感觉刺激中提取有意义的高级信息。神经处理在非常大的范围内具有高度的自主性、灵活性、互动性和高效率。相比之下，机器学习界目前开发的深度学习算法，首先需要研究人员的强烈参与，其次，与哺乳动物的感觉系统相比，规模相对较小，效率较低。研究人员的依赖是由于大量的自由参数集必须是手工设置的，以及大量数据必须手动标记。深度学习的可扩展性取决于学习自主性和学习的技术实现。因此，我们在这个项目中的目标是：(A)最大限度地减少深度学习对开发研究人员的依赖，以及(B)将深度学习有效地扩展到大型网络。为了实现我们的第一个目标(A)，我们将研究具有较少自由参数和内部自我调整的深度学习。我们的研究将建立在泊松混合的基础上，它是有向的和概率的图形模型，具有精确的闭合形式的学习方程和完全可解释的隐藏状态。它们在弱标签数据上具有功能竞争力，并提供复杂的不确定性信息。这种不确定性反馈将通过与环境的主动交互来进一步提高自主性。为了实现我们的第二个目标(B)，我们将有效的近似方法与神经电路中非常紧凑和可局部实现的学习相结合。这种电路在理论上和经验上都可以被证明是泊松混合中的近似最优学习。它们本质上是并行的、无监督的和在线的，它们在GPU集群和模拟VLSI上的实现是直接的。综上所述，我们寻求开发到目前为止最自主和最高效的可扩展深度学习系统。

Neural circuits of humans and animals extract meaningful high-level information from sensory stimuli using different stages of information processing. Neural processing is highly autonomous, flexible, interactive and efficient at very large scales. In contrast, Deep Learning algorithms as currently developed by the Machine Learning community require, first, strong involvements of researchers and, second, are relatively small scale and inefficient compared, e.g., to mammalian sensory systems. Researcher dependence is due to large sets hsof free parameters that have to be hand-set and large amounts of data that have to be hand-labeled. Scalability of Deep Learning depends on learning autonomy and on technical implementations of learning. Our goals in this project are therefore: (A) to minimize the dependency of Deep Learning on developing researchers, and (B) to efficiently scale Deep Learning to large networks.To achieve our first goal (A), we will study Deep Learning with low numbers of free parameters and internal self-tuning. Our investigations will build up on Poisson mixtures which are directed and probabilistic graphical models with exact closed-form learning equations and fully interpretable hidden states. They are functionally competitive on weakly labeled data and provide sophisticated uncertainty information. This uncertainty feedback will be used to further increase autonomy by an active interaction with the environment.To achieve our second goal (B), we apply efficient approximation methods in combination with very compact and locally implementable learning in neural circuits. Such circuits can theoretically and empirically be shown to approximate optimal learning in Poisson mixtures. They are inherently parallel, learn unsupervised and online, and their implementation on GPU clusters and analog VLSI is straight-forward.In summary we seek to develop the most autonomous and most efficiently scalable Deep Learning systems to date.