Recurrent Deep Learning Machines

本研究的目的是开发一种具有反馈结构的深度学习机器的新范式。反馈将来自同一层或更高层的其他计算节点的相邻或更大的接受域中包含的当前或过去的信息带到计算节点，以形成更好的局部表示或特征。这些信息是处理动态数据和最大化静态数据泛化能力所必需的。本研究的方法是选择或设计深度和循环架构，开发生成和判别学习技术，并将凸化训练标准的风险规避方法集成到训练循环深度学习机器中。智力优势：递归神经网络在涉及动态数据的应用中是不可替代的，即使在静态数据上也比前馈网络更好。然而，训练递归网络的困难阻碍了它们的发展和理解。这项提议的研究有望帮助消除这一困难，发挥递归神经网络的全部力量，并提高人们对神经网络的兴趣，近年来，神经网络不幸地不受欢迎。更广泛的影响：循环深度学习机器在静态和动态分类和回归方面功能强大，包括图像和视频识别、分析和压缩；非线性系统辨识/控制；信号处理/过滤;以及关键系统运行状况/故障监控/检测。因此，所提出的工作将对医疗仪器，计算机/机器人/信息技术，无线通信，国防和国土安全做出重大贡献。循环深度学习机器将成为工程和计算机科学研究生教育的重要组成部分

The objective of this research is to develop a new paradigm of deep learning machine - those with a feedback structure. Feedbacks bring to computing nodes current or past information contained in neighboring or larger receptive fields of other computing nodes from the same or higher layers for forming better local representations or features. Such information is required for processing dynamical data and for maximizing generalization capabilities on static data. The approach of this research is to select or design deep and recurrent architectures, develop generative and discriminative learning techniques, and integrating the risk-averting method of convexifying training criteria into training recurrent deep learning machines. Intellectual Merit: Recurrent neural networks are irreplaceable for applications involving dynamical data and are fundamentally better than feedforward networks even on static data. However, difficulty in training recurrent networks has stifled development and understanding of them. The proposed research is expected to help remove this difficulty, bring forth the full power of recurrent neural networks, and boost interests in neural networks in general, which have unfortunately and undeservedly fallen out of favor in recent years. Broader Impact: Recurrent deep learning machines are powerful for static and dynamical classification and regression, including image and video recognition, analysis and compression; nonlinear system identification/control; signal processing/filtering; and critical system health/fault monitoring/detection. Therefore, the proposed work will contribute greatly to medical instrumentation, computer/robot/information technology, wireless telecommunication, national defense, and homeland security. Recurrent deep learning machines will ecome an important component in the graduate education in engineering and computer science