CDS&E: Efficient and Robust Recurrent Neural Networks

在过去的十年中，深度神经网络已经成为最强大的机器学习方法之一。递归神经网络(RNN)是一种特殊的神经网络，它通过利用序列中的时间连接和处理数据集中不同的序列长度来有效地对序列数据(如语音和文本数据)进行建模。虽然RNN及其变体在许多实际应用程序中取得了成功，但仍有各种问题使它们难以在实践中使用。该项目将系统地解决其中的一些困难，并开发一个高效和健壮的RNN。在这个项目中派生的计算机代码将免费提供。这些研究成果将在计算机视觉、语音识别、自然语言处理、金融数据分析和生物信息学等涉及顺序数据学习的各种领域中得到应用。与其他神经网络一样，神经网络的训练通常涉及到梯度下降优化的一些变体，这容易导致所谓的梯度消失或爆炸问题。RNN的正则化也是具有挑战性的，RNN指的是用于防止模型过度拟合降雨数据从而对新数据的泛化能力较差的技术。当前优选的RNN体系结构，例如长-短期-记忆网络，具有高度复杂的结构，具有许多不容易理解的附加交互元素。这个项目开发了一种RNN，它扩展了最近的正交/酉RNN，以更有效地对序列数据的长期和短期相关性进行建模。通过递归矩阵的间接参数化，将发展丢弃正则化技术。该项目开发的网络将保持基本RNN的简单性和效率，但增强了一些关键功能，以实现强大的应用。特别是，该项目将包括RNN在一些生物信息学问题上的应用研究。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks have emerged over the last decade as one of the most powerful machine learning methods. Recurrent neural networks (RNNs) are special neural networks that are designed to efficiently model sequential data such as speech and text data by exploiting temporal connections within a sequence and handling varying sequence lengths in a dataset. While RNN and its variants have found success in many real-world applications, there are various issues that make them difficult to use in practice. This project will systematically address some of these difficulties and develop an efficient and robust RNN. Computer codes derived in this project will be made freely available. The research results will have applications in a variety of areas involving sequential data learning, including computer vision, speech recognition, natural language processing, financial data analysis, and bioinformatics.As in other neural networks, training of RNNs typically involves some variants of gradient descent optimization, which is prone to so-called vanishing or exploding gradient problems. Regularization of RNNs, which refers to techniques used to prevent the model from overfitting the raining data and hence poor generalization to new data, is also challenging. The current preferred RNN architectures such as the Long-Short-Term-Memory networks have highly complex structures with numerous additional interacting elements that are not easy to understand. This project develops an RNN that extends a recent orthogonal/unitary RNNs to more effectively model long and short term dependency of sequential data. Through an indirect parametrization of recurrent matrix, dropout regularization techniques will be developed. The network developed in this project will retain the simplicity and efficiency of basic RNNs but enhance some key capabilities for robust applications. In particular, the project will include a study of applications of RNNs to some bioinformatics problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.