CDS&E: Efficient and Robust Recurrent Neural Networks

深度神经网络在过去十年中已经成为最强大的机器学习方法之一。递归神经网络（RNN）是一种特殊的神经网络，旨在通过利用序列中的时间连接并处理数据集中不同的序列长度来有效地对语音和文本数据等序列数据进行建模。虽然RNN及其变体在许多现实世界的应用中取得了成功，但仍存在各种问题使其难以在实践中使用。该项目将系统地解决其中的一些困难，并开发一个高效和强大的RNN。在这个项目中产生的计算机代码将免费提供。这些研究成果将在涉及序列数据学习的各种领域中得到应用，包括计算机视觉、语音识别、自然语言处理、金融数据分析和生物信息学。与其他神经网络一样，RNN的训练通常涉及梯度下降优化的一些变体，这很容易出现所谓的消失或爆炸梯度问题。RNN的正则化是指用于防止模型过度拟合训练数据并因此对新数据泛化能力差的技术，也具有挑战性。当前首选的RNN架构（如长短期记忆网络）具有高度复杂的结构，其中包含许多不容易理解的额外交互元素。该项目开发了一种RNN，扩展了最近的正交/酉RNN，以更有效地对序列数据的长期和短期依赖性进行建模。通过递归矩阵的间接参数化，将开发辍学正则化技术。该项目中开发的网络将保留基本RNN的简单性和效率，但增强了一些关键功能，以实现强大的应用程序。特别是，该项目将包括一个RNN的应用研究，一些生物信息学问题。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Deep neural networks have emerged over the last decade as one of the most powerful machine learning methods. Recurrent neural networks (RNNs) are special neural networks that are designed to efficiently model sequential data such as speech and text data by exploiting temporal connections within a sequence and handling varying sequence lengths in a dataset. While RNN and its variants have found success in many real-world applications, there are various issues that make them difficult to use in practice. This project will systematically address some of these difficulties and develop an efficient and robust RNN. Computer codes derived in this project will be made freely available. The research results will have applications in a variety of areas involving sequential data learning, including computer vision, speech recognition, natural language processing, financial data analysis, and bioinformatics.As in other neural networks, training of RNNs typically involves some variants of gradient descent optimization, which is prone to so-called vanishing or exploding gradient problems. Regularization of RNNs, which refers to techniques used to prevent the model from overfitting the raining data and hence poor generalization to new data, is also challenging. The current preferred RNN architectures such as the Long-Short-Term-Memory networks have highly complex structures with numerous additional interacting elements that are not easy to understand. This project develops an RNN that extends a recent orthogonal/unitary RNNs to more effectively model long and short term dependency of sequential data. Through an indirect parametrization of recurrent matrix, dropout regularization techniques will be developed. The network developed in this project will retain the simplicity and efficiency of basic RNNs but enhance some key capabilities for robust applications. In particular, the project will include a study of applications of RNNs to some bioinformatics problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.