Recurrent Deep Learning Machines for Robust, Adaptive, or Accommodative Filtering

对一个过程的预测和估计，称为信号过程，给定一个相关的过程，称为测量过程，这两者通常都包含随机性，在广泛的领域中是一个基本问题。随着信号的发展和测量的不断到来，需要一种算法来使用同一时刻的测量来预测或估计每个时刻的信号，以更新预测或估计，而不需要使用先前的测量。这样的算法被称为过滤器。当信号或测量过程受到不确定或变化的环境的影响时，适应该环境的滤波器称为自适应滤波器。在许多应用中，无论是涉及不确定的环境还是变化的环境，估计或预测中的大的个体误差都可能导致不良的甚至灾难性的后果，因此必须避免。能够减少大误差的滤波器称为鲁棒滤波。稳健的滤波器必须在滤波精度和稳健性之间取得平衡。非线性信号或测量过程的最优滤波是一个长期存在的臭名昭著的问题，直到1992年神经滤波器被提出。尽管神经网络与其主要竞争对手粒子滤波相比有许多优点，但神经网络训练过程中的局部极小值问题一直困扰着该方法。在美国国家科学基金会最近的一项拨款下开发的一种名为逐步去凸化方法的技术终于克服了局部最小化问题。神经过滤器现在可以应用了。本项目的目的是开发自适应和鲁棒的神经滤波器，特别是将开发以下滤波器：(1)调节神经滤波器。经过适当训练的具有固定权值的递归神经网络(RNN)被证明具有自适应能力，被称为调节性神经网络。它们不会在网上进行调整。这是一个重要的优势，因为信号处理通常不能在线用于权重调整。作为调节神经网络的自适应过滤器称为调节神经过滤器。(2)具有长期和短期记忆的自适应神经过滤器。如果将影响神经网络输出的非线性权值和线性权值分别用作长期记忆和短期记忆(LASTM)，已证明长期记忆可以针对不同的环境离线训练，只需在线调整短期记忆即可适应环境。具有LASTM的自适应神经过滤器称为自适应神经过滤器(具有LASTM)。这种滤波器比调节神经滤波器具有更好的泛化能力。(3)鲁棒神经滤波器。神经网络训练的归一化风险敏感误差(NRSE)准则中的风险敏感性指数决定了神经网络的鲁棒性程度。根据是积极的还是消极的，NRSE分别避免了更大的“风险”或忽略了“离群值”，以诱导稳健的工程或稳健的统计表现。稳健神经网络滤波器的存在已被证明。证明了当风险敏感指数无界增长时，NRSE逼近极小极大准则。(4)鲁棒调节神经网络滤波器。如果一个滤波器同时需要自适应和稳健性能，而在线调整不理想，则可以使用稳健调节滤波器。(5)具有长期和短期记忆的稳健自适应神经网络滤波器。如果同时要求滤波器的自适应和稳健性能，并且希望该滤波器具有更好的泛化能力，则可以使用带有LASTM的稳健自适应滤波器。

Prediction and estimation of a process, called a signal process, given a relevant process, called a measurement process, both of which usually involve randomness, is a fundamental problem in a broad range of fields. As the signal evolves and measurements keep coming in, an algorithm is needed to predict or estimate the signal at each time instant using the measurement at the same instant to update the prediction or estimate without requiring the use of the preceding measurements. Such an algorithm is called a filter. When the signal or measurement process is affected by an uncertain or changing environment, a filter that adapts to the environment is called an adaptive filter. In many applications, whether an uncertain or changing environment is involved, large individual errors in estimation or prediction may cause undesirable or even disastrous consequences and are to be avoided. A filter that can reduce large errors is called a robust filter. A robust filter must balance filtering accuracy and robustness. Optimal Filtering for nonlinear signal or measurement processes was a long-standing notorious problem until neural filters were proposed in 1992. Although neural filtering has many advantages over its main competitor, the particle filter, the local-minimum problem in training neural filters plagued the approach until now. The local-minimum problem has finally been overcome by a technique called the gradual deconvexification method developed under a recent NSF grant. Neural filters are now ready for application. The purpose of this project is to develop adaptive and robust neural filters.In particular, the following filters will be developed:(1) Accommodative neural filters. Properly trained RNNs (recurrent neural networks) with fixed weights are proven to have adaptive ability and are called accommodative neural networks. They are not adjusted online. This is an important advantage because the signal process is usually unavailable online for weight adjustment. An adaptive filter that is an accommodative neural network is called an accommodative neural filter.(2) Adaptive neural filters with long- and short-term memories. If the nonlinear and linear weights of an RNN, which affect the RNN's outputs in a nonlinear and linear manner respectively, are used as long- and short-term memories (LASTMs) respectively, it has been proven that the long-term memory can be trained offline for different environments and only the short-term memory needs to be adjusted online to adapt to the environment. An adaptive neural filter that has LASTMs is called an adaptive neural filter (with LASTMs). Such filters are expected to have better generalization capability than accommodative neural filters.(3) Robust neural filters. The risk-sensitivity index in the normalized risk-sensitive error (NRSE) criterion for training a neural network determines its degree of robustness. Depending on whether; being positive or negative, the NRSE averts larger "risks" or ignores "outliers" to induce robust engineering or robust statistical performance respectively. Existence of robust neural filters has been proven. It is also proven that as the risk-sensitivity index grows without bound, the NRSE approaches the minimax criterion.(4) Robust accommodative neural filters. If both adaptive and robust performances are required of a filter and online adjustment of the filter is undesirable, then a robust accommodative filter can be used.(5) Robust adaptive neural filters with long- and short-term memories. If both adaptive and robust performances are required of a filter and better generalization ability of the filter is desirable, then a robust adaptive filter with LASTMs can be used.