Recurrent Deep Learning Machines for Robust, Adaptive, or Accommodative Filtering

对一个过程（称为信号过程）的预测和估计，给定一个相关过程（称为测量过程），这两者通常都涉及随机性，是广泛领域中的一个基本问题。随着信号的演变和测量值的不断进入，需要一种算法来预测或估计每个时刻的信号，使用同一时刻的测量值来更新预测或估计，而不需要使用先前的测量值。这种算法称为过滤器。当信号或测量过程受到不确定或变化的环境影响时，适应环境的滤波器称为自适应滤波器。在许多应用中，无论是涉及不确定的还是变化的环境，估计或预测中的大的个体误差都可能导致不期望的甚至灾难性的后果，并且要避免。能够减少较大误差的滤波器称为鲁棒滤波器。鲁棒滤波器必须平衡滤波精度和鲁棒性。非线性信号或测量过程的最优滤波是一个长期存在的臭名昭著的问题，直到1992年神经滤波器被提出。虽然神经网络滤波与其主要竞争对手粒子滤波相比具有许多优点，但神经网络滤波器训练中的局部极小值问题一直困扰着该方法。局部最小值问题最终被一种称为渐进解凸化方法的技术所克服，该方法是在最近的NSF资助下开发的。神经过滤器现在可以应用了。本计画的目的是发展适应性与强健性的类神经网路滤波器，特别是下列之滤波器：（1）补偿类神经网路滤波器。经过适当训练的具有固定权重的递归神经网络（RNN）被证明具有自适应能力，称为递归神经网络。他们没有在线调整。这是一个重要的优点，因为信号处理通常无法在线进行权重调整。作为自适应神经网络的自适应滤波器被称为自适应神经滤波器。(2)具有长期和短期记忆的自适应神经滤波器。如果分别以非线性和线性方式影响RNN输出的RNN的非线性和线性权重分别用作长期和短期记忆（LASTM），则已经证明可以针对不同的环境离线训练长期记忆，并且仅需要在线调整短期记忆以适应环境。具有LASTM的自适应神经滤波器被称为自适应神经滤波器（具有LASTM）。这样的过滤器，预计有更好的泛化能力比递归神经过滤器。(3)强大的神经过滤器。神经网络训练的标准化风险敏感误差（NRSE）准则中的风险敏感性指标决定了神经网络的鲁棒性程度。取决于是否;无论是积极的还是消极的，NRSE都避免了更大的“风险”或忽略了“离群值”，以分别产生稳健的工程或稳健的统计性能。证明了鲁棒神经滤波器的存在性。同时证明了当风险敏感性指数无限增长时，NRSE趋近于极小极大准则。(4)鲁棒的递归神经滤波器。如果要求滤波器具有自适应和鲁棒性能，并且不希望滤波器的在线调整，则可以使用鲁棒补偿滤波器。(5)具有长期和短期记忆的鲁棒自适应神经滤波器。如果要求滤波器具有自适应和鲁棒性能，并且期望滤波器具有更好的泛化能力，则可以使用具有LASTM的鲁棒自适应滤波器。

Prediction and estimation of a process, called a signal process, given a relevant process, called a measurement process, both of which usually involve randomness, is a fundamental problem in a broad range of fields. As the signal evolves and measurements keep coming in, an algorithm is needed to predict or estimate the signal at each time instant using the measurement at the same instant to update the prediction or estimate without requiring the use of the preceding measurements. Such an algorithm is called a filter. When the signal or measurement process is affected by an uncertain or changing environment, a filter that adapts to the environment is called an adaptive filter. In many applications, whether an uncertain or changing environment is involved, large individual errors in estimation or prediction may cause undesirable or even disastrous consequences and are to be avoided. A filter that can reduce large errors is called a robust filter. A robust filter must balance filtering accuracy and robustness. Optimal Filtering for nonlinear signal or measurement processes was a long-standing notorious problem until neural filters were proposed in 1992. Although neural filtering has many advantages over its main competitor, the particle filter, the local-minimum problem in training neural filters plagued the approach until now. The local-minimum problem has finally been overcome by a technique called the gradual deconvexification method developed under a recent NSF grant. Neural filters are now ready for application. The purpose of this project is to develop adaptive and robust neural filters.In particular, the following filters will be developed:(1) Accommodative neural filters. Properly trained RNNs (recurrent neural networks) with fixed weights are proven to have adaptive ability and are called accommodative neural networks. They are not adjusted online. This is an important advantage because the signal process is usually unavailable online for weight adjustment. An adaptive filter that is an accommodative neural network is called an accommodative neural filter.(2) Adaptive neural filters with long- and short-term memories. If the nonlinear and linear weights of an RNN, which affect the RNN's outputs in a nonlinear and linear manner respectively, are used as long- and short-term memories (LASTMs) respectively, it has been proven that the long-term memory can be trained offline for different environments and only the short-term memory needs to be adjusted online to adapt to the environment. An adaptive neural filter that has LASTMs is called an adaptive neural filter (with LASTMs). Such filters are expected to have better generalization capability than accommodative neural filters.(3) Robust neural filters. The risk-sensitivity index in the normalized risk-sensitive error (NRSE) criterion for training a neural network determines its degree of robustness. Depending on whether; being positive or negative, the NRSE averts larger "risks" or ignores "outliers" to induce robust engineering or robust statistical performance respectively. Existence of robust neural filters has been proven. It is also proven that as the risk-sensitivity index grows without bound, the NRSE approaches the minimax criterion.(4) Robust accommodative neural filters. If both adaptive and robust performances are required of a filter and online adjustment of the filter is undesirable, then a robust accommodative filter can be used.(5) Robust adaptive neural filters with long- and short-term memories. If both adaptive and robust performances are required of a filter and better generalization ability of the filter is desirable, then a robust adaptive filter with LASTMs can be used.