Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization
Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization
复制标题
DOI：
10.48550/arxiv.2211.05262
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
Alexander Wikner;B. Hunt;Joseph Harvey;M. Girvan;E. Ott
中科院分区：
文献类型：
--
作者：
Alexander Wikner;B. Hunt;Joseph Harvey;M. Girvan;E. Ott
最近的研究表明，机器学习（ML）模型可以被训练来准确预测未知混沌动力系统的动态。状态演变的短期预测和动态（"气候“）的统计模式的长期预测可以通过采用反馈回路来产生，由此模型被训练为向前预测一个时间步，然后模型输出被用作多个时间步的输入。然而，在缺乏缓解技术的情况下，这种技术可能导致人为的快速错误增长。在本文中，我们系统地研究了在训练过程中向ML模型输入添加噪声的技术，以提高稳定性和预测准确性。此外，我们还引入了线性化多噪声训练（LMNT），这是一种正则化技术，可以确定性地近似在训练过程中添加到模型输入中的许多小的独立噪声实现的效果。我们的案例研究使用水库计算，机器学习方法，使用递归神经网络，预测时空混沌Kuramoto-Sivashinsky方程。我们发现，用噪声或LMNT训练的水库计算机产生的气候预测似乎是无限稳定的，并且具有与真实系统非常相似的气候，而未经正则化训练的水库计算机是不稳定的。与在某些情况下产生稳定性的其他正则化技术相比，我们发现用噪声或LMNT训练的水库计算机的短期和气候预测都要准确得多。最后，我们表明，与带噪声的训练相比，我们的LMNT正则化的确定性方面有利于快速超参数调整。
Recent work has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of unknown chaotic dynamical systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (``climate'') can be produced by employing a feedback loop, whereby the model is trained to predict forward one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this technique can result in artificially rapid error growth. In this article, we systematically examine the technique of adding noise to the ML model input during training to promote stability and improve prediction accuracy. Furthermore, we introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. Our case study uses reservoir computing, a machine-learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto-Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while reservoir computers trained without regularization are unstable. Compared with other regularization techniques that yield stability in some cases, we find that both short-term and climate predictions from reservoir computers trained with noise or with LMNT are substantially more accurate. Finally, we show that the deterministic aspect of our LMNT regularization facilitates fast hyperparameter tuning when compared to training with noise.