Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing
Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing
复制标题
稳定机器学习的动力学预测：通过水库计算测试新颖的受噪声启发的正则化
DOI：
10.1016/j.neunet.2023.10.054
复制
发表时间：
2024
期刊：
影响因子：
7.8
通讯作者：
Ott, Edward
中科院分区：
文献类型：
--
作者：
Wikner, Alexander;Harvey, Joseph;Girvan, Michelle;Hunt, Brian R.;Pomerance, Andrew;Antonsen, Thomas;Ott, Edward
最近的工作表明，机器学习（ML）模型可以巧妙地预测未知混沌系统的动态。状态演变的短期预测和动态（“气候”）的统计模式的长期预测可以通过采用反馈回路来产生，由此模型被训练为仅向前预测一个时间步，然后模型输出被用作多个时间步的输入。然而，在缺乏缓解技术的情况下，这种反馈可能导致人为的快速错误增长（“不稳定性”）。一种已建立的缓解技术是向ML模型训练输入添加噪声。基于这种技术，我们在具有过去输入记忆的ML模型的损失函数中制定了一个新的惩罚项，该惩罚项确定性地近似了在训练期间添加到模型输入中的许多小的独立噪声实现的影响。我们将这种惩罚和由此产生的正则化称为线性化多噪声训练（LMNT）。我们系统地研究了LMNT，输入噪声和其他已建立的正则化技术的影响，在一个案例研究中使用水库计算，机器学习方法，使用递归神经网络，预测时空混沌Kuramoto-Sivashinsky方程。我们发现，用噪声或LMNT训练的水库计算机产生的气候预测似乎是无限稳定的，并且具有与真实系统非常相似的气候，而短期预测比用其他正则化技术训练的预测准确得多。最后，我们证明了我们的LMNT正则化的确定性方面有利于快速水库计算机正则化超参数调整。
Recent work has shown that machine learning (ML) models can skillfully forecast the dynamics of unknown chaotic systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (“climate”) can be produced by employing a feedback loop, whereby the model is trained to predict forward only one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this feedback can result in artificially rapid error growth (“instability”). One established mitigating technique is to add noise to the ML model training input. Based on this technique, we formulate a new penalty term in the loss function for ML models with memory of past inputs thatdeterministicallyapproximates the effect of many small, independent noise realizations added to the model input during training. We refer to this penalty and the resulting regularization as Linearized Multi-Noise Training (LMNT). We systematically examine the effect of LMNT, input noise, and other established regularization techniques in a case study using reservoir computing, a machine learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto–Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while the short-term forecasts are substantially more accurate than those trained with other regularization techniques. Finally, we show the deterministic aspect of our LMNT regularization facilitates fast reservoir computer regularization hyperparameter tuning.
影响因子：
5.2
作者：
Gentine, P.;Pritchard, M.;Yacalis, G.
通讯作者：
Yacalis, G.
DOI：
10.1073/pnas.1810286115
发表时间：
2018-09-25
影响因子：
11.1
作者：
Rasp S;Pritchard MS;Gentine P
通讯作者：
Gentine P