Transform Once: Efficient Operator Learning in Frequency Domain
Transform Once: Efficient Operator Learning in Frequency Domain
复制标题
DOI：
10.48550/arxiv.2211.14453
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
Michael Poli;Stefano Massaroli;Federico Berto;J. Park;Tri Dao;Christopher Ré;Stefano Ermon
中科院分区：
文献类型：
--
作者：
Michael Poli;Stefano Massaroli;Federico Berto;J. Park;Tri Dao;Christopher Ré;Stefano Ermon
谱分析提供了一种最有效的信息保持降维的范例，因为自然发生的信号的简单描述通常是通过周期基函数的少数项获得的。在这项工作中，我们研究了旨在利用频域结构来有效学习空间或时间上的远程相关性的深度神经网络：频域模型（fdm）。现有的fdm基于复值变换，即傅里叶变换（FT），以及分别对频谱和输入数据进行计算的层。该设计引入了相当大的计算开销：对于每一层，都有一个正演和逆傅立叶变换。相反，这项工作引入了通过单个变换进行频域学习的蓝图：变换一次（T1）。为了在频域中实现有效的直接学习，我们推导了一种保持方差的权重初始化方案，并研究了降阶fdm的频率选择方法。我们的结果显着简化了fdm的设计过程，修剪了冗余变换，并随着数据分辨率和模型大小的增加而增加3到10倍的速度。我们进行了大量的实验来学习时空动力学的解算符，包括不可压缩的纳维-斯托克斯，机翼周围的湍流和高分辨率的烟雾视频。T1模型提高了fdm的测试性能，同时需要更少的计算（5小时而不是我们大规模实验的32小时），跨任务的平均预测误差降低了20%以上。
Spectral analysis provides one of the most effective paradigms for information-preserving dimensionality reduction, as simple descriptions of naturally occurring signals are often obtained via few terms of periodic basis functions. In this work, we study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time: frequency-domain models (FDMs). Existing FDMs are based on complex-valued transforms i.e. Fourier Transforms (FT), and layers that perform computation on the spectrum and input data separately. This design introduces considerable computational overhead: for each layer, a forward and inverse FT. Instead, this work introduces a blueprint for frequency domain learning through a single transform: transform once (T1). To enable efficient, direct learning in the frequency domain we derive a variance-preserving weight initialization scheme and investigate methods for frequency selection in reduced-order FDMs. Our results noticeably streamline the design process of FDMs, pruning redundant transforms, and leading to speedups of 3x to 10x that increase with data resolution and model size. We perform extensive experiments on learning the solution operator of spatio-temporal dynamics, including incompressible Navier-Stokes, turbulent flows around airfoils and high-resolution video of smoke. T1 models improve on the test performance of FDMs while requiring significantly less computation (5 hours instead of 32 for our large-scale experiment), with over 20% reduction in average predictive error across tasks.