Signal Processing for Implicit Neural Representations
Signal Processing for Implicit Neural Representations
复制标题
DOI：
10.48550/arxiv.2210.08772
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Dejia Xu;Peihao Wang;Yifan Jiang-;Zhiwen Fan;Zhangyang Wang
中科院分区：
文献类型：
--
作者：
Dejia Xu;Peihao Wang;Yifan Jiang-;Zhiwen Fan;Zhangyang Wang
隐式神经表征（INRs）通过多层感知器编码连续多媒体数据，在各种计算机视觉任务中显示出无可争议的前景。尽管有许多成功的应用，编辑和处理INR仍然很棘手，因为信号是由神经网络的潜在参数表示的。现有的作品通过对离散实例的处理来处理这种连续表示，这破坏了INR的紧凑性和连续性。在这项工作中，我们提出了一个关于这个问题的试点研究：如何直接修改INR而不进行显式解码？我们通过INR上的微分算子提出了一个隐式神经信号处理网络，称为inspn - net，来回答这个问题。我们的关键见解是，神经网络的空间梯度可以解析计算，并且对平移是不变的，而在数学上，我们表明任何连续卷积滤波器都可以由高阶微分算子的线性组合统一逼近。通过这两个旋钮，inspi - net将信号处理算子实例化为与inr的高阶导数对应的计算图的加权组合，其中加权参数可以通过数据驱动学习。基于我们提出的inspi - net，我们进一步构建了第一个隐式运行在inr上的卷积神经网络（CNN），命名为inspi - convnet。我们的实验验证了insps - net和insps - convnet在拟合低级图像和几何处理内核（如模糊、去模糊、去噪、补漆和平滑）以及在隐式领域（如图像分类）的高级任务方面的表达能力。
Implicit Neural Representations (INRs) encoding continuous multi-media data via multi-layer perceptrons has shown undebatable promise in various computer vision tasks. Despite many successful applications, editing and processing an INR remains intractable as signals are represented by latent parameters of a neural network. Existing works manipulate such continuous representations via processing on their discretized instance, which breaks down the compactness and continuous nature of INR. In this work, we present a pilot study on the question: how to directly modify an INR without explicit decoding? We answer this question by proposing an implicit neural signal processing network, dubbed INSP-Net, via differential operators on INR. Our key insight is that spatial gradients of neural networks can be computed analytically and are invariant to translation, while mathematically we show that any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators. With these two knobs, INSP-Net instantiates the signal processing operator as a weighted composition of computational graphs corresponding to the high-order derivatives of INRs, where the weighting parameters can be data-driven learned. Based on our proposed INSP-Net, we further build the first Convolutional Neural Network (CNN) that implicitly runs on INRs, named INSP-ConvNet. Our experiments validate the expressiveness of INSP-Net and INSP-ConvNet in fitting low-level image and geometry processing kernels (e.g. blurring, deblurring, denoising, inpainting, and smoothening) as well as for high-level tasks on implicit fields such as image classification.