DML: Differ-Modality Learning for Building Semantic Segmentation
DML: Differ-Modality Learning for Building Semantic Segmentation
复制标题
DML：用于构建语义分割的不同模态学习
DOI：
10.1109/tgrs.2022.3148383
复制
发表时间：
2022
影响因子：
8.2
通讯作者：
Baier Gerald
中科院分区：
文献类型：
--
作者：
Xia Junshi;Yokoya Naoto;Baier Gerald
本文分析了遥感领域中不同模态建筑物语义分割存在的问题。随着多模态数据集的增长，如光学，合成孔径雷达（SAR），光探测和测距（LiDAR），以及语义知识的稀缺性，学习多模态信息的任务在过去几年中变得越来越重要。然而，由于许多因素，多模态数据集不能同时获得。假设我们在一个地方有参考信息的SAR图像和在另一个地方没有参考的光学图像，如何从SAR图像学习光学图像的相关特征？我们称之为不同模态学习（DML）。为了解决DML问题，我们提出了新的深度神经网络架构，其中包括图像自适应、特征自适应、知识蒸馏和针对不同场景的自训练（SL）模块。我们在不同模态的遥感数据集（非常高分辨率的SAR和RGB从SpaceNet 6）上测试所提出的方法，以建立语义分割，并实现上级效率。所提出的方法实现了最佳的性能相比，国家的最先进的方法。
This work critically analyzes the problems arising from differ-modality building semantic segmentation in the remote sensing domain. With the growth of multimodality datasets, such as optical, synthetic aperture radar (SAR), light detection and ranging (LiDAR), and the scarcity of semantic knowledge, the task of learning multimodality information has increasingly become relevant over the last few years. However, multimodality datasets cannot be obtained simultaneously due to many factors. Assume that we have SAR images with reference information in one place and optical images without reference in another; how to learn relevant features of optical images from SAR images? We refer to it as differ-modality learning (DML). To solve the DML problem, we propose novel deep neural network architectures, which include image adaptation, feature adaptation, knowledge distillation, and self-training (SL) modules for different scenarios. We test the proposed methods on the differ-modality remote sensing datasets (very high-resolution SAR and RGB from SpaceNet 6) to build semantic segmentation and to achieve a superior efficiency. The presented approach achieves the best performance when compared with the state-of-the-art methods.