Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision
Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision
复制标题
DOI：
--
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Haitian Zheng;Haofu Liao;Lele Chen;Wei Xiong;Tianlang Chen;Jiebo Luo
中科院分区：
文献类型：
--
作者：
Haitian Zheng;Haofu Liao;Lele Chen;Wei Xiong;Tianlang Chen;Jiebo Luo
示例引导的图像合成最近已经尝试从语义标签图和示例性图像合成图像。在该任务中，附加样本图像提供控制合成输出的外观的样式指导。尽管具有可控性优势，但现有模型是在具有特定和粗略对齐对象的数据集上设计的。在本文中，我们解决了一个更具挑战性和一般性的任务，其中的样本是一个任意的场景图像，是从给定的标签地图语义不同。为此，我们首先提出了一个掩蔽空间通道注意力（MSCA）模块，通过有效的解耦注意力模型的两个任意场景之间的对应关系。接下来，我们提出了一个端到端的网络，用于联合全局和局部特征对齐和合成。最后，我们提出了一个新的自我监督任务，使培训。在大规模和更多样化的COCO-stuff数据集上的实验表明，与现有方法相比，该方法有显着的改进。此外，我们的方法提供了可解释性，并可以很容易地扩展到其他内容操作任务，包括风格和空间插值或外推。
Example-guided image synthesis has recently been attempted to synthesize an image from a semantic label map and an exemplary image. In the task, the additional exemplar image provides the style guidance that controls the appearance of the synthesized output. Despite the controllability advantage, the existing models are designed on datasets with specific and roughly aligned objects. In this paper, we tackle a more challenging and general task, where the exemplar is an arbitrary scene image that is semantically different from the given label map. To this end, we first propose a Masked Spatial-Channel Attention (MSCA) module which models the correspondence between two arbitrary scenes via efficient decoupled attention. Next, we propose an end-to-end network for joint global and local feature alignment and synthesis. Finally, we propose a novel self-supervision task to enable training. Experiments on the large-scale and more diverse COCO-stuff dataset show significant improvements over the existing methods. Moreover, our approach provides interpretability and can be readily extended to other content manipulation tasks including style and spatial interpolation or extrapolation.