Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding
Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding
复制标题
DOI：
10.1109/tpami.2021.3058684
复制
发表时间：
2021-02
影响因子：
23.6
通讯作者：
Mingjie Sun;Jimin Xiao;Eng Gee Lim;Si Liu;J. Y. Goulermas
中科院分区：
文献类型：
--
作者：
Mingjie Sun;Jimin Xiao;Eng Gee Lim;Si Liu;J. Y. Goulermas
在本文中，我们解决的是弱监督指代表达式基础任务，用于根据查询语句定位图像中的参照对象，其中图像区域与查询之间的映射在训练阶段不可用。在传统的方法中，首先选择与指代表达式最匹配的目标区域，然后从选择的区域重构查询语句，重构差异作为反向传播的损失。然而，现有的方法都是近似地进行匹配和重建，因为它们忽略了匹配正确性未知的事实。为了克服这一局限性，这里设计了一个判别三元组作为解决方案的基础，通过它可以将一个查询以非常可伸缩的方式转换为一个或多个判别三元组。在区分三元组的基础上，进一步提出了轻量级但对弱监督训练有效的三元组匹配和重建模块，使其比现有的方法轻量级和快速性提高了三倍。我们的工作的一个重要优点是它的卓越性能，尽管设计简单而整洁。具体地说，在RefCOCO(39.21%)、RefCOCO+(39.18%)和RefCOg(43.24%)数据集上进行评估时，该方法获得了新的最高准确率，分别比以前的方法提高了4.17%、4.08%和7.8%。代码可在https://github.com/insomnia94/DTWREG.上获得
In this paper, we are tackling the weakly-supervised referring expression grounding task, for the localization of a referent object in an image according to a query sentence, where the mapping between image regions and queries are not available during the training stage. In traditional methods, an object region that best matches the referring expression is picked out, and then the query sentence is reconstructed from the selected region, where the reconstruction difference serves as the loss for back-propagation. The existing methods, however, conduct both the matching and the reconstruction approximately as they ignore the fact that the matching correctness is unknown. To overcome this limitation, a discriminative triad is designed here as the basis to the solution, through which a query can be converted into one or multiple discriminative triads in a very scalable way. Based on the discriminative triad, we further propose the triad-level matching and reconstruction modules which are lightweight yet effective for the weakly-supervised training, making it three times lighter and faster than the previous state-of-the-art methods. One important merit of our work is its superior performance despite the simple and neat design. Specifically, the proposed method achieves a new state-of-the-art accuracy when evaluated on RefCOCO (39.21 percent), RefCOCO+ (39.18 percent) and RefCOCOg (43.24 percent) datasets, that is 4.17, 4.08 and 7.8 percent higher than the previous one, respectively. The code is available at https://github.com/insomnia94/DTWREG.