Action Modifiers: Learning From Adverbs in Instructional Videos
Action Modifiers: Learning From Adverbs in Instructional Videos
复制标题
DOI：
10.1109/cvpr42600.2020.00095
复制
发表时间：
2019-12
期刊：
影响因子：
--
通讯作者：
Hazel Doughty;I. Laptev;W. Mayol-Cuevas;D. Damen
中科院分区：
文献类型：
--
作者：
Hazel Doughty;I. Laptev;W. Mayol-Cuevas;D. Damen
我们提出了一种方法来学习表示副词从教学视频使用弱监督从伴随的叙述。我们的方法的关键是这样一个事实，即副词的视觉表示高度依赖于它所适用的动作，尽管同一个副词会以类似的方式修饰多个动作。例如，虽然“快速传播”和“快速混合”看起来不一样，但我们可以学习一个共同的表示，使我们能够识别两者以及其他动作。我们将其表述为嵌入问题，并使用缩放的点积注意力来从弱监督视频叙事中学习。我们共同学习副词作为可逆变换，在嵌入空间上操作，以便添加或删除副词的效果。由于之前没有关于副词弱监督学习的工作，我们从HowTo 100 M数据集的一个子集中收集了6个副词的成对动作副词注释：快速/缓慢，精细/粗略和部分/完全。我们的方法优于所有基线的视频到副词检索的性能为0.719 mAP。我们还展示了我们的模型的能力，参加相关的视频部分，以确定一个给定的动作的副词。
We present a method to learn a representation for adverbs from instructional videos using weak supervision from the accompanying narrations. Key to our method is the fact that the visual representation of the adverb is highly dependent on the action to which it applies, although the same adverb will modify multiple actions in a similar way. For instance, while ‘spread quickly’ and ‘mix quickly’ will look dissimilar, we can learn a common representation that allows us to recognize both, among other actions. We formulate this as an embedding problem, and use scaled dot product attention to learn from weakly-supervised video narrations. We jointly learn adverbs as invertible transformations which operate on the embedding space, so as to add or remove the effect of the adverb. As there is no prior work on weakly supervised learning from adverbs, we gather paired action-adverb annotations from a subset of the HowTo100M dataset, for 6 adverbs: quickly/slowly, finely/coarsely and partially/completely. Our method outperforms all baselines for video-to-adverb retrieval with a performance of 0.719 mAP. We also demonstrate our model’s ability to attend to the relevant video parts in order to determine the adverb for a given action.