Set-Supervised Action Learning in Procedural Task Videos via Pairwise Order Consistency
Set-Supervised Action Learning in Procedural Task Videos via Pairwise Order Consistency
复制标题
DOI：
10.1109/cvpr52688.2022.01928
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Zijia Lu;Ehsan Elhamifar
中科院分区：
文献类型：
--
作者：
Zijia Lu;Ehsan Elhamifar
我们解决了集合监督动作学习的问题，其目标是使用训练视频中发生的动作集形式的弱监督来学习动作分割模型。我们的主要观察结果是，同一任务中的视频具有相似的动作顺序，可以利用这些顺序进行有效的学习。因此，我们提出了一种基于注意力的方法，具有新的成对排序一致性（POC）损失，鼓励对于同一任务的两个视频中的每个常见动作对，动作的注意力遵循相似的顺序。现有的序列对齐方法会导致不同顺序的视频中的动作错位，或者无法可靠地将更多的动作与不太一致的顺序分开，而我们的 POC 损失可以有效地将视频与不同的动作顺序对齐，并且是可微分的，这与现有的序列对齐方法不同，这使得端到端训练成为可能。此外，它避免了先前作品耗时的伪标签生成。我们的方法有效地学习动作及其时间位置，因此，使用我们的 POC 损失以及视频级和帧级损失，将现有的基于注意力的动作定位方法从学习每个视频的一个动作扩展到多个动作。通过对三个数据集的实验，我们证明我们的方法显着提高了现有技术水平。我们还表明，我们的方法经过少量修改，可以有效地解决转录监督动作学习任务，其中动作及其顺序在训练期间可用。11代码可在 https://github.com/ZijiaLewisLu/CVPR22-POC 获取。
We address the problem of set-supervised action learning, whose goal is to learn an action segmentation model using weak supervision in the form of sets of actions occurring in training videos. Our key observation is that videos within the same task have similar ordering of actions, which can be leveraged for effective learning. Therefore, we propose an attention-based method with a new Pairwise Ordering Consistency (POC) loss that encourages that for each common action pair in two videos of the same task, the attentions of actions follow a similar ordering. Unlike existing sequence alignment methods, which misalign actions in videos with different orderings or cannot reliably separate more from less consistent orderings, our POC loss efficiently aligns videos with different action orders and is differentiable, which enables end-to-end training. In addition, it avoids the time-consuming pseudo-label generation of prior works. Our method efficiently learns the actions and their temporal locations, therefore, extends the existing attention-based action localization methods from learning one action per video to multiple actions using our POC loss along with video-level and frame-level losses. By experiments on three datasets, we demonstrate that our method significantly improves the state of the art. We also show that our method, with a small modification, can effectively address the transcript-supervised action learning task, where actions and their ordering are available during training.11Code available at https://github.com/ZijiaLewisLu/CVPR22-POC.