Procedure-Aware Pretraining for Instructional Video Understanding
Procedure-Aware Pretraining for Instructional Video Understanding
复制标题
DOI：
10.1109/cvpr52729.2023.01033
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Honglu Zhou;Roberto Mart'in-Mart'in-Roberto-Mart'in-Mart'in-2196148773;M. Kapadia;S. Savarese;Juan Carlos Niebles
中科院分区：
文献类型：
--
作者：
Honglu Zhou;Roberto Mart'in-Mart'in-Roberto-Mart'in-Mart'in-2196148773;M. Kapadia;S. Savarese;Juan Carlos Niebles
我们的目标是学习一种对教学视频中的下游过程理解任务有用的视频表示。由于可用注释数量较少，程序理解中的一个关键挑战是能够从未标记的视频中提取程序知识，例如任务的身份（例如“制作拿铁”）、其步骤（例如“倒牛奶”）或在执行中取得部分进展的潜在后续步骤。我们的主要见解是，教学视频描述了在相同或不同任务的实例之间重复的步骤序列，并且这种结构可以通过程序知识图（PKG）很好地表示，其中节点是离散步骤，边连接教学活动中顺序发生的步骤。然后，该图可用于生成伪标签来训练视频表示，该视频表示以更易于访问的形式对程序知识进行编码，以推广到多个程序理解任务。我们通过结合来自基于文本的程序知识数据库和未标记的教学视频语料库的信息来构建 PKG，然后使用它来生成具有四个新颖的预训练目标的训练伪标签。我们将这种基于 PKG 的预训练过程和由此产生的模型称为 Paprika，用于教学知识获取的过程感知预训练。我们在 COIN 和 CrossTask 上评估 Paprika 的过程理解任务，例如任务识别、步骤识别和步骤预测。 Paprika 产生的视频表示比现有技术有所改进：在 12 种评估设置中，准确度提高了 11.23%。可以在 https://github.com/salesforce/paprika 上进行实施。
Our goal is to learn a video representation that is useful for downstream procedure understanding tasks in instructional videos. Due to the small amount of available annotations, a key challenge in procedure understanding is to be able to extract from unlabeled videos the procedural knowledge such as the identity of the task (e.g., ‘make latte’), its steps (e.g., ‘pour milk’), or the potential next steps given partial progress in its execution. Our main insight is that instructional videos depict sequences of steps that repeat between instances of the same or different tasks, and that this structure can be well represented by a Procedural Knowledge Graph (PKG), where nodes are discrete steps and edges connect steps that occur sequentially in the instructional activities. This graph can then be used to generate pseudo labels to train a video representation that encodes the procedural knowledge in a more accessible form to generalize to multiple procedure understanding tasks. We build a PKG by combining information from a text-based procedural knowledge database and an unlabeled instructional video corpus and then use it to generate training pseudo labels with four novel pre-training objectives. We call this PKG-based pre-training procedure and the resulting model Paprika, Procedure-Aware PRe-training for Instructional Knowledge Acquisition. We evaluate Paprika on COIN and CrossTask for procedure understanding tasks such as task recognition, step recognition, and step forecasting. Paprika yields a video representation that improves over the state of the art: up to 11.23% gains in accuracy in 12 evaluation settings. Implementation is available at https://github.com/salesforce/paprika.