Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows

We present a new multimodal dataset called Visual Recipe Flow, which enables us to learn a cooking action result for each object in a recipe text. The dataset consists of object state changes and the workflow of the recipe text. The state change is represented as an image pair, while the workflow is represented as a recipe flow graph. We developed a web interface to reduce human annotation costs. The dataset allows us to try various applications, including multimodal information retrieval.

我们提出了一个名为“视觉食谱流程”的新的多模态数据集，它使我们能够学习食谱文本中每个对象的烹饪动作结果。该数据集由对象状态变化和食谱文本的工作流程组成。状态变化以一对图像表示，而工作流程以食谱流程图表示。我们开发了一个网络界面以降低人工标注成本。该数据集使我们能够尝试各种应用，包括多模态信息检索。