Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention-猫眼课题宝

DOI：

10.1109/cvpr52729.2023.00145

发表时间：

2023-03

期刊：

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Sounak Mondal;Zhibo Yang;Seoyoung Ahn;D. Samaras;G. Zelinsky;Minh Hoai

人眼注视预测在人机交互（HCI）中具有重要意义。然而，为了实际服务于HCI应用，凝视预测模型必须具有可扩展性、快速和准确的空间和时间凝视预测。最近的扫描路径预测模型主要关注目标导向注意力（搜索）。这些模型的应用受到限制，因为一种常见的方法依赖于所有可能物体的训练目标检测器，以及用于训练的人类注视数据的可用性（两者都不可扩展）。作为回应，我们提出了一个名为ZeroGaze的新任务，这是零射击学习的一个新变体，它预测了从未搜索过的物体的凝视，我们开发了一个新的模型，Gazeformer；来解决零点问题。与使用目标检测器模块的现有方法相比，Gazeformer使用自然语言模型对目标进行编码，从而在扫描路径预测中利用语义相似性。我们使用基于转换器的编码器-解码器架构，因为转换器对于生成上下文表示特别有用。Gazeformer在ZeroGaze设置上大大超过其他型号（19%-70%）。在目标存在和目标不存在搜索任务的标准凝视预测方面，它也优于现有的目标检测模型。除了性能的提高，Gazeformer比目前最先进的目标视觉搜索模型快五倍以上。代码可以在https://github.com/cvlab-stonybrook/Gazeformer/上找到

Predicting human gaze is important in Human-Computer Interaction (HCI). However, to practically serve HCI applications, gaze prediction models must be scalable, fast, and accurate in their spatial and temporal gaze predictions. Recent scanpath prediction models focus on goaldirected attention (search). Such models are limited in their application due to a common approach relying on trained target detectors for all possible objects, and the availability of human gaze data for their training (both not scalable). In response, we pose a new task called ZeroGaze, a new variant of zero-shot learning where gaze is predicted for never-before-searched objects, and we develop a novel model, Gazeformer; to solve the ZeroGaze problem. In contrast to existing methods using object detector modules, Gazeformer encodes the target using a natural language model, thus leveraging semantic similarities in scanpath prediction. We use a transformer-based encoder-decoder architecture because transformers are particularly useful for generating contextual representations. Gazeformer surpasses other models by a large margin (19%-70%) on the ZeroGaze setting. It also outperforms existing target-detection models on standard gaze prediction for both target-present and target-absent search tasks. In addition to its improved performance, Gazeformer is more than five times faster than the state-of-the-art target-present visual search model. Code can be found at https://github.com/cvlab-stonybrook/Gazeformer/

Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention

文献摘要

被引文献

相似文献