Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosis
Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosis
复制标题
DOI：
10.1145/3587819.3590988
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Mindi Ruan;Xiang Yu;Naifeng Zhang;Chuanbo Hu;Shuo Wang;Xin Li
中科院分区：
文献类型：
--
作者：
Mindi Ruan;Xiang Yu;Naifeng Zhang;Chuanbo Hu;Shuo Wang;Xin Li
我们如何教计算机识别10,000种不同的动作？深度学习已经从监督和无监督发展到自我监督的方法。在本文中，我们提出了一个新的基于对比学习的框架，用于基于决策树的动作分类，包括人与人的交互（HHI）和人与物体的交互（HOI）。其核心思想是将原始的多类动作识别转化为一系列预先构建的决策树上的二元分类任务。在对比学习的新框架下，我们提出了一个交互邻接矩阵（IAM）的设计与骨架图作为骨干建模的各种动作相关的属性，如周期性和对称性。通过构造各种托词任务，我们在决策树上获得了一系列的二进制分类节点，这些节点可以组合起来支持更高级别的识别任务。实验证明，我们的方法在现实世界中的应用范围从互动识别对称检测的潜力。特别是，我们已经证明了基于视频的自闭症谱系障碍（ASD）诊断在加州理工学院面试视频数据库中的良好表现。
How can we teach a computer to recognize 10,000 different actions? Deep learning has evolved from supervised and unsupervised to self-supervised approaches. In this paper, we present a new contrastive learning-based framework for decision tree-based classification of actions, including human-human interactions (HHI) and human-object interactions (HOI). The key idea is to translate the original multi-class action recognition into a series of binary classification tasks on a pre-constructed decision tree. Under the new framework of contrastive learning, we present the design of an interaction adjacent matrix (IAM) with skeleton graphs as the backbone for modeling various action-related attributes such as periodicity and symmetry. Through the construction of various pretext tasks, we obtain a series of binary classification nodes on the decision tree that can be combined to support higher-level recognition tasks. Experimental justification for the potential of our approach in real-world applications ranges from interaction recognition to symmetry detection. In particular, we have demonstrated the promising performance of video-based autism spectrum disorder (ASD) diagnosis on the CalTech interview video database.