Activity Recognition with Moving Cameras and Few Training Examples: Applications for Detection of Autism-Related Headbanging
Activity Recognition with Moving Cameras and Few Training Examples: Applications for Detection of Autism-Related Headbanging
复制标题
DOI：
10.1145/3411763.3451701
复制
发表时间：
2021-01
期刊：
影响因子：
--
通讯作者：
P. Washington;A. Kline;O. Mutlu;É. Leblanc;C. Hou;N. Stockham;K. Paskov;B. Chrisman;D. Wall
中科院分区：
文献类型：
--
作者：
P. Washington;A. Kline;O. Mutlu;É. Leblanc;C. Hou;N. Stockham;K. Paskov;B. Chrisman;D. Wall
活动识别计算机视觉算法可用于检测自闭症相关行为的存在，包括诊断仪器所谓的“限制性和重复性行为”或刺激。刺激的例子包括拍手、旋转和撞头。实施此类分类器的最重要瓶颈之一是缺乏针对儿科发育迟缓的足够大的人类行为训练集。确实存在的数据通常是用手持相机记录的，手持相机本身是摇晃甚至移动的，这对用于活动检测的传统特征表示方法提出了挑战，传统的特征表示方法将相机的运动捕获为特征。为了解决这些问题，我们首先记录了当前用于活动识别的特征表示技术在应用于头部撞击检测时的优点和局限性。然后，我们提出一种仅由头部姿势关键点组成的特征表示。我们创建了一个计算机视觉分类器，使用时间分布式卷积神经网络 (CNN) 来检测家庭视频中的头部碰撞，其中单个 CNN 从输入序列中的每个帧中提取特征，并将这些提取的特征作为输入馈送到长短期记忆 (LSTM) 网络。在预测自刺激行为数据集 (SSBD) 视频中撞头和不撞头的二元任务中，我们使用 3 折交叉验证（各个折叠 F1 得分分别为 83.3%、89.0% 和 100.0%），达到了 90.77% 的平均 F1 分数，同时确保训练集中出现的孩子没有出现在所有折叠的测试集中。这项工作记录了训练计算机视觉分类器的成功过程，该分类器可以通过很少的训练示例来检测特定的人体运动模式，甚至在记录源剪辑的摄像机不稳定时也是如此。如本文所述，通过视觉检查表示来设计有用的特征表示的过程对于交互系统的设计者和开发者来说是一种有用的实践，用于检测移动和无处不在的交互系统中使用的人体运动模式。
Activity recognition computer vision algorithms can be used to detect the presence of autism-related behaviors, including what are termed “restricted and repetitive behaviors”, or stimming, by diagnostic instruments. Examples of stimming include hand flapping, spinning, and head banging. One of the most significant bottlenecks for implementing such classifiers is the lack of sufficiently large training sets of human behavior specific to pediatric developmental delays. The data that do exist are usually recorded with a handheld camera which is itself shaky or even moving, posing a challenge for traditional feature representation approaches for activity detection which capture the camera's motion as a feature. To address these issues, we first document the advantages and limitations of current feature representation techniques for activity recognition when applied to head banging detection. We then propose a feature representation consisting exclusively of head pose keypoints. We create a computer vision classifier for detecting head banging in home videos using a time-distributed convolutional neural network (CNN) in which a single CNN extracts features from each frame in the input sequence, and these extracted features are fed as input to a long short-term memory (LSTM) network. On the binary task of predicting head banging and no head banging within videos from the Self Stimulatory Behaviour Dataset (SSBD), we reach a mean F1-score of 90.77% using 3-fold cross validation (with individual fold F1-scores of 83.3%, 89.0%, and 100.0%) when ensuring that no child who appeared in the train set was in the test set for all folds. This work documents a successful process for training a computer vision classifier which can detect a particular human motion pattern with few training examples and even when the camera recording the source clip is unstable. The process of engineering useful feature representations by visually inspecting the representations, as described here, can be a useful practice by designers and developers of interactive systems detecting human motion patterns for use in mobile and ubiquitous interactive systems.