Cross-Modality Graph-based Language and Sensor Data Co-Learning of Human-Mobility Interaction
Cross-Modality Graph-based Language and Sensor Data Co-Learning of Human-Mobility Interaction
复制标题
DOI：
10.1145/3610904
复制
发表时间：
2023-09
影响因子：
--
通讯作者：
Mahan Tabatabaie;Suining He;Kang G. Shin
中科院分区：
文献类型：
--
作者：
Mahan Tabatabaie;Suining He;Kang G. Shin
学习交互式场景上的人-移动交互（HMI）（例如，车辆如何响应交通灯和其他迎面而来的车辆在交叉路口转弯）可以增强智能移动系统的安全性、效率和弹性（例如，自动驾驶车辆）和许多其他无处不在的计算应用。对于无处不在和可理解的HMI学习，本文考虑了“口语”（例如，人类文本注释）和“未说出的语言”（例如，与HMI场景相关的视觉和基于传感器的行为移动性信息）。我们的目标是通过一种新的人类语言和传感器数据共同学习设计，从文本注释（由人类注释者提供）中提取重要但可能隐含的HMI概念（作为命名实体）。为此，我们提出了CG-HMI，一种新的跨模态图融合方法，用于从文本注释以及视觉和行为传感器数据的共同学习中提取重要的人类移动交互概念。为了融合无声和有声的“语言”，我们为与HMI场景相关的每种模态设计了一种统一的表示，称为人-移动交互图（HRM），即，文本注释、可视视频帧和行为传感器时间序列（例如，来自车载或智能手机惯性测量单元）。这些模态中的障碍物的节点对应于与HMI概念、检测到的交通参与者/环境类别以及从行为传感器时间序列确定的车辆操纵行为类型相关的文本词（标记化以便于处理）。为了提取Hestival中的模态间和模态内语义对应和交互，我们设计了一种新的基于可区分池的图注意力的图交互融合方法。然后处理所得到的图嵌入以识别和检索注释中的HMI概念，这可以有益于下游人机交互和普适计算应用。我们已经将CG-HMI开发并实现为系统原型，并对三个真实世界的HMI数据集（两个关于汽车驾驶，第三个关于电动滑板车驾驶）进行了广泛的研究。我们已经证实了CG-HMI在通过跨模态学习识别和提取重要HMI概念方面的出色性能（在精度，召回率和F1测量方面平均比其他基线高13.11%）和有效性。我们的CG-HMI研究还提供了现实世界的影响（例如，道路安全和驾驶行为）关于驾驶员和其他交通参与者之间的相互作用。
Learning the human--mobility interaction (HMI) on interactive scenes (e.g., how a vehicle turns at an intersection in response to traffic lights and other oncoming vehicles) can enhance the safety, efficiency, and resilience of smart mobility systems (e.g., autonomous vehicles) and many other ubiquitous computing applications. Towards the ubiquitous and understandable HMI learning, this paper considers both "spoken language" (e.g., human textual annotations) and "unspoken language" (e.g., visual and sensor-based behavioral mobility information related to the HMI scenes) in terms of information modalities from the real-world HMI scenarios. We aim to extract the important but possibly implicit HMI concepts (as the named entities) from the textual annotations (provided by human annotators) through a novel human language and sensor data co-learning design. To this end, we propose CG-HMI, a novel Cross-modality Graph fusion approach for extracting important Human-Mobility Interaction concepts from co-learning of textual annotations as well as the visual and behavioral sensor data. In order to fuse both unspoken and spoken "languages", we have designed a unified representation called the human--mobility interaction graph (HMIG) for each modality related to the HMI scenes, i.e., textual annotations, visual video frames, and behavioral sensor time-series (e.g., from the on-board or smartphone inertial measurement units). The nodes of the HMIG in these modalities correspond to the textual words (tokenized for ease of processing) related to HMI concepts, the detected traffic participant/environment categories, and the vehicle maneuver behavior types determined from the behavioral sensor time-series. To extract the inter- and intra-modality semantic correspondences and interactions in the HMIG, we have designed a novel graph interaction fusion approach with differentiable pooling-based graph attention. The resulting graph embeddings are then processed to identify and retrieve the HMI concepts within the annotations, which can benefit the downstream human-computer interaction and ubiquitous computing applications. We have developed and implemented CG-HMI into a system prototype, and performed extensive studies upon three real-world HMI datasets (two on car driving and the third one on e-scooter riding). We have corroborated the excellent performance (on average 13.11% higher accuracy than the other baselines in terms of precision, recall, and F1 measure) and effectiveness of CG-HMI in recognizing and extracting the important HMI concepts through cross-modality learning. Our CG-HMI studies also provide real-world implications (e.g., road safety and driving behaviors) about the interactions between the drivers and other traffic participants.