MultiCLU: Multi-stage Context Learning and Utilization for Storefront Accessibility Detection and Evaluation
MultiCLU: Multi-stage Context Learning and Utilization for Storefront Accessibility Detection and Evaluation
复制标题
DOI：
10.1145/3512527.3531361
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
X. Wang;Jiajun Chen;Hao Tang;Zhigang Zhu
中科院分区：
文献类型：
--
作者：
X. Wang;Jiajun Chen;Hao Tang;Zhigang Zhu
在这项工作中，从谷歌街景中收集了一个店面可达性图像数据集，并标记了三个主要的店面可达性对象：门（用于商店入口），门把手（用于进入入口）和楼梯（用于通往入口）。在此基础上，提出了一种新的多阶段语境学习和利用方法MultiCLU，该方法分为四个阶段：标记语境（CIL）、训练语境（CIT）、检测语境（CID）和评价语境（CIE）。CIL阶段自动扩展每个旋钮的标签，以包含更多的本地上下文信息。在CIT阶段，使用深度学习方法将基于Faster R-CNN的对象检测器提取的视觉信息投影到由Graph Convolutional Network生成的语义空间中。CID阶段使用类别之间的空间关系推理来细化置信分数。最后，在CIE阶段，提出了一个新的松散的店面可达性评价指标，特别是旋钮类别，以有效地帮助BLV用户找到估计的旋钮位置。实验结果表明，与使用Faster R-CNN的基准检测器相比，所提出的MultiCLU框架的性能明显更好，mAP和recall分别为+13.4%和+15.8%。我们的新评价指标也引入了一种新的评价店面可访问性对象的方法，这对现实生活中的BLV群体有很大的帮助。
In this work, a storefront accessibility image dataset is collected from Google street view and is labeled with three main objects for storefront accessibility: doors (for store entrances), doorknobs (for accessing the entrances) and stairs (for leading to the entrances). Then MultiCLU, a new multi-stage context learning and utilization approach, is proposed with the following four stages: Context in Labeling (CIL), Context in Training (CIT), Context in Detection (CID) and Context in Evaluation (CIE). The CIL stage automatically extends the label for each knob to include more local contextual information. In the CIT stage, a deep learning method is used to project the visual information extracted by a Faster R-CNN based object detector to semantic space generated by a Graph Convolutional Network. The CID stage uses the spatial relation reasoning between categories to refine the confidence score. Finally in the CIE stage, a new loose evaluation metric for storefront accessibility, especially for knob category, is proposed to efficiently help BLV users to find estimated knob locations. Our experiment results show that the proposed MultiCLU framework can achieve significantly better performance than the baseline detector using Faster R-CNN, with +13.4% on mAP and +15.8% on recall, respectively. Our new evaluation metric also introduces a new way to evaluate storefront accessibility objects, which could benefit BLV group in real life.