Low to High Dimensional Modality Hallucination Using Aggregated Fields of View
Low to High Dimensional Modality Hallucination Using Aggregated Fields of View
复制标题
DOI：
10.1109/lra.2020.2970679
复制
发表时间：
2020-01
影响因子：
5.2
通讯作者：
K. Gunasekar;Qiang Qiu;Yezhou Yang
中科院分区：
文献类型：
--
作者：
K. Gunasekar;Qiang Qiu;Yezhou Yang
现实世界的机器人系统处理来自多种模式的数据，尤其是导航和识别等任务。当由于传感器故障或不利环境等因素而无法访问一种或多种模式时，这些系统的性能可能会急剧下降。在这里，我们认为模态幻觉是确保一致的模态可用性并从而减少不利后果的一种有效方法。虽然来自具有更丰富信息的模态（例如 RGB 到深度）的幻觉数据已被广泛研究，但我们研究了更具挑战性的低到高模态幻觉以及机器人和自主系统中有趣的用例。我们提出了一种新颖的幻觉架构，它聚合来自当地社区多个视野的信息，以从现有模态中恢复丢失的信息。该过程是通过捕获数据模态之间的非线性映射来实现的，并且学习的映射用于帮助现有模态减轻在涉及模态丢失的不利场景中对系统造成的风险。我们还在 UWRGBD 和 NYUD 数据集上进行了广泛的分类和分割实验，并证明幻觉减轻了模态损失的负面影响。实现和模型：https://github.com/kausic94/Hallucination。
Real-world robotics systems deal with data from a multitude of modalities, especially for tasks such as navigation and recognition. The performance of those systems can drastically degrade when one or more modalities become inaccessible, due to factors such as sensors’ malfunctions or adverse environments. Here, we argue modality hallucination as one effective way to ensure consistent modality availability and thereby reduce unfavorable consequences. While hallucinating data from a modality with richer information, e.g., RGB to depth, has been researched extensively, we investigate the more challenging low-to-high modality hallucination with interesting use cases in robotics and autonomous systems. We present a novel hallucination architecture that aggregates information from multiple fields of view of the local neighborhood to recover the lost information from the extant modality. The process is implemented by capturing a non-linear mapping between the data modalities and the learned mapping is used to aid the extant modality to mitigate the risk posed to the system in the adverse scenarios which involve modality loss. We also conduct extensive classification and segmentation experiments on UWRGBD and NYUD datasets and demonstrate that hallucination allays the negative effects of the modality loss. Implementation and models: https://github.com/kausic94/Hallucination.