Sequential Prediction with Logic Constraints for Surgical Robotic Activity Recognition
Sequential Prediction with Logic Constraints for Surgical Robotic Activity Recognition
复制标题
DOI：
10.1109/ro-man50785.2021.9515358
复制
发表时间：
2021-08
期刊：
影响因子：
--
通讯作者：
Md Masudur Rahman;R. Voyles;J. Wachs;Yexiang Xue
中科院分区：
文献类型：
--
作者：
Md Masudur Rahman;R. Voyles;J. Wachs;Yexiang Xue
许多现实世界的时间敏感和高风险应用程序（例如，手术、救援和恢复机器人）表现出顺序性质;因此，应用基于递归神经网络（RNN）的顺序模型是检测机器人活动的有吸引力的方法。这种方法的一个局限性是数据稀缺。因此，有限的训练样本可能会导致过度拟合，从而在部署期间产生不正确的预测。然而，丰富的领域知识可能仍然是可用的，这可能有助于制定逻辑约束。在本文中，我们提出了一种新的方法，将领域知识集成到基于RNN的顺序预测中。我们构建了一个基于马尔可夫逻辑网络（MLN）的分类器，可以自动从数据中学习约束权重。我们提出了两种方法来合并这种基于MLN的预测：（i）PriorLayer，其中RNN隐藏层的值与从附加神经网络层中的逻辑约束学习的权重相结合，以及（ii）合并，其中来自RNN预测的类概率和约束权重基于类概率的合并进行组合。我们在模拟的OpenAI Gym环境和真实世界的手术机器人DESK数据集上评估了机器人活动分类方法。我们观察到，我们提出的基于MLN的方法提高了基于LSTM的网络的性能。特别是，MLN将Gym数据集上LSTM的准确率从71%提高到84%，将Taurus机器人数据集上的准确率从68%提高到72%。此外，MLN（即，PriorLayer）显示了正则化能力，它提高了初始LSTM训练的准确性，同时避免了早期的过度拟合，从而提高了对未知数据的最终分类准确性。该代码可在https://github.com/masud99r/prediction-with-logic-constraints上获得。
Many real-world time-sensitive and high-stake applications (e.g., surgical, rescue, and recovery robotics) exhibit sequential nature; thus, applying Recurrent Neural Network (RNN)-based sequential models is an attractive approach to detect robotic activity. One limitation of such approaches is data scarcity. As a result, limited training samples may lead to over-fitting, producing incorrect predictions during deployment. Nevertheless, abundant domain knowledge may still be available, which may help formulate logic constraints. In this paper, we propose a novel way to integrate domain knowledge into RNN-based sequential prediction. We build a Markov Logic Network (MLN)-based classifier that automatically learns constraint weights from data. We propose two methods to incorporate this MLN-based prediction: (i) PriorLayer, in which the values of the hidden layer of the RNN are combined with weights learned from logic constraints in an additional neural network layer, and (ii) Conflation, in which class probabilities from RNN predictions and constraint weights are combined based on the conflation of class probabilities. We evaluate robotic activity classification methods on a simulated OpenAI Gym environment and a real-world DESK dataset for surgical robotics. We observe that our proposed MLN-based approaches boost the performance of LSTM-based networks. In particular, MLN boosts the accuracy of LSTM from 71% to 84% on the Gym dataset and from 68% to 72% on the Taurus robot dataset. Furthermore, MLN (i.e., PriorLayer) shows regularization capability where it improves accuracy in initial LSTM training while avoiding over-fitting early, thus improves the final classification accuracy on unseen data. The code is available at https://github.com/masud99r/prediction-with-logic-constraints.