Statistical Stratification and Benchmarking of Robotic Grasping Performance
Statistical Stratification and Benchmarking of Robotic Grasping Performance
复制标题
DOI：
10.1109/tro.2023.3306613
复制
发表时间：
2023-12
影响因子：
7.8
通讯作者：
Brice D. Denoun;Miles Hansard;Beatriz León;L. Jamone
中科院分区：
文献类型：
--
作者：
Brice D. Denoun;Miles Hansard;Beatriz León;L. Jamone
机器人抓取是许多现实世界应用的基础，必须系统地评估新方法。然而，在大多数情况下，通过简单地计算给定任务中成功尝试的次数来评估特定方法的性能，然后将此成功率与其他解决方案的成功率进行比较，而不考虑不同实验之间的随机变化（例如，由于传感器噪声或对象放置的变化）。为了解决这个问题，我们将观察到的性能分类为定性排序的结果，从而对结果进行分层。然后，我们将展示如何在统计框架中分析这些结果，该框架解释了实验之间的变异性。我们的方法的优势，证明了在实际比较的四个把握规划算法。特别是，我们表明，所提出的方法允许我们进行几个不同的评价，从一组实验，而不必重复的数据收集过程。我们证明，算法之间的差异，这将是不明显的整体成功率，可以识别和评估。
Robotic grasping is fundamental to many real-world applications, and new approaches must be systematically evaluated. However, in most cases, the performance of a specific approach is assessed by simply counting the number of successful attempts in a given task, and this success rate is then compared to those of other solutions, without taking into account the random variability across different experiments (e.g. due to sensor noise or variations in object placement). In order to address this issue, we classify the observed performance into qualitatively ordered outcomes, thereby stratifying the results. We then show how to analyze these results in a statistical framework, which accounts for the variability between experiments. The advantages of our approach are demonstrated in the practical comparison of four grasp planning algorithms. In particular, we show that the proposed approach allows us to carry out several distinct evaluations from a single set of experiments, without having to repeat the data collection process. We demonstrate that differences between the algorithms, which would not be apparent from overall success rates, can be identified and evaluated.