Utilizing Mask R-CNN for Solid-Volume Food Instance Segmentation and Calorie Estimation
Utilizing Mask R-CNN for Solid-Volume Food Instance Segmentation and Calorie Estimation
复制标题
DOI：
10.3390/app122110938
复制
发表时间：
2022-11-01
影响因子：
2.7
通讯作者：
Lee, Kidong
中科院分区：
文献类型：
--
作者：
Dai, Yanyan;Park, Subin;Lee, Kidong
为了预防或应对慢性疾病，使用智能设备自动分类食物类别、估计食物量和营养成分以及记录饮食摄入量被认为是挑战。在这项工作中，采用了一种基于 Mask R-CNN 的新型实时视觉方法，用于固体体积食物实例分割和卡路里估计。为了在现实生活中解决所提出的方法，将其与使用 3D LiDAR 或 RGB-D 相机的其他方法区分开来，本工作应用 RGB 图像来训练模型，并使用简单的单目相机来测试结果。选择紫菜包饭作为固体体积食品的例子来展示所提出方法的利用。首先，为了提高检测精度，提出了基于 Gimbap 在盘子中的姿势的 Gimbap 图像数据集的标记方法。其次，通过微调 Mask R-CNN 架构创建了检测 Gimbap 的优化模型。训练后，模型达到 Gimbap1 的 AP (0.5 IoU) 88.13% 和 Gimbap2 的 AP (0.5 IoU) 82.72%。实现了 85.43% 的 mAP（0.5 IoU）。第三，结合校准结果和 Gimbap 实例分割结果，提出了一种新颖的卡路里估计方法。在第四部分中，还展示了如何将卡路里估算方法扩展到任何固体体积食物，例如披萨、蛋糕、汉堡、炸虾、橙子和甜甜圈。与其他基于 Faster R-CNN 的食物热量估计方法相比，该方法使用掩模信息并考虑看不见的食物。因此，本文的方法优于食物分割和卡路里估计的准确性。所提出方法的有效性已得到证明。
To prevent or deal with chronic diseases, using a smart device, automatically classifying food categories, estimating food volume and nutrients, and recording dietary intake are considered challenges. In this work, a novel real-time vision-based method for solid-volume food instance segmentation and calorie estimation is utilized, based on Mask R-CNN. In order to address the proposed method in real life, distinguishing it from other methods which use 3D LiDARs or RGB-D cameras, this work applies RGB images to train the model and uses a simple monocular camera to test the result. Gimbap is selected as an example of solid-volume food to show the utilization of the proposed method. Firstly, in order to improve detection accuracy, the proposed labeling approach for the Gimbap image datasets is introduced, based on the posture of Gimbap in plates. Secondly, an optimized model to detect Gimbap is created by fine-tuning Mask R-CNN architecture. After training, the model reaches AP (0.5 IoU) of 88.13% for Gimbap1 and AP (0.5 IoU) of 82.72% for Gimbap2. mAP (0.5 IoU) of 85.43% is achieved. Thirdly, a novel calorie estimation approach is proposed, combining the calibration result and the Gimbap instance segmentation result. In the fourth section, it is also shown how to extend the calorie estimation approach to be used in any solid-volume food, such as pizza, cake, burger, fried shrimp, oranges, and donuts. Compared with other food calorie estimation methods based on Faster R-CNN, the proposed method uses mask information and considers unseen food. Therefore, the method in this paper outperforms the accuracy of food segmentation and calorie estimation. The effectiveness of the proposed approaches is proven.