CMRDF: A Real-Time Food Alerting System Based on Multimodal Data
CMRDF: A Real-Time Food Alerting System Based on Multimodal Data
复制标题
DOI：
10.1109/jiot.2020.2996009
复制
发表时间：
2022-05
影响因子：
10.6
通讯作者：
Pengfei Zhou;Cong Bai;Jie Xia;Shengyong Chen
中科院分区：
文献类型：
--
作者：
Pengfei Zhou;Cong Bai;Jie Xia;Shengyong Chen
健康的饮食是每个人的主要关注点，特别是对于那些患有特定疾病的人，如糖尿病。与此同时，随着新技术的快速发展，我们可以发现日常饮食与健康之间的深层潜在关系。先进的物联网设备，如智能手环和可穿戴摄像头，使人们随时了解食物与健康的关系成为可能。然而，对于个人来说，记住所有的健康信息并利用它们来调节他们的饮食仍然是困难的。为了解决这些问题，我们提出了一种新的系统，称为跨模态检索糖尿病食品（CMRDF），实现了实时饮食通知的基础上捕获的可穿戴设备的多模态数据。在该系统中，我们提出了一种新的基于图的跨模态检索方法--带排序损失的图相关性分析方法，该方法发现了多模态数据中的潜在信息。我们使用图卷积网络来挖掘模态中的深层潜在信息，并以更细的粒度表示数据。它使用视觉和生理信息来估计用户试图获得的食物是否会导致糖尿病，并详细反馈原因。在MSCOCO数据集和新提出的多模态致糖尿病食物数据库real-life diabetogenic上进行的大量实验表明，所提出的跨模态检索方法优于最先进的方法，CMRDF可以在预防糖尿病患者不适当的食物方面取得可靠的结果。
A healthy diet is a major concern for everyone, especially for those with specific diseases, such as diabetes. Meanwhile, with the rapid development of new technologies, it is feasible for us to detect the deep latent relationship between daily meals and wellbeing. Advanced Internet of Things devices, such as smart bracelets and wearable cameras, make it possible for people to know how food is related to health at any time. However, it is still arduous for individuals to memorize all the health information and utilize them to regulate their diet. To deal with such problems, we propose a novel system called cross-modal retrieval on diabetogenic food (CMRDF) which realizes a real-time dietary notice based on multimodal data captured from wearable devices. In this system, we propose a new graph-based cross-modal retrieval method named graph correlation analysis with ranking loss that finds the latent information in multimodal data. We use graph convolutional networks to dig the deep latent information in modalities and represent the data in finer granularity. It uses visual and physiological information to estimate whether the food that a user tries to obtain is diabetogenic or not, and feeds back the reasons in detail. Extensive experiments on the MSCOCO data set and the new proposed multimodal diabetogenic food database real-life diabetogenic show that the proposed cross-modal retrieval method outperforms state-of-the-art methods and CMRDF can achieve reliable results on preventing diabetic patients from inappropriate food.