CRII: SCH: Semi-Supervised Physics-Based Generative Model for Data Augmentation and Cross-Modality Data Reconstruction

CRII：SCH：基于半监督物理的数据增强和跨模态数据重建生成模型

• 批准号: 1755695
• 负责人: Sarah Ostadabbas
• 金额: $ 16.87万
• 依托单位: Northeastern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-06-01 至 2021-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1755695&HistoricalAwards=false
• 关键词: CRII; SCH; Semi; Supervised; Physics

Deep learning approaches have been rapidly adopted across a wide range of fields because of their accuracy and flexibility, but require large labeled training sets. This presents a fundamental problem for applications with limited, expensive or private data, such as in healthcare. One example of these applications with the small data challenges is human in-bed pose and pressure estimation. In-bed pose estimation can be a critical part of prevention, prediction, and management of movement-related problems like pressure ulcers. These pressure ulcers often lead to costly and painful conditions such as bedsores. In this research, we propose a semi-supervised generative model based on novel data augmentation and cross-modality data reconstruction techniques to expand the use of powerful deep learning approaches to the in-bed pose and pressure estimation problems. This grant will directly fund the education and mentorship of graduate students involved in researching these problems. In addition, middle school and high school students will be engaged through summer school mentorship programs at Northeastern University.The educational outreach funded by this grant will be used to mentor at schools primarily serving minority student populations. This comprehensive mentorship from middle school to PhD creates a pipeline of experienced students in this important area. The PI actively maintains a diverse research group which includes 50% women and other members of under-represented groups. This proposed research explores the use of semi-supervised physics-based generative models to bridge the gap between state-of-the-art deep learning techniques and the small data problem common in personalized healthcare and other data-limited domains. The use of a physics-based approach to generate image data from a low-dimensional parameter space is unique and transformative. This proposal organizes the research to two Thrusts: (I) data augmentation, which synthesizes the large training set required to train a deep learning model to recognize the in-bed pose from an image; and (II) cross-modality data reconstruction, which extracts pose parameters from one image modality to generate data in another image modality. The success of the data augmentation will be measured by using the synthesized image data to train a network, which will be tested against deep and non-deep models trained on publicly-available pose datasets. The accuracy of the pressure image reconstruction will be tested by comparing the results to pressure images taken from a high-resolution pressure sensing mat. The successful completion of this project enables (1) the use of high-accuracy deep learning techniques for robustly recognizing objects and object poses for which articulated 3D models are available or can be generated; and (2) generating highly realistic images of posable figures in one sensory domain using data from another, when one sensory domain is cheaper or easier to gather data in than others.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习方法由于其准确性和灵活性而在广泛的领域中迅速被采用，但需要大量的标记训练集。这对于具有有限、昂贵或私有数据的应用（例如医疗保健）来说是一个根本问题。具有小数据挑战的这些应用的一个示例是人在床上的姿势和压力估计。床上姿势估计可以是预防、预测和管理与运动相关的问题（如压疮）的关键部分。这些压力性溃疡往往导致昂贵和痛苦的条件，如褥疮。在这项研究中，我们提出了一种基于新的数据增强和跨模态数据重建技术的半监督生成模型，以将强大的深度学习方法扩展到床内姿势和压力估计问题。这笔赠款将直接资助研究这些问题的研究生的教育和指导。此外，初中和高中学生将通过东北大学的暑期学校辅导计划参与。这笔赠款资助的教育推广活动将用于主要为少数民族学生服务的学校的辅导。这种从中学到博士的全面指导在这一重要领域创造了一个有经验的学生管道。PI积极维持一个多样化的研究小组，其中包括50%的妇女和代表性不足群体的其他成员。这项拟议的研究探索了使用基于半监督物理的生成模型来弥合最先进的深度学习技术与个性化医疗保健和其他数据有限领域中常见的小数据问题之间的差距。使用基于物理的方法从低维参数空间生成图像数据是独特的和变革性的。该提案将研究组织为两个重点：（I）数据增强，其合成训练深度学习模型以从图像中识别床上姿势所需的大型训练集;以及（II）跨模态数据重建，其从一种图像模态中提取姿势参数以生成另一种图像模态中的数据。数据增强的成功将通过使用合成图像数据来训练网络来衡量，该网络将针对在公开可用的姿势数据集上训练的深度和非深度模型进行测试。将通过将结果与从高分辨率压力传感垫拍摄的压力图像进行比较来测试压力图像重建的准确性。该项目的成功完成使得（1）能够使用高精度深度学习技术来鲁棒地识别对象和对象姿态，其中关节式3D模型可用或可以生成;以及（2）使用来自另一个感觉域的数据在一个感觉域中生成可摆姿势的人物的高度逼真的图像，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响进行评估来支持审查标准。

项目成果

A Semi-Supervised Data Augmentation Approach using 3D Graphical Engines

• DOI: 10.1007/978-3-030-11012-3_31
• 发表时间: 2018-08
• 作者: Shuangjun Liu;S. Ostadabbas

In-Bed Pose Estimation: Deep Learning With Shallow Dataset

• DOI: 10.1109/jtehm.2019.2892970
• 发表时间: 2019-01-01
• 期刊: IEEE JOURNAL OF TRANSLATIONAL ENGINEERING IN HEALTH AND MEDICINE
• 影响因子: 3.4
• 作者: Liu, Shuangjun;Yin, Yu;Ostadabbas, Sarah
• 通讯作者: Ostadabbas, Sarah