CRII: SCH: Semi-Supervised Physics-Based Generative Model for Data Augmentation and Cross-Modality Data Reconstruction

由于其准确性和灵活性，深度学习方法已迅速应用于广泛的领域，但需要大量的标记训练集。这对于具有有限、昂贵或私有数据的应用程序（例如医疗保健应用程序）提出了一个基本问题。这些具有小数据挑战的应用程序的一个例子是人在床上的姿势和压力估计。卧床姿势估计是预防、预测和管理与运动有关的问题（如压力溃疡）的关键部分。这些压疮通常会导致昂贵和痛苦的情况，如褥疮。在这项研究中，我们提出了一种基于新型数据增强和跨模态数据重建技术的半监督生成模型，以扩展强大的深度学习方法在床内姿态和压力估计问题中的应用。这项拨款将直接资助研究这些问题的研究生的教育和指导。此外，初中生和高中生将通过东北大学的暑期学校辅导项目参与其中。这笔赠款资助的教育外展活动将用于在主要为少数民族学生服务的学校提供指导。这种从中学到博士的全面指导为这一重要领域的经验丰富的学生创造了一条管道。PI积极维持一个多元化的研究小组，其中包括50%的妇女和其他代表性不足群体的成员。本研究探索了半监督物理生成模型的使用，以弥合最先进的深度学习技术与个性化医疗保健和其他数据有限领域中常见的小数据问题之间的差距。使用基于物理的方法从低维参数空间生成图像数据是独特的和变革性的。本提案将研究分为两个重点：(1)数据增强，即综合训练深度学习模型从图像中识别床上姿势所需的大型训练集；（II）跨模态数据重构，即从一种图像模态中提取位姿参数，生成另一种图像模态的数据。数据增强的成功将通过使用合成图像数据来训练网络来衡量，该网络将针对公开可用的姿态数据集训练的深度和非深度模型进行测试。通过将结果与高分辨率压力传感垫拍摄的压力图像进行比较，将测试压力图像重建的准确性。该项目的成功完成使(1)能够使用高精度深度学习技术来稳健地识别物体和物体姿势，这些物体和物体姿势可以使用或可以生成关节3D模型；(2)使用来自另一个感官领域的数据在一个感官领域生成高度逼真的可能人物图像，当一个感官领域比其他感官领域更便宜或更容易收集数据时。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning approaches have been rapidly adopted across a wide range of fields because of their accuracy and flexibility, but require large labeled training sets. This presents a fundamental problem for applications with limited, expensive or private data, such as in healthcare. One example of these applications with the small data challenges is human in-bed pose and pressure estimation. In-bed pose estimation can be a critical part of prevention, prediction, and management of movement-related problems like pressure ulcers. These pressure ulcers often lead to costly and painful conditions such as bedsores. In this research, we propose a semi-supervised generative model based on novel data augmentation and cross-modality data reconstruction techniques to expand the use of powerful deep learning approaches to the in-bed pose and pressure estimation problems. This grant will directly fund the education and mentorship of graduate students involved in researching these problems. In addition, middle school and high school students will be engaged through summer school mentorship programs at Northeastern University.The educational outreach funded by this grant will be used to mentor at schools primarily serving minority student populations. This comprehensive mentorship from middle school to PhD creates a pipeline of experienced students in this important area. The PI actively maintains a diverse research group which includes 50% women and other members of under-represented groups. This proposed research explores the use of semi-supervised physics-based generative models to bridge the gap between state-of-the-art deep learning techniques and the small data problem common in personalized healthcare and other data-limited domains. The use of a physics-based approach to generate image data from a low-dimensional parameter space is unique and transformative. This proposal organizes the research to two Thrusts: (I) data augmentation, which synthesizes the large training set required to train a deep learning model to recognize the in-bed pose from an image; and (II) cross-modality data reconstruction, which extracts pose parameters from one image modality to generate data in another image modality. The success of the data augmentation will be measured by using the synthesized image data to train a network, which will be tested against deep and non-deep models trained on publicly-available pose datasets. The accuracy of the pressure image reconstruction will be tested by comparing the results to pressure images taken from a high-resolution pressure sensing mat. The successful completion of this project enables (1) the use of high-accuracy deep learning techniques for robustly recognizing objects and object poses for which articulated 3D models are available or can be generated; and (2) generating highly realistic images of posable figures in one sensory domain using data from another, when one sensory domain is cheaper or easier to gather data in than others.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.