RI: Small: Taming Massive Pre-trained Models under Label Scarcity via an Optimization Lens

深度迁移学习(DTL)在图像和语音识别等实际应用中取得了显著的进展。在这些应用中训练深度学习模型通常需要大量的标记数据(例如，带有注释对象的图像)。然而，人工为这些数据贴上标签可能非常昂贵和耗时，这大大限制了深度学习的更广泛采用。这一问题在某些领域(例如生物医学领域)更为明显，因为这些领域的标签数据非常稀缺。为了解决标签稀缺的问题，研究人员求助于深度迁移学习，即大规模的深度学习模型首先只使用未标记的数据进行预训练，然后在只有有限的标记数据的情况下适应我们感兴趣的下游任务。然而，由于庞大的预训练模型和有限的标签数据之间的差距，这种深度迁移学习方法容易过度拟合，并且不能很好地对未知数据进行泛化，特别是在存在噪声标签的情况下。此外，当存储/内存使用、推理延迟和能耗受到限制时，巨大的模型大小使实际部署变得非常困难，尤其是在边缘设备上。该项目旨在开发一种高效的计算框架，以提高深度迁移学习的泛化能力，并利用前沿优化和机器学习技术来减少模型规模。具体而言，本项目旨在开发：(I)新的对抗性正则化方法，它可以使深度学习模型的复杂性正则化，并防止训练数据过拟合；(Ii)新的自我训练方法，它对训练数据中的噪声标签具有鲁棒性；(Iii)新的优化方法，它可以改进深度迁移学习中紧凑深度学习模型的训练。此外，我们将发展新的推广和逼近理论，以了解我们提出的方法在迁移学习中的好处。这项拟议的研究还将以易于使用的库的形式提供开源软件，这有助于研究人员和从业者在相关领域应用DTL。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep transfer learning (DTL) has made significant progress in many real-world applications such as image and speech recognition. Training deep learning models in these applications often requires large amounts of labeled data, (e.g., images with annotated objects). Labelling these data by human labor, however, can be very expensive and time-consuming, which significantly limits the broader adoption of deep learning. Such an issue is more pronounced in certain domains (e.g. biomedical domain), where labeled data are scarce. To address the concern of label scarcity, researchers have resorted to deep transfer learning, where a massive deep learning model is first pre-trained only using unlabeled data and then adapted to the downstream task of our interests with only limited labelled data. Due to the gap between the enormous sizes of the pre-trained models and the limited labeled data, however, such a deep transfer learning approach is prone to overfitting and fail to generalize well on the unseen data, especially when there are noisy labels. Moreover, the enormous model sizes make practical deployment very difficult when there are constraints on storage/memory usage, inference latency and energy consumption, especially on edge devices. This project aims to develop an efficient computational framework to improve the generalization of deep transfer learning and reduce the model sizes by leveraging cutting-edge optimization and machine learning techniques.Specifically, this project aims to develop: (I) new adversarial regularization methods, which can regularize the complexity of deep learning models and prevent overfitting of the training data, (II) new self-training methods robust to noisy labels in the training data, and (III) new optimization methods, which can improve the training of compact deep learning models in deep transfer learning. Moreover, we will develop new generalization and approximation theories for understanding the benefits of our proposed methods in transfer learning. The proposed research will also deliver open-source software in the form of easy-to-use libraries, which facilitate researchers and practitioners to apply DTL in related fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.