CAREER: Mitigating the Lack of Labeled Training Data in Machine Learning Based on Multi-level Optimization

机器学习在自动驾驶、疾病早期检测、药物设计等众多应用中取得了巨大成功。机器学习模型的准确性高度依赖于大规模、人工标记的训练数据的可访问性。然而，在医疗保健、立法、环境科学等专业领域，由于获取高级人类标签和数据隐私问题所涉及的高昂成本，此类数据通常非常难以获取。该项目将通过提供算法、软件和系统来推进科学，这些算法、软件和系统可以自动生成高质量的标记数据，以缓解特定领域缺乏标记训练数据的问题，并允许训练高精度的机器学习模型。该项目将通过降低数据壁垒，大大拓宽机器学习在各个应用领域的适用性，并将大大降低人工数据注释的人工成本。例如，它将促进结构生物学和高能物理领域的科学发现，简化无线通信领域的工程设计。它将促进败血症、肺癌、帕金森病和睡眠呼吸暂停的早期发现，改善患者的预后和生活质量。应用于化合物设计和水泥生产，开发的技术有可能加快药物发现和降低能源消耗。为了实现创建高质量标记训练数据的目标，本项目将开发基于多层次优化和大型语言模型的三种互补的新方法范式，用于：1)端到端标记数据生成；2)对未标注数据进行标注；3)分别对标记的源数据进行实例化适配/选择。首先，所提出的数据生成方法将利用下游模型的最坏情况和特定类别的性能，为生成（带有复杂标签的）数据提供端到端的细粒度指导，以提高下游模型的准确性和鲁棒性，并促进不同类别之间的平衡性能。其次，建议的数据注释方法将利用端到端机制，利用大型语言模型、一系列验证过程和可用的侧信息来最大化生成标签的准确性。第三，提出的适应/选择方法将区分目标域内部或外部的源示例，并随后确定端到端特定于示例的适应/选择操作，以确保源数据的最佳使用。此外，所提出的新型优化算法和分布式系统将有效地解决与多级优化相关的新挑战，包括不可微性、与大型语言模型的优化器不兼容以及可扩展性。该项目是第一个系统地利用多层次优化来创建标记数据的项目，有效地解决了一个基本的知识缺口，即现有方法通常缺乏执行多个学习阶段的端到端执行的能力，因此在定制生成的数据以提高下游模型的性能方面存在不足。这个项目的另一个重要创新是它有效地利用了大型语言模型进行数据标注，这将大大降低人工标注的成本。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning has demonstrated great success in numerous applications such as autonomous driving, early detection of diseases, drug design, etc. The accuracy of machine learning models highly depends on the accessibility of large-scale, human-labeled training data. However, such data is often very challenging to acquire in specialized domains such as healthcare, legislation, environmental sciences due to the high costs involved in obtaining high-grade human labels and data privacy concerns. This project will advance science by providing algorithms, software, and systems that can automatically generate high-quality labeled data to mitigate the lack of labeled training data in specific domains and and allow training of highly accurate machine learning models. The project will significantly broaden the applicability of machine learning across various application areas by lowering data barriers and will substantially reduce the labor costs of manual data annotation. For example, it will promote scientific discovery in structural biology and high-energy physics and streamline engineering design in wireless communication. It will facilitate the early detection of sepsis, lung cancer, Parkinson's disease, and sleep apnea, improving patient outcomes and quality of life. Applied to compound design and cement production, the developed technologies have the potential to expedite drug discovery and reduce energy consumption. To achieve the goal of creating high-quality labeled training data, this project will develop three complementary paradigms of novel approaches based on multi-level optimization and large language models, for: 1) end-to-end generation of labeled data; 2) annotation of unlabeled data; and, 3) example-specific adaptation/selection of labeled source data, respectively. First, the proposed data generation methods will leverage the worst-case and class-specific performance of downstream models to provide end-to-end and fine-grained guidance for generating data (with complex labels) that is tailored to improve the accuracy and robustness of downstream models, and to promote balanced performance across different classes. Second, the proposed data annotation methods will leverage an end-to-end mechanism that capitalizes on large language models, a sequence of verification procedures, and available side information to maximize the accuracy of generated labels. Third, the proposed adaptation/selection methods will distinguish between source examples that are inside or outside of a target domain and subsequently determine an example-specific adaptation/selection action end-to-end to ensure optimal use of source data. In addition, the proposed novel optimization algorithms and distributed systems will effectively tackle new challenges related to multi-level optimization, including non-differentiability, incompatibility with the optimizers of large language models, and scalability. This project represents the first one systematically leveraging multi-level optimization to create labeled data, effectively addressing a fundamental knowledge gap that existing methods often lack capabilities to perform end-to-end execution of multiple learning stages and therefore fall short in tailoring generated data to improve downstream models’ performance. Another significant innovation of this project is its effective harnessing of large language models for data annotation, which will substantially reduce the costs of manual labeling.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.