CAREER: Personalized Speech Enhancement: Test-Time Adaptation Using No or Few Private Data

目前的通用语音增强系统采用从音频信号大数据集训练的大型模型，这些模型过于庞大，无法在小型个人设备上运行。个性化模型可以是一种资源高效的解决方案，因为它关注于一个特定的用户和一个特定的测试环境，对于这些环境，一个较小的模型体系结构就足够好了。然而，训练一个个性化的模型需要事先从测试时间的用户那里获得干净的语音数据，由于用户的隐私问题或录音问题，这些数据并不总是可用的。这个CAREER项目开发了机器学习方法来实现个性化目标，同时不需要或只需要很少的测试时间用户的数据样本。由于该项目以保护隐私和节约资源的方式实现了个性化目标，因此它是向所有社会成员更容易获得和负担得起的人工智能使用迈出的一步。该项目在对抗性学习和自我监督学习的帮助下，使用无镜头和少镜头学习框架，解决了个性化语音增强背景下缺乏个人数据的问题。首先，它验证了降低计算复杂度的个性化系统仍然可以在语音增强性能上与通用模型竞争。为此，训练算法将潜在的大模型分成多个子模块，每个子模块处理一个特定的子问题（例如，一个特定的用户的话语）。如果将子问题定义为互斥的，则只需使用最合适的子模块即可有效地进行测试时间推断。由于子模块的选择是在噪声语音上完成的，因此无需对测试用户的数据进行额外的训练即可实现个性化。其次，该项目探索了一种无射击学习方法，其中最基本的挑战在于优化没有可用目标的机器学习模型。为此，使用对抗性优化对已经训练好的通用模型进行微调，以适应看不见的测试环境。第三个研究课题处理少量用户干净语音的情况，属于few-shot学习的范畴。该项目通过一种自监督学习方法克服了数据短缺的问题，该方法从有噪声的语音数据中学习有效的特征，这些特征比干净的语音数据更可用。这样，模型就可以为后续的微调步骤做好准备，这可以只用几个干净的用户特定语音来完成。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Current general-purpose speech enhancement systems employ large models trained from big datasets of audio signals which are too bulky to run on small personal devices. A personalized model can be a resource-efficient solution because it focuses on a particular user and a specific test environment for which a smaller model architecture can be good enough. However, training a personalized model requires clean voice data from the test-time user in advance, which are not always available because of the user’s privacy concerns or problems with recording. This CAREER project develops machine-learning methods to achieve the personalization goal while requiring no or few data samples from the test-time users. Because the project achieves the personalization goal in a privacy-preserving and resource-efficient way, it is a step towards a more available and affordable use of artificial intelligence for all members of society.The project circumvents the lack of personal data in the context of personalized speech enhancement using no- and few-shot learning frameworks with help from adversarial and self-supervised learning. First, it verifies that a personalized system with reduced computational complexity can still compete with a generic model in speech enhancement performance. To this end, the training algorithm divides the potentially large model into multiple sub-modules, each of which handles a particular sub-problem (e.g., a particular user's utterance). If the sub-problems are defined to be mutually exclusive, the test-time inference can be made efficiently by using only the most suitable sub-module. Since the sub-module selection is done on noisy speech, it achieves personalization with no additional training on the test user's data. Second, the project explores a no-shot learning approach, in which the fundamental challenge lies in optimizing a machine learning model with no available target. To this end, an already-trained general-purpose model is fine-tuned for an unseen test environment using adversarial optimization. The third research topic handles the case when a small amount of user's clean speech is available, which falls in the category of few-shot learning. The project overcomes data shortage via a self-supervised learning method that learns effective features from noisy speech data, which are more available than the clean ones. That way, the model can be prepared for a subsequent fine-tuning step, which can be done with only a few clean user-specific speech utterances.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.