带隐私保护的异步多模态线索语自动识别研究

线索语(Cued Speech, CS)是一种利用手对音素的提示来辅助唇读的交流方式，主要用于聋哑人之间及其与听力正常人之间的沟通。作为聋哑人的人机互动研究的一部分，CS视频到文本的自动识别可以使得聋哑人的交流更加便利和高效，具有重要的科学意义和应用价值。该领域的研究在近些年已经得到了较好的发展，但仍存在三个难题：CS多模态特征表征能力受限、异步异质多模态特征融合效果不佳，以及缺乏高效且带隐私保护的CS自动识别模型。本项目拟创新性地研究自监督对比学习与无偏学习的协同建模、基于统计分析与生成对抗训练顺序优化的融合机理，以及基于联邦学习的加密共享训练机制，以期实现首个高效且带隐私保护的CS自动识别模型。同时，本项目还将建立首个多语种-多编码者的CS公开数据集，为此研究打好基础。实际上，本项目的研究成果也可服务于听力障碍者早期教育、视听转换以及人机互动等智能应用，促进这些领域的发展。

Cued Speech (CS) is an augmented lip reading system complemented by hand coding in the phonetic level, and it is mainly used for the communication among the deaf people. As one part of research on the human-computer interaction of deaf people, the automatic CS recognition aims to make the communication of deaf people more convenient and efficient, and thus has important social and scientific significance. The automatic CS recognition research based on machine learning has been developed in recent years. However, there are still three challenging problems unsolved that inhibit further development of the CS recognition model: (1) CS multi-modal feature representation ability is still limited; (2) the asynchronous multi-modal feature fusion performance is not satisfactory, and (3) the lack of an efficient and data privacy-protected automatic CS recognition model. In this project, we will try to solve these three problems by studying (1) the collaborative modeling of self-supervised contrastive learning and unbiased learning; (2) statistical analysis and adversarial training-based sequential optimization for multi-modal fusion method, and (3) the encrypted shared training mechanism of the CS recognition federated learning framework. Besides, this project will build the first public “multilingual-multispeaker” CS dataset to lay a solid foundation for this research. In fact, the research outcomes of this project can serve many intelligent applications, such as the early education of the hearing impaired, audio-visual conversion and human-computer interaction, further promoting the development of these fields.