I-Corps: Multi-Cue Facial Restoration (McFAR) for Recognition and Identification

I-Corps项目的更广泛影响/商业潜力是开发多线索正面面部恢复和识别系统，该系统可被安全当局或部署在监视设施上，以合成嫌疑人肖像并根据有限信息进行身份匹配。最近，在公共安全和安保领域越来越多地采用面部监控是获得市场显著牵引力的重要因素。随着对国家边界、机场、海港和公共交通枢纽的威胁，对先进安全系统的需求很高。监控录像经常被自动记录和分析。由于强大的深度学习模型，视频人脸识别性能得到了显著提升，但在现实世界中，由于环境的限制，监控摄像头提供的分辨率有限，只能捕捉目标嫌疑人的侧视图。低分辨率的非正面人脸识别仍然是一个挑战。大多数现有的人脸识别算法都假设高分辨率、近正面的人脸，但对于低分辨率的侧视图像却无法提供足够的性能。该技术利用多方面的信息进行更忠实的人脸识别，为安全应用提供了一种新的选择。I-Corps项目基于多线索正面人脸恢复系统的开发，该系统提供了一个识别和识别框架，可以从低分辨率侧面人脸和目击者提供的叙述描述中合成身份保留、高分辨率正面人脸图像。该技术利用人脸图像的一系列极端姿态，通过超分辨率集成网络合成高分辨率正面人脸。它的目标是通过监控系统提供的多种线索以及目击者的描述来识别低分辨率的极端姿态人脸。为了提高学习表征的判别能力，采用类内和类间约束来惩罚冗余特征。在生成模型中使用正交正则化代替原始融合方法进行最优训练并学习更大跨度的综合表示。目击者可以提供面部的叙述性描述，这些描述被输入语言编码器并转换为属性级表示。通过利用基于转换器的语言编码器进行图像翻译，具有空间约束的描述引导面部编辑网络能够编辑输入图像的属性级和几何内容。人脸编辑网络对编码后的人脸进行各种属性修改，生成精细化的人脸，实现高保真图像匹配。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact/commercial potential of this I-Corps project is the development of a multi-cue frontal face restoration and identification system that may be utilized by security authorities or deployed on surveillance facilities to synthesize suspect portraits and conduct identity matching with limited information. Recently, the rising adoption of facial surveillance in public safety and security is a vital factor gaining significant tractions in the market. With the threats to national borders, airports, seaports and public transportation hubs, advanced security systems are in high demand. Surveillance camera videos are often recorded and analyzed automatically. Video face recognition performance has been significantly boosted due to the powerful deep learning models but in the real-world, surveillance cameras provide limited resolution and only capture side-views of the target suspects due to limitations of the environment. Recognition with low-resolution, non-frontal faces remains a challenge. Most existing face recognition algorithms assume high-resolution, near-frontal faces yet cannot provide sufficient performance with low-resolution, side-view images. The proposed technology utilizes multiple aspects of information to conduct a more faithful face identification and recognition, which provides a novel option for security applications.This I-Corps project is based on the development of a multi-cue frontal face restoration system that provides a recognition and identification framework to synthesize identity-preserving, high-resolution frontal face images from low-resolution, side-view faces and a narrative description provided by eyewitnesses. The proposed technology utilizes a series of extreme poses of face images via a super-resolution integrated network to synthesize high-resolution frontalized faces. It aims to recognize low-resolution extreme-pose faces with multiple cues provided by the surveillance system as well as witness descriptions. To improve the discriminative ability of learning representation, intra- and inter-class constraints are imposed to penalize redundant features. Instead of employing naive fusion methods, orthogonal regularization is used in a generative model for optimal training and to learn a comprehensive representation of broader spans. Eyewitnesses may provide narrative description of the face that is fed into a language encoder and converted to attribute-level representations. A description-guided face editing network with spatial constraints enables the ability of editing both attribute-level and geometric contents of input images by leveraging transformer-based language encoder for image translation. The face editing network takes the encoded features and modifies the frontalized faces with various attributes to generate refined frontal faces for high fidelity image matching.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.